Classification of Date/Time/DateTime

Question

I'd like to share a methodology I've been using when faced with a classification problem. This particular example is designed to classify time data as Date/Time/DateTime. Though I believe it could easily be adapted to many other problems

The basic idea is:

1. Create a list of regular expression's (re) that match features in the string to be classified. The re's should be ordered in the list from highest -> lowest importance
2. Create a match_code that is a list representing what re matched where in the string. Duplicate matches will be replaced with the match of highest importance
3. Create an answer_table that contains the possible classifications
4. Create a response_table that contains match_codes, placed in the corresponding index of the answer_table
5. Check membership of the match_code in the response_table and return the answer that corresponds to the match.

I'm aware machine learning solutions are more robust. This solution came about when I was attempting to create a data set to teach such a system. In the end I found I didn't actually need to go down that path.

Currently the creation of the response table is manual.

Quite new to python programming. And felt feedback would be useful. Review at leisure.

import re
import itertools

def time_classifier(string):
    '''
    classify the string according to features extracted by regular expressions
    '''
    _string = string
    _string = str.lower(_string)
    _string = re.sub(r'\s', ' ', _string).strip()
    #Strings to remove
    match_code = [[]] * 10
    match_code[0] = re.finditer(r'\d+', _string)
    match_code[1] = re.finditer(r'[/\-.|]', _string)
    match_code[2] = re.finditer(r':', _string)
    match_code[3] = re.finditer(r'am|pm', _string)
    match_code[4] = re.finditer(r'jan(uary)?|feb(ruary)?|mar(ch)?|apr(il)?'
                                r'|may|jun(e)?|jul(y)?|aug(ust)?'
                                r'|sep(tember)?|oct(ober)?|nov(ember)?|dec(ember)?', _string)
    match_code[5] = re.finditer(r'mon(day)?|tue(sday)?|wed(nesday)?|thu(rsday)?'
                                r'|fri(day)?|sat(urday)?|sun(day)?', _string)
    match_code[6] = re.finditer(r',', _string)
    match_code[7] = re.finditer(r'today|tomorrow|yesterday', _string)
    match_code[8] = re.finditer(r'aest', _string)
    match_code[9] = re.finditer(r'[a-z]+', _string)
    #Convert re items to an ordered list
    match_code = construct_key(match_code)
    #These are the possible classifications
    answer_table = ['Date', 'Time', 'DateTime']
    response_table = [[]] * len(answer_table)
    #Each response_type element correlates to the elements in the answer_key
    ####################DATE####################
    response_table[0] = [[3, 1, 0, 1, 0, 3],
                        [5, 0, 1, 0, 1, 0],
                        [0, 1, 0],
                        [5, 0, 1, 0],
                        [0, 1, 0, 1, 0, 1, 0],
                        [4, 0, 1, 0, 6, 0, 3],
                        [4, 0],
                        [3, 4, 0, 1, 0, 6, 0, 3],
                        [5, 4, 0, 0],
                        [5, 6, 0, 4, 1],
                        [0, 4, 0],
                        [5, 6, 0, 4, 0, 9],
                        [5, 1, 4, 0],
                        [7, 1, 4, 0],
                        [9, 0, 1, 5, 0, 1, 0, 1, 0],
                        [4, 6, 0, 0],
                        [0, 3]]
    ####################TIME####################
    response_table[1] = [[0, 2, 0, 3],
                        [0, 2, 0],
                        [0, 2, 0, 2, 0, 0],
                        [0, 0, 2, 0],
                        [0, 2, 0, 1],
                        [0, 2, 0, 2, 0],
                        [0, 0, 2, 0, 3],
                        [9, 9, 0, 0, 2, 0],
                        [0, 2, 0, 2, 0, 8],
                        [0, 2, 0, 8],
                        [9, 0, 0, 2, 0, 3],
                        [9, 0, 0, 2, 0, 3]]
    ####################DATETIME####################
    response_table[2] = [[5, 0, 4, 1, 0, 2, 0, 3],
                        [5, 0, 4, 0, 2, 0],
                        [5, 0, 4, 0, 0, 2, 0, 3],
                        [0, 2, 0, 3, 0, 4],
                        [0, 1, 0, 1, 0, 0, 2, 0],
                        [5, 0, 4, 1, 0, 2, 0],
                        [1, 5, 0, 4, 0, 2, 0],
                        [5, 6, 0, 4, 0],
                        [0, 1, 0, 1, 0, 0, 2, 0, 3, 0],
                        [0, 4, 0, 0, 2, 0, 3],
                        [5, 0, 4, 1, 0, 2, 0, 3, 9],
                        [0, 4, 6, 0, 2, 0, 8],
                        [4, 0, 6, 0, 6, 0, 1, 0, 3],
                        [4, 6, 0, 0, 6, 0, 2, 0],
                        [4, 6, 0, 9, 0, 0, 2, 0],
                        [4, 6, 0, 9, 0, 0, 2, 0, 3]]

    result = calculate_classification(match_code, response_table, answer_table, '%s' % string)
    if result:
        return result
    else:
        failed_to_classify_output('Date/Time/DateTime',string, _string, match_code)
        return None

def list_to_string(arg):
    '''
    Same as str(arg) but removes square brackets '[' & ']'
    '''
    return re.sub(r'^\[|\]$', '', str(arg))

def find_matches(response_table, match_code, response_key=None):
    '''
    Compare the match code against items in the response table
    '''
    if not response_key:
        response_key = lambda x: str(x)
    result = [(index, len(ii)) for index, i in enumerate(response_table)
              for ii in i if response_key(ii) in str(match_code)]
    #longer matches are considered 'better'
    result = sorted(result, key=lambda x: x[1], reverse=True)
    return result

def construct_key(key):
    '''
    Unpack the match iterators and remove duplicate matches
    '''
    #Unpack generators
    _key = [(index, y.start()) for index, i in enumerate(key) for y in i]
    _key = sorted(_key, key=lambda x: x[1])
    # Matches removes duplicate matches, the first match gets priority over the rest
    _key = [next(group) for i, group in itertools.groupby(_key, key=lambda x: x[1])]
    _key = [i[0] for i in _key]
    return _key

def calculate_classification(match_code, response_table, answer_table, warning_output='None'):
    '''
    Find which item in response_key is the best fit for the given key.
    Return the corresponding value in answer key.
    If no match is found print error msg and string that couldnt be classified
    '''
    #Check for an exact match
    answer_index = find_matches(response_table, match_code)
    if not answer_index: #If no match exists, find the best fit
        answer_index = find_matches(response_table, list_to_string(match_code), response_key= list_to_string)
        print('Warning: Incomplete match on classifying: "{} MATCH CODE: {} "'.format(str(warning_output), match_code))
    #Use the index to look up the answer
    if answer_index:
        answer_index = next(iter(answer_index))[0]
        return answer_table[answer_index]
    else:
        return None

def failed_to_classify_output(classification_type, raw_string, formatted_string, key):
    '''
    Print the failed classification for review
    '''
    def align(text): #Column width
        return ' ' * (30 - len(text))
    row = [[]] *4
    #Column 1
    row[0] = 'FAILED TO CLASSIFY:{}\n\n'.format(classification_type)
    row[1] = 'Raw string:'
    row[2] = 'Formatted string:'
    row[3] = 'key:'
    #Add column 2 to column 1
    row[1] = '{}{}{}\n'.format(row[1], align(row[1]), raw_string)
    row[2] = '{}{}{}\n'.format(row[2], align(row[2]), formatted_string)
    row[3] = '{}{}{}\n'.format(row[3], align(row[3]), key)

    output = ''.join((i for y in row for i in y))
    print(output + '{}'.format('-'*len(max(row, key= lambda x: len(x)))))

if __name__ == '__main__':

    while True:
        string = input("Please enter a time to classify:\n>?")
        print(time_classifier(string))

Answer 1

Some things I'd do:

• Rename _string to something like normalised_string for clarity.

• Don't overwrite the normalised string repeatedly. This case is a bit of a grey area, because you're effectively just splitting up a chain, but joining the chain back up again would be nicer:

  normalised_string = re.sub(r'\s', ' ', string.lower()).strip()
  

  On second thought, avoiding the normalisation would be even better. Instead use character classes and case insensitive modifiers on the regular expressions. This makes the regular expressions slightly more complex, but can avoid subtle bugs because you're not applying assumptions to the input other than those in the regular expressions.

• Don't declare the blank list; there's no need for it in Python.
• Use list.append() to add items to a list.
• Throw an exception in calculate_classification if it fails, rather than returning a falsy None value.
• readline() from standard input to get the strings to classify. This makes your script scriptable, as in, it can be used from other scripts without having to handle a prompt/feedback loop.
• In general there are too many magic values. Pull out properly named constants, methods or classes to clarify things.

• The response_table should contain references to the actual match_code entries rather than simply the indexes. For example:

  response_table.append([match_code[3], etc.])
  

  Using lots of numeric references makes the code really hard to follow.

• It looks like match_code entries are matched to the input string sequentially using the response_table sequences. A more obvious way of doing this would be to construct more complete regular expressions and match the whole input string at once, such as this answer for ISO dates.