I created a program that parses a Credit Reporting file which is in the Metro 2 file format as described here. Parsing a file is generally trivial but the issue is that in this case, there are no sample files that I could find online that show sample data in this file format; however, in the link provided above, it is stated that the file has one long line of data for each account that is reported and that it is comprised of multiple different segments, the first two of which are the Header Record and the base segment.

Given that there are no partitions (commas, brackets, etc.) to separate each field, the only way I could think to properly parse the file is by slicing the file by the ranges for each field that are specified in the Metro2 layout. This works, but it makes the solution look a little inelegant. I initially thought I could possibly use regex somehow, but it doesn't really seem to apply here. Also, I'm using module-level functions here instead of creating a class to encapsulate the methods. It seems appropriate, but I just want to make sure everything is OK style-wise. Does this solution seem unwieldy or is it the best that can be done here? Here is a sample file I created in Metro 2 format (it's supposed to be all one line but paste bin broke it into multiple). The code is below:

""" credit_report_parser module

The functions contained in this module can be used to parse a Metro2 file and write the 
fields of each file segment to a JSON file.

    The restructure_data function can be imported and passed an initial value for the input data
    file from the command line from within the package like this:

        $ python main.py name_of_data_file.in    
import json

def parse_header(header_segment):
    """Parses the passed in string of characters and breaks it down based on its fields.

    Takes a string and slices it into its different constituent fields which are 
    stored in a dictionary.

        header_segment: The header record segment of the metro 2 file

        A dictionary that contains keys and values of the file fields.
    header_dict = {}
    header_dict["Record Descriptor"] = header_segment[0:4].strip(" ")
    header_dict["Record Identifier"] = header_segment[4:10].strip(" ")
    header_dict["Cycle Number"] = header_segment[10:12].strip(" ")
    header_dict["Innovis Program Identifier"] = header_segment[12:22].strip(" ")
    header_dict["Equifax Program Identifier"] = header_segment[22:32].strip(" ")
    header_dict["Experian Program Identifier"] = header_segment[32:37].strip(" ")
    header_dict["TransUnion Program Identifier"] = header_segment[37:47].strip(" ")
    header_dict["Activity Date"] = header_segment[47:55].strip(" ")
    header_dict["Date Created"] = header_segment[55:63].strip(" ")
    header_dict["Program Date"] = header_segment[63:71].strip(" ")
    header_dict["Program Revision Date"] = header_segment[71:79].strip(" ")
    header_dict["Reporter Name"] = header_segment[79:119].strip(" ")
    header_dict["Reporter Address"] = header_segment[119:215].strip(" ")
    header_dict["Reporter Telephone Number"] = header_segment[215:225].strip(" ")
    header_dict["Software Vendor Name"] = header_segment[225:265].strip(" ")
    header_dict["Software Version Number"] = header_segment[265:270].strip(" ")
    header_dict["Reserved"] = header_segment[270:426].strip(" ")
    return header_dict

def parse_base_segment(base_segment):
    """Parses the passed in string of characters and breaks it down based on its fields.

    Takes a string and slices it into its different constituent fields which are 
    stored in a dictionary.

        base_segment: A segment of the metro 2 file

        A dictionary that contains keys and values of the file fields.
    base_dict = {}
    base_dict["Record Descriptor Word"] = base_segment[0:4].strip(" ")
    base_dict["Processing Indicator"] = base_segment[4].strip(" ")
    base_dict["Time Stamp"] = base_segment[5:19].strip(" ")
    base_dict["Correction Indicator"] = base_segment[19].strip(" ")
    base_dict["Identification Number"] = base_segment[20:40].strip(" ")
    base_dict["Cycle Identifier"] = base_segment[40:42].strip(" ")
    base_dict["Consumer Account Number"] = base_segment[42:72].strip(" ")
    base_dict["Portfolio Type"] = base_segment[72].strip(" ")
    base_dict["Account Type"] = base_segment[73:75].strip(" ")
    base_dict["Date Opened"] = base_segment[75:83].strip(" ")
    base_dict["Credit Limit"] = base_segment[83:92].strip(" ")
    base_dict["Highest Credit or Original Loan Amount"] = base_segment[92:101].strip(" ")
    base_dict["Terms Duration"] = base_segment[101:104].strip(" ")
    base_dict["Terms Frequency"] = base_segment[104].strip(" ")
    base_dict["Scheduled Monthly Payment"] = base_segment[105:114].strip(" ")
    base_dict["Actual Payment Amount"] = base_segment[114:123].strip(" ")
    base_dict["Account Status"] = base_segment[123:125].strip(" ")
    base_dict["Payment Rating"] = base_segment[125].strip(" ")
    base_dict["Payment History Profile"] = base_segment[126:150].strip(" ")
    base_dict["Special Comment"] = base_segment[150:152].strip(" ")
    base_dict["Compliance Condition Code"] = base_segment[152:154].strip(" ")
    base_dict["Current Balance"] = base_segment[154:163].strip(" ")
    base_dict["Amount Past Due"] = base_segment[163:172].strip(" ")
    base_dict["Original Charge-off Amount"] = base_segment[172:181].strip(" ")
    base_dict["Billing Date"] = base_segment[181:189].strip(" ")
    base_dict["FCRA Compliance/Date of First Delinquency"] = base_segment[189:197].strip(" ")
    base_dict["Date Closed"] = base_segment[197:205].strip(" ")
    base_dict["Date of Last Payment"] = base_segment[205:213].strip(" ")
    base_dict["Reserved"] = base_segment[213:230].strip(" ")
    base_dict["Consumer Transaction Type"] = base_segment[230].strip(" ")
    base_dict["Surname"] = base_segment[231:256].strip(" ")
    base_dict["First Name"] = base_segment[256:276].strip(" ")
    base_dict["Middle Name"] = base_segment[276:296].strip(" ")
    base_dict["Generation Code"] = base_segment[296].strip(" ")
    base_dict["Social Security Number"] = base_segment[297:306].strip(" ")
    base_dict["Date of Birth"] = base_segment[306:314].strip(" ")
    base_dict["Telephone Number"] = base_segment[314:324].strip(" ")
    base_dict["ECOA Code"] = base_segment[324].strip(" ")
    base_dict["Consumer Information Indicator"] = base_segment[325:327].strip(" ")
    base_dict["Country Code"] = base_segment[327:329].strip(" ")
    base_dict["First Line of Address"] = base_segment[329:361].strip(" ")
    base_dict["Second Line of Address"] = base_segment[361:393].strip(" ")
    base_dict["City"] = base_segment[393:413].strip(" ")
    base_dict["State"] = base_segment[413:415].strip(" ")
    base_dict["Postal/Zip Code"] = base_segment[415:424].strip(" ")
    base_dict["Address Indicator"] = base_segment[424].strip(" ")
    base_dict["Residence Code"] = base_segment[425].strip(" ")
    return base_dict

def parse_file(file_line):
    """Parses the passed in string of characters in the Metro 2 format and breaks it down based on its fields in each segment.

    Takes a string and slices it into its different constituent fields which are 
    stored in a dictionary.

        file_line: The line from the Metro 2 file

        A dictionary that contains keys and values of the file fields.
    json_dict = {}
    header_dict = parse_header(file_line[0:426])
    base_dict = parse_base_segment(file_line[426:])
    json_dict["Header Record"] = header_dict
    json_dict["Base Segment"] = base_dict
    return json_dict

def restructure_data(in_file, out_file):
    """Parses an input file and writes a JSON file with the properly formatted lines.

    Takes the line from the Metro 2 file and creates a dictionary that contains
    a list of entries comprised of fields from each segment input file. Creates a JSON object 
    of data.
    file_objects_dict = {}
    with open(in_file,'r') as info_file:
        file_objects_dict = parse_file(info_file.readline())
    with open(out_file,'w') as out_file:
            json.dump(file_objects_dict, out_file, indent = 2)

I see a lot of repeated code. This line:

base_dict[key] = base_segment[A:B].strip(" ")

Could be put in only once, and embedded in a loop that takes combinations of key, A and B from some kind of dictionary.

Furthermore, the A:B pairs are a bit redundant as no fields overlap; so you would be fine with only encoding the lengths of the fields. This also ensures that no part of the base_segment is left un-interpreted.

So to sum it up: why not encode a list of key:length pairs and loop over that?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.