Parsing files with county codes

Question

I am running through a file and inserting elements one by one. The counties all contain specific county codes which are duplicated many times throughout the file. I am looking for a way to assign these codes to a specific county while ignoring duplicate county codes.

I have two versions I wrote below with runtimes:

getCodes()

def get_codes(data):
    code_info = {}
    for row in data:
        county = row["county"]
        code = int(float(row["code"]))
        if code > 100000:
            code = code/100
        if county not in code_info:
            code_info[county] = []
        code_info[county].append(code)
    for county in code_info:
        code_info[county] = list(set(code_info[county]))
    return code_info

get_codes2()

def get_codes2(data):
    code_info = {}
    for row in data:
        county = row["county"]
        code = int(float(row["code"]))
        if code > 100000:
            code = code/100

        if county in code_info:
            if not code in code_info[county]:
                code_info[county].append(code)
        else:
            code_info[county] = []
            code_info[county].append(code)
    return code_info

county_data = csv.DictReader(open("county_file.txt"))
start = time.time()
county_codes = get_codes(county_data)
end = time.time()
print "run time: " + str(end-start)

county_data = csv.DictReader(open("county_file.txt"))
start = time.time()
county_codes = get_codes2(county_data)
end = time.time()
print "run time: " + str(end-start)

Also, it's probably obvious from this, but county codes that are greater than 100000 can have trailing zeroes accidentally added, so I'm removing them by dividing by 100. As another note, the int(float()) conversion is intentional. Sometimes the county codes are values such as "27.7" and need to be converted to "27", other times they are just basic ints.

The runtimes on my system:

• get_codes: 9 seconds
• get_codes2: 14 seconds

How can I improve this further for better performance?

Answer 1

I think you're pretty close to optimal here. I made a few tweaks, to avoid some conditionals by using sets instead. I don't actually know if it'll run faster; you'll need to benchmark it, and it likely depends on the breakdown of how many dupes per county there are.

def get_codes3(data):
    from collections import defaultdict
    codeinfo = defaultdict(set)
    for row in data:
        county = row["county"]
        # remove float() if you can get away with it - are there
        # actually values like '1.5' in the data?
        code = int(float(row["code"]))
        if code > 100000:
            code = code/100

        codeinfo[county].add(code)
    # if you can get away with sets instead of lists in your return
    # value, you're good
    return code_info
    # otherwise:
    # return dict([(county, list(info)) for county, info in code_info.iteritems()])

• You don't need float() unless the data is actually like "123.45" - it's not clear if that's the case
• sets can work in most places that lists work, so you might not need to convert to a list
• It might be worth it to write a script that does just the x = x/100 if x > 100000 part and writes that out to a new file