# Parsing files with county codes

I am running through a file and inserting elements one by one. The counties all contain specific county codes which are duplicated many times throughout the file. I am looking for a way to assign these codes to a specific county while ignoring duplicate county codes.

I have two versions I wrote below with runtimes:

getCodes()

def get_codes(data):
code_info = {}
for row in data:
county = row["county"]
code = int(float(row["code"]))
if code > 100000:
code = code/100
if county not in code_info:
code_info[county] = []
code_info[county].append(code)
for county in code_info:
code_info[county] = list(set(code_info[county]))
return code_info


get_codes2()

def get_codes2(data):
code_info = {}
for row in data:
county = row["county"]
code = int(float(row["code"]))
if code > 100000:
code = code/100

if county in code_info:
if not code in code_info[county]:
code_info[county].append(code)
else:
code_info[county] = []
code_info[county].append(code)
return code_info


county_data = csv.DictReader(open("county_file.txt"))
start = time.time()
county_codes = get_codes(county_data)
end = time.time()
print "run time: " + str(end-start)

start = time.time()
county_codes = get_codes2(county_data)
end = time.time()
print "run time: " + str(end-start)


Also, it's probably obvious from this, but county codes that are greater than 100000 can have trailing zeroes accidentally added, so I'm removing them by dividing by 100. As another note, the int(float()) conversion is intentional. Sometimes the county codes are values such as "27.7" and need to be converted to "27", other times they are just basic ints.

The runtimes on my system:

• get_codes: 9 seconds
• get_codes2: 14 seconds

How can I improve this further for better performance?

• Python 2.5 supports a defaultdict, which might save some time with the "if county in code_info" checks. – Handyman5 Feb 1 '12 at 5:52

I think you're pretty close to optimal here. I made a few tweaks, to avoid some conditionals by using sets instead. I don't actually know if it'll run faster; you'll need to benchmark it, and it likely depends on the breakdown of how many dupes per county there are.

def get_codes3(data):
from collections import defaultdict
codeinfo = defaultdict(set)
for row in data:
county = row["county"]
# remove float() if you can get away with it - are there
# actually values like '1.5' in the data?
code = int(float(row["code"]))
if code > 100000:
code = code/100


• You don't need float() unless the data is actually like "123.45" - it's not clear if that's the case
• sets can work in most places that lists work, so you might not need to convert to a list
• It might be worth it to write a script that does just the x = x/100 if x > 100000 part and writes that out to a new file
• Additionally, you could write code = code / 100 as code /= 100 if you're going for brevity. It doesn't do much for performance, but it's quicker to read. – Elmer Jan 26 '12 at 15:02