# Using Comprehensions to Handle a Large Dataset in Python 2.7

I have a Python 2.7.6 script parsing large files (~60MB to ~2GB) containing lines of the following format:

componentA componentB < floating point value >

My goal is to sum the floating point values for duplicate pairs of components and reflected pairs, reducing each pair to a single entry in the output file. For example, if I have the following three lines in the input file:

componentA componentB 1.5
componentB componentA 2.0
componentA componentB 0.5

The output file should combine them into one entry:

componentA componentB 4.0

Other notes:

1. The output also needs to be sorted by the lesser of each component pair.

2. I am not very concerned about the memory footprint of the routine because it will be run on a server with substantial amounts of RAM available. I am optimizing for speed and code maintainability.

The code I currently have accomplishes this by storing component names in nested dictionaries. The top-level dictionary maps component names to a second dictionary. The second-level dictionary maps the second half of each component pair to the floating-point value corresponding to the pair.

Example:

base_dict = {componentA: {componentB: 1.2,
componentC: 0.7}}


Output from this example would look like this:

componentA componentB 1.2
componentA componentC 0.7

Here is my code:

base_dict = {}

with open(input_file_name, 'r') as fi:
for line in fi:
component1, component2, value = line.split()
(min, max) = (component1, component2) if (component1 < component2) else (component2, component1)

if not min in base_dict:
base_dict[min] = {max:float(value)}

elif not max in base_dict[min]:
base_dict[min][max] = float(value)

else:
base_dict[min][max] += float(value)

out_list = []
for component1, nested_dict in base_dict.iteritems():
for component2, value in nested_dict.iteritems():
out_list.append("{0}\t{1}\t{2}\n".format(component1, component2, value))

with open(output_file_name), 'w') as fo:
fo.writelines(sorted(out_list))


In addition to a general code review for this snippet, there are two sections that I would like to convert to comprehensions if possible:

1. Constructing the nested dictionaries - I may be dealing with close to 10 million lines in the input file, and I would like to utilize implicit loops to do this.

2. Constructing the output list for sorting

If anyone has any completely different approaches to the problem of summing duplicate and reflected pairs, I'm all ears.

• If I understand correctly, you do not really need any additional data structures. While reading the list swap component names when necessary, then just sort the list. – vnp Jul 17 '14 at 17:53
• That won't sum the floating point values associated with each pair though. Any pair of component names may appear multiple times in the input file, and the order of the names may be reversed. There needs to be exactly one entry in the output file for each pair, with the sum of the FP values associated with each instance of that pair from the input file. – skrrgwasme Jul 17 '14 at 18:03
• I should've made myself more clear. Once the list is properly sorted, all entries which need to be added together are adjacent. Do I need to expand on how to complete summation? – vnp Jul 17 '14 at 18:09
• Nope! I understand now, and that's an excellent point. If you repost that suggestion in an answer I'll accept it. – skrrgwasme Jul 17 '14 at 18:11

Sort the list according to the component names. To achieve a correct order, either swap component names when necessary, or supply the custom comparator (which will account for (componentA, componentB) being the same thing as (componentB, componentA)).