Transforming lists of products to boolean vectors

Question

Task

Inputs

Lists of products stored in a CSV file, where each line represents the shopping list of a single client:

Outputs

A CSV file:

• Header: products names sorted by number of appearances in the input file.
• For each line k we output boolean vectors where each of its elements indicates whether the i-th product was present on the line k in the input file or not.

Approach 1

from collections import defaultdict

with open('bigtest.csv', 'r') as inp:
    index = defaultdict(lambda: set())
    for line_number, line_str in enumerate(inp):
        products = line_str[:-1].split(',')
        for product in products:
            index[product].add(line_number)

product_names = index.iterkeys()
sorting_key = lambda name: len(index[name])

header = sorted(product_names, key=sorting_key, reverse=True)

def generate_table():
    for line in range(line_number + 1):
        yield (1 if line in index[product] else 0 for product in header)

with open('formal0.csv', 'w') as out:
    print >> out, ','.join(header)
    for vector in generate_table():
        print >> out, ','.join(str(val) for val in vector)

Approach 2

More declarative

from collections import defaultdict

with open('bigtest.csv', 'r') as inp:
    data = (line[:-1].split(',') for line in inp)

    index = defaultdict(lambda: set())
    for line_number, products in enumerate(data):
        for product in products:
            index[product].add(line_number)

header = sorted(
    index.iterkeys(),
    key=lambda p: len(index[p]),
    reverse=True
)

table = ((1 if line in index[product] else 0 for product in header)
         for line in range(line_number + 1))

with open('formal0.csv', 'w') as out:
    print >> out, ','.join(header) + '\n' + '\n'.join(','.join(map(str, vector))
                                                      for vector in table)

Concerns

• Readability
• Clean way to make nested generators?
• Am I abusing .join()?
• How much declarative style is too much?

Answer 1

1. Review

Just reviewing approach 2.

1. Even if you have a good reason to be using Python 2.7, it is worth noting that the Python developers plan to end support for this version in 2020 and so it is a good idea to get into the habit of writing code that will be easy to port to Python 3 should you have to take that step.

   In this case it would be easy for you to start the program with

   from __future__ import print_function
   

   and then use the print function instead of the print statement. (But we'll see later that we don't even need the print function since we can use the csv module instead.)

2. If a script has hard-coded filenames like 'bigtest.csv' then you have to edit it every time you want to run it on files with different names. The script would be more reusable if it took the filenames from the command line.

3. It looks as though the input is supposed to be in CSV format. But the input is parsed like this:

   data = (line[:-1].split(',') for line in inp)
   for line_number, products in enumerate(data):
   

   This doesn't take into account the full details of the CSV format: in particular, fields can be quoted, and quoted fields can contain commas. Python has a built-in csv module that handles the CSV format. This allows you to write the input loop like this:

   reader = csv.reader(inp)
   for line_number, products in enumerate(reader):
   

4. Instead of lambda:set(), you can just write set! Computer scientists call this η-conversion.

5. The variable name index does not clearly indicate what it contains. The value of this variable is a mapping from a product name to the set of line numbers on which that product appears, so I would call it something like product_lines.

6. The expression

   1 if line in index[product] else 0
   

   can be written more simply as

   int(line in index[product])
   

   since False converts to int as 0 and True as 1.

7. Using the csv module as discussed above, the output loop can be written like this:

   writerow = csv.writer(out).writerow
   writerow(header)
   for line in range(line_number + 1):
       writerow([int(line in index[product]) for product in header])
   

8. The rewritten code still looks up index[product] for every product on every row. These lookups can be avoided by constructing the header list so that it contains the sets of line numbers as well as the product names:

   header = sorted(index.items(), key=lambda p:len(p[1]), reverse=True)
   

   and then:

   writerow = csv.writer(out).writerow
   writerow([product for product, _ in header])
   for line in range(line_number + 1):
       writerow([int(line in lines) for _, lines in header])
   

2. Revised code

from collections import defaultdict
import csv
import sys

def main(in_filename='bigtest.csv', out_filename='formal0.csv'):
    """Read lists of products from the CSV file in_filename, convert them
    to Boolean vectors of products, and write the results to out_filename.

    """
    # Mapping from product to set of line numbers on which it appears.
    product_lines = defaultdict(set)

    with open(in_filename) as in_file:
        reader = csv.reader(in_file)
        for line_number, products in enumerate(reader):
            for product in products:
                product_lines[product].add(line_number)

    # List of pairs (products, lines) in reverse order by len(lines)
    header = sorted(product_lines.items(), key=lambda p:len(p[1]), reverse=True)

    with open(out_filename, 'w') as out_file:
        writerow = csv.writer(out_file).writerow
        writerow([product for product, _ in header])
        for line in range(line_number + 1):
            writerow([int(line in lines) for _, lines in header])

if __name__ == '__main__':
    main(*sys.argv[1:])