So here is what I came up with, now that I had access to a computer:
raw_data.csv:
20110117559515,1,10,faa
20110117559515,2,20,bar
20110117559515,3,30,baz
20110117559572,4,40,fii
20110117559572,5,50,bir
20110117559572,6,60,biz
20110117559574,7,70,foo
20110117559587,8,80,bor
20110117559588,9,90,boz
code (written in several steps for readability):
import csv
from collections import defaultdict
if __name__ == '__main__':
rows = [row for row in csv.reader(open('raw_data.csv', 'r'), delimiter=',')]
cols = zip(*rows)
unik = set(cols[0])
positions = [cols[0].index(x) for x in unik]
results = [rows[p] for p in positions]
indexes = [rows.index(result) for result in results]
indexed = defaultdict(list)
for result in results:
indexed[rows.index(result)] = result
print 'positions:'
pprint(positions)
print '\nresults:'
pprint(results)
print '\nindexes:'
pprint(indexes)
print '\nindexed:'
pprint(dict(indexed))
Output (Python counts from 0, not 1):
positions:
[3, 7, 0, 6, 8]
results:
[['20110117559572', '4', '40', 'fii'],
['20110117559587', '8', '80', 'bor'],
['20110117559515', '1', '10', 'faa'],
['20110117559574', '7', '70', 'foo'],
['20110117559588', '9', '90', 'boz']]
indexes:
[3, 7, 0, 6, 8]
indexed:
{0: ['20110117559515', '1', '10', 'faa'],
3: ['20110117559572', '4', '40', 'fii'],
6: ['20110117559574', '7', '70', 'foo'],
7: ['20110117559587', '8', '80', 'bor'],
8: ['20110117559588', '9', '90', 'boz']}
Simplified code:
import csv
from collections import defaultdict
from pprint import pprint
if __name__ == '__main__':
rows = [row for row in csv.reader(open('raw_data.csv', 'r'), delimiter=',')]
cols = zip(*rows)
unik = set(cols[0])
indexed = defaultdict(list)
for x in unik:
i = cols[0].index(x)
indexed[i] = rows[i]
print 'indexed:'
pprint(dict(indexed))
Output:
indexed:
{0: ['20110117559515', '1', '10', 'faa'],
3: ['20110117559572', '4', '40', 'fii'],
6: ['20110117559574', '7', '70', 'foo'],
7: ['20110117559587', '8', '80', 'bor'],
8: ['20110117559588', '9', '90', 'boz']}
Second method:
rows = [row for row in csv.reader(open('raw_data.csv', 'r'), delimiter=',')]
cols = zip(*rows)
unik = set(cols[0])
indexed = defaultdict(list)
for x in unik:
i = next(index for index,fid in enumerate(cols[0]) if fid == x)
indexed[i] = rows[i]
Is this better in terms of memory usage or performance?