Here's the exercise in brief:
Consider the following file: Code:
before.csv
A; ; B; B; A; H; C; ; D; D; C; G; E; D; F; F; E; H; G; D; ; H; G; ;
And a modified version of the file:
after.csv
A; ; B; B; A; H; C; ; D; D; ; G; E; D; F; F; E; H; G; D; ; K; ; E;
The first field of the CSV is a unique identifier of each line. The exercise consists of detecting the changes applied to the file, by comparing before and after.
There are 3 types of changes you should detect:
- ADDED (line is present in after.csv but not in before.csv)
- REMOVED (line is present in before.csv but not in after.csv)
- MODIFIED (line is present in both, but second and/or third field are modified)
In my example, there are three modifications:
- ADDED line (K)
- REMOVED line (H)
- MODIFIED line (D)
And my code:
import collections
import csv
import sys
class P_CSV(dict):
'''A P_CSV is a dict representation of the csv file:
{"id": dict(csvfile)} '''
fieldnames = ["id", "col2", "col3"]
def __init__(self, input):
map(self.readline, csv.DictReader(input, self.fieldnames, delimiter=";",\
skipinitialspace=True))
def readline(self, line):
self[line["id"]] = line
def get_elem(self, name):
for i in self:
if i == name:
return self[i]
class Change:
''' a Change element will be instanciated
each time a difference is found'''.
def __init__(self, *args):
self.args=args
def echo(self):
print "\t".join(self.args)
class P_Comparator(collections.Counter):
'''This class holds 2 P_CSV objects and counts
the number of occurrence of each line.'''
def __init__(self, in_pcsv, out_pcsv):
self.change_list = []
self.in_pcsv = in_pcsv
self.out_pcsv = out_pcsv
self.readfile(in_pcsv, 1)
self.readfile(out_pcsv, -1)
def readfile(self, file, factor):
for key in file:
self[key] += factor
def scan(self):
for i in self:
if self[i] == -1:
self.change_list.append(Change("ADD", i))
elif self[i] == 1:
self.change_list.append(Change("DELETE", i))
else: # element exists in two files. Check if modified
j = J_Comparator(self.in_pcsv.get_elem(i), self.out_pcsv.get_elem(i))
if len(j) > 0:
self.change_list += j
class J_Comparator(list):
'''This class compares the attributes of two lines and return a list
of Changes object for every difference'''
def __init__(self, indict, outdict):
for i in indict:
if indict[i] != outdict[i]:
self.append(Change("MODIFY", indict["id"], i, indict[i], "BECOMES", outdict[i]))
if len(self) == 0:
self = None
#Main
p = P_Comparator(P_CSV(open(sys.argv[1], "rb")), P_CSV(open(sys.argv[2], "rb")))
p.scan()
print "{} changes".format(len(p.change_list))
[c.echo() for c in p.change_list]
In real life, the code is supposed to compare two very large files (>6500 lines) with much more fields (>10).
How can I improve, both the style of my programming as well as the performance of the script? For the record I'm using Python2.7
id
s? Because if you can I would suggest a different approach, i.e. parsing the two files side by side. \$\endgroup\$Change
class in particular will have multiple methods but that's beyond the scope of this code review. (I'm assuming you meant the Unixdiff
tool. If you meant some Python utility then please show me some link). Maybe you reckon I rundiff
on both files and pipe the output to my script? \$\endgroup\$