The program counts attributes of various (geo-)objects and prints them to different files.
One of the options compares two different datasets with each other.
parser.add_option("-c", "--compare datasets", dest="compare",
help="If you want to campare two datasets (from different years for example) set the c-flag to 1. Run both datasets\
one after the other. After the second run you will find a file named COMPARE_* in the output folder. The default\
behaviour is no comaprison.")
I am concerned about the behavior of my current implementation. The basic idea is to pickle a list of file names when the -c flag is set to 1. When the program is run for the second time with the -c flag it will load all file names needed for the comparison from the pickle file.
if options.compare == '1':
# check if there is a pickle file
p = glob.glob('%s/*.pikl' % data_output)
if len(p) > 1:
raise Exception('There are two pickle files. Comparison not possible.')
elif len(p) == 1:
pfiles_list = pickle.load(open(p[0],'rb'))
if len(pfiles_list) == len(files_to_compare):
for i,(f1,f2) in enumerate(zip(pfiles_list,files_to_compare)):
print 'comparing files...'
if i == 0:
csv_cmp_lazy(f1,f2, compare_file)
else:
csv_cmp_lazy(f1,f2, compare_file, append_file = True)
info_str = 'Deleting pickle file %s' % os.path.basename(p[0])
print info_str
logging.info(info_str)
os.remove(p[0])
else:
raise ValueError('len of files current and last run are not equal: Number of file current run/last run: %s/%s' % (len(files_to_compare),len(pfiles_list))
else:
pickle.dump(files_to_compare, open(pikl_file,'wb'))
This implementation relies on a few assumptions:
- that the pickle file won't be moved away from
data_output
- that
os.remove(p[0])
will be successful
Not very secure, right? Besides, imagine the user finds an error in his data and wants to do the second run again: the comparison will fail because the pickle file won't be there anymore. This can be very annoying as the program may take longer than one hour to terminate (depending on input options and dataset).
Any suggestions on how to do this in a smarter way?