Compare result files from two different runs the smart way?

Question

The program counts attributes of various (geo-)objects and prints them to different files.

One of the options compares two different datasets with each other.

parser.add_option("-c", "--compare datasets", dest="compare",
                  help="If you want to campare two datasets (from different years for example) set the c-flag to 1. Run both datasets\
                    one after the other. After the second run you will find a file named COMPARE_* in the output folder. The default\
                  behaviour is no comaprison.") 

I am concerned about the behavior of my current implementation. The basic idea is to pickle a list of file names when the -c flag is set to 1. When the program is run for the second time with the -c flag it will load all file names needed for the comparison from the pickle file.

if options.compare == '1':
    # check if there is a pickle file
    p = glob.glob('%s/*.pikl' % data_output)
    if len(p) > 1:
        raise Exception('There are two pickle files. Comparison not possible.')
    elif len(p) == 1:
        pfiles_list = pickle.load(open(p[0],'rb'))
        if len(pfiles_list) == len(files_to_compare):
            for i,(f1,f2) in enumerate(zip(pfiles_list,files_to_compare)):
                print 'comparing files...'
                if i == 0: 
                    csv_cmp_lazy(f1,f2, compare_file)
                else:
                    csv_cmp_lazy(f1,f2, compare_file, append_file = True)
            info_str = 'Deleting pickle file %s' % os.path.basename(p[0])
            print info_str
            logging.info(info_str)
            os.remove(p[0])
        else:
            raise ValueError('len of files current and last run are not equal: Number of file current run/last run: %s/%s' % (len(files_to_compare),len(pfiles_list))
    else:
        pickle.dump(files_to_compare, open(pikl_file,'wb'))

This implementation relies on a few assumptions:

• that the pickle file won't be moved away from data_output
• that os.remove(p[0]) will be successful

Not very secure, right? Besides, imagine the user finds an error in his data and wants to do the second run again: the comparison will fail because the pickle file won't be there anymore. This can be very annoying as the program may take longer than one hour to terminate (depending on input options and dataset).

Any suggestions on how to do this in a smarter way?

Answer 1

Simplify all the things! You and your users will benefit from simplicity.

Option parsing

If the --compare option is either True or False, make it a boolean flag with False as the default value. Your users won't have to remember if the --compare argument is 1, true or yes, they will simply write --compare.

You should also consider moving from optparse to argparse: optparse is deprecated since Python 2.7 and Python 3.2 and could be removed in any future release.

About your assumptions

• If the pickle file is moved away from data_output, then the users will simply see that the file is missing. You could catch the resulting exception to ask them to first run the program without comparison.
• If the removing fails, then something is really wrong anyway.

I don't think resolving those specific issues is the way to go, and it should be easier for your users to provide this comparison feature differently.

Comparison logic

Your current solution is mostly a hack. It looks like you only asked yourself: what is the simplest way to implement this? The question should be different: what is the simplest way to use this?

Say I want to compare 2010 and 2011. I first run your program with 2010, then with 2011 plus comparison. What if I actually made a typo and typed 2010 instead of 2009? What if one of the options for 2011 was wrong? I have to start all over again. Very frustrating since the processing is slow.

I don't know your program well enough to be sure, but the simplest way could be to provide a --dump option, so that users would dump the first dataset (slow), dump the second dataset (slow), then compare the two (fast!).

It would also be even clearer to provide two binaries, one for processing the datasets, and another one for comparison: this would reduce confusion.

Users are now in control and understand what happens and how the comparison works. Fixing any issue does not require to run processing for the two datasets. User won't delete the dumped files, and if they did, that would be their fault since you exposed those files to them.