I have code that backup files to remote servers.
There are 2 pandas data frames:
storedfiles: that points to the files already stored in previous runs
filelist: a list of candidates to be stored in one of the depots (comes from a config file).
Filelist structure has the following columns:
In : filelist Out: Name Objects Class Subclass Creator \ 0 /backups/dir001/test00 400 waves time john 1 /backups/dir002/test00 400 waves time john Created Datahost Res Total Size 0 12-Feb-15 10:10:59 NaN D 1609728 1 04-Jan-15 14:40:38 NaN D 1609728
To evaluate the storage I check the version on
filelist against the version on
storedfiles. If the version on file list is newer, it will be stored in some depot (the old version will be deleted). (edit) But to get the "version" I "need" to fetch it from a XML structure. Using an class to read the XML and process the data on it.
# 'a' is a instance of a internal XML processing class # This fetchs the XML structure related to the file 'f' from filelist. # Its a type(xmlfile) is 'str' xmlfile = a.get_xml(f) # This is a object to the structured data loaded on xmlfile. # type(xmldata) is aXML xmldata = a.load(xmlfile) # With that I can fetch informations from my file as: # That returns an datetime.datetime structure type(xmldata.get_data_created()) <type 'datetime.datetime'> # As I do at: version = int(xmldata.get_data_created().strftime("%s"))
To control versions I get the Created field (that stores a datetime) and convert it to a Unix timestamp.
My main concern is that I use too many
ifs. I feel the code is ugly and I wish have hints to do a neater and prettier code.
# Stored files is in fact a local csv file with this format # It holds all file names that has being processed and stored in "depot" server storedfiles= """ Filename,Version,Depot foo.txt,12342412,server1 bar.mov,14144862,server2 tmp.log,13327702,server1 """ cfcolumns=['Filename','Version','Depot'] # stdf (stored files dataframe) # Is the dataframe that maps the local csv file with processed files sfdf= panda.Dataframe(storedfiles,...) # filelist is a pandas dataframe that holds a list of files to be archived filelist = pandas.DataFrame(...) # If there is some candidate to archive if len(filelist.index)!=0: for f in filelist.filename: # To fetch the attributes I need to process some XML data... xmlfile = a.get_xml(f) xmldata = a.load(xmlfile) # Use data unix timestamp as version version = int(xmldata.get_data_created().strftime("%s")) # Get the panda series that match the 'filename' with f in loop # That is, look if the filename was already backuped. ssts = sfdf.loc[sfdf['filename'] == f] # Retrieves the "version" from panda series. # In this case I can' t call sstv.item() on an empty series # Should "try/except" be used instead this? Probably yes. # If is the best solution, I don't know. sstv = ssts['Version'] if sstv.empty: storedversion=0 else: storedversion = sstv.item() # Verify if version found is newer than stored on control file if version > storedversion: # get_best_depot() returns a single hostname that can hold the file for backup dstdepot = get_best_depot() if ssts.empty: # data Serie is empty. Add a new entry (to be later wrote on local filesystem) # Stores copy the f file to the correct location on dstdepot if store(f,dstdepot): # If successfuly stored, add to the stored files dataframe sfdf=sfdf.append(pandas.DataFrame([[ f, version, dstdepot ]], columns=cfcolumns),ignore_index=True) else: # Fails and exit.. Needs to correctly handle this sys.exit(1) else: # File exists but in older version # Get actual depot for the file from "storedfiles" dataFrame (sfdf) dstdepot = get_depot(f,sfdf) # Send the file to the depot server. if store(f,dstdepot): dstidx = ssts.index[-1] sfdf.loc[dstidx]=[ f,version, dstdepot ]