# function responsible for parsing 2 csv txt files, comparing them and writing csv txt files as output

  def Ana_exc():

global global_dic, missing_key_w, out_put_defult, ffd_ana_exception_path_w, ana_exc_input_path, ana_5min_input_path, min_flag

count_path1 = 0
count_path2 = 0

meow2 = ''
ana_exc_time = ''

ana_ffm_track = []
ana_exc_missing = []
time_track = []

with open(ffm_all_w + 'ana_ffm.txt', 'w') as ana_ffm, open(missing_key_w + 'ana_missint_keys.txt', 'w') as ana_missing_keys:
for i in range(len(ana_5min_input_path)):
if not count_path1 > len(ana_5min_input_path):
with open(ana_5min_input_path[count_path1], 'r') as ana_5min:
count_path1 = count_path1 + 1
for x in range(len(ana_exc_input_path)):
if not count_path2 > len(ana_exc_input_path):
with open(ana_exc_input_path[count_path2], 'r') as ana_exc, open(ffd_ana_exception_path_w + 'ana_ffd.txt' + str(count_path2), 'w') as ffd_ana:

count_path2 = count_path2 + 1

# per 2 files exist a metadata file, this will write the header for the txt file
ana_ffm.write('header' + ',' + '1' + '\n')

# in charge of reading and processing file1 (with random time stamp)
for line in ana_exc:
min_flag = True
# spliting the fields of csv txt file
col = line.split(",")

# to ignore random rows that contain random numbers
if str(line[2]).startswith('/'):
# making a unique key to allow compareson between files
ana_exc_key = (col[1] + '|' + col[2] + '|' + col[3] + '|' + col[4])
# extract time stamp from field
ana_exc_time = col[0]
# match with a cross refrence dictionary to ensure the point is acceptable
if ana_exc_key in global_dic:
# transfer human readble time to unix time stamp
meow = datetime.datetime.strptime(ana_exc_time, "%d/%m/%Y %H:%M:%S")  # change str time to date/time obj
unix_timestamp = calendar.timegm(meow.timetuple())  # do the conversion to unix stamp
time_ms1 = unix_timestamp * 1000
time_exc = time_ms1

# write metadata file, after chacking the point has not been written before
if ana_exc_key not in ana_ffm_track:
ana_ffm.write('point' + ',' + str(global_dic[ana_exc_key]['cpKey']) + ',' + str(global_dic[ana_exc_key]['SCADA Key']) + ',' + str(global_dic[ana_exc_key]['Point Name']) + ',' + 'analog' + ',' + ',' + '1' + '\n')
ana_ffm_track.append(ana_exc_key)
# if time stamp of file 1 is same as the time stamp of file 2. for the points fitting this critiria process from file 2 insted of file 1
if meow.minute % 5 or meow.minute == 00 and time_ms1 not in time_track:
min_flag = False
for line2 in ana_5min:

col2 = line2.split(",")

if str(line2[2]).startswith('/'):
ana_5min_key = (col2[1] + '|' + col2[2] + '|' + col2[3] + '|' + col2[4])
ana_5min_time = col2[0]

if ana_5min_key in global_dic:

meow2 = datetime.datetime.strptime(ana_5min_time, "%d/%m/%Y %H:%M:%S")  # change str time to date/time obj
unix_timestamp = calendar.timegm(meow2.timetuple())  # do the conversion to unix stamp
time_ms = unix_timestamp * 1000

time_ana = time_ms

ffd_ana.write('header' + ',' + str(time_ms) + ',' + '1' + '\n')

ffd_ana.write('value' + ',' + str(global_dic[ana_5min_key]['cpKey']) + ',' + str(global_dic[ana_5min_key]['SCADA Key']) + ',' + str(col2[6]) + ',' + str(time_ana) + ',' + str(time_ana) + ',' + '0' + ',' + '0' + ',' + '0' + '\n')

if ana_5min_key not in ana_ffm_track:
ana_ffm.write('point' + ',' + str(global_dic[ana_5min_key]['cpKey']) + ',' + str(global_dic[ana_5min_key]['SCADA Key']) + ',' + str(global_dic[ana_5min_key]['Point Name']) + ',' + 'analog' + ',' + ',' + '1' + '\n')
ana_ffm_track.append(ana_5min_key)

else:
if ana_5min_key not in ana_exc_missing:
ana_missing_keys.write(ana_5min_key + '\n')
ana_exc_missing.append(ana_5min_key)

if meow.hour != meow2.hour or meow.minute != meow2.minute or meow.second != meow2.second:
break
time_track.append(time_ms1)
ffd_ana.write('header' + ',' + str(time_exc) + ',' + '1' + '\n')
if time_ms1 not in time_track:
ffd_ana.write('value' + ',' + str(global_dic[ana_exc_key]['cpKey']) + ',' + str(global_dic[ana_exc_key]['SCADA Key']) + ',' + str(col[6]) + ',' + str(time_exc) + ',' + str(time_exc) + ',' + '0' + ',' + '0' + ',' + '0' + '\n')
else:
if ana_exc_key not in ana_exc_missing:
ana_missing_keys.write(ana_exc_key + '\n')
ana_exc_missing.append(ana_exc_key)
else:
break
else:
break
return None


I need help cleaning up the function posted above, this function will open multiple txt files, read and extract some information, than write back to multiple txt files. the code is too dirty and at times very slow.

1. the function will be handling files with millions of lines
2. im new to coding
3. txt files are comma delimited
4. specially with the section of the code that is handling file opening and closing
5. the code does work, just needs clean up and improvements

summery of the code: It opens files to read from and write to. It looks at each line of one file, separate them to columns. the fist column in both files are time stamps, first file the time stamps are random, second file contains lines with time-stamps that are increments of 5 minutes. whenever the fist file has time stamps that match the time stamp of the second file the line from second file is processed, otherwise the line from first file is processed. Lines from both file1 and file2 will only be processed if they have a match to the global_dict (dictionary) otherwise they will be written to missing files. Also a key is constructed from multiple field of each line to serve as a unique identifier

file ex:

file 1: (csv format)(time-stamps random)
time-stamp0,field1.1,field1.2,field1.3,field1.4,field1.5,...
3
time-stamp1,field2.1,field2.2,field2.3,field2.4,field2.5,...
5
time-stampn,fieldn.1,fieldn.2,fieldn.3,fieldn.4,fieldn.5,...
12
.......

file 2: (csv format)(time-stamps 5 minute increments)
time-stamp0,field1.1,field1.2,field1.3,field1.4,field1.5,...
1
time-stamp1,field2.1,field2.2,field2.3,field2.4,field2.5,...
5,           6
time-stampn,fieldn.1,fieldn.2,fieldn.3,fieldn.4,fieldn.5,...
.......


• Break this into multiple functions that each accomplish one task and one task only and name the functions to explain exactly what they do. That will help reduce some of the indenting and make it easier to read in the future. If you find yourself indenting more than a couple of times, you should ask how you can refactor the ifs or for loops into functions.
• Use Python's built-in csv module. As you can see in this StackOverflow question, it will help by reading files one line at a time. Hopefully one line at a time is ok for comparing the two files (it's not clear from the code above).
• Always try to look for built-in or 3rd party modules before you write your own code to do the same. Once you get more familiar with coding then you can fall into the trap of thinking everyone else's code is dumb and you need to reinvent the wheel. For now, find existing wheels that have been tested and cleaned up and use them. To that point, there is no reason not to start tasks with searches like python diff two csv files to see what's out there. The top result for that search is a package called csvdiff. I think I might scrap the code you have and use that. If it works, take a look the code itself and see how they accomplished the task. It may be a bit hard to follow at times because the package has to handle a bunch of different issues but you probably will learn something.
• In Python (and languages like it), looping over a number instead of the collection itself (like you're doing with all these range functions) is a "code smell". You usually don't need the number and it's a performance drain. If you do need a number inside the loop, consider enumerate(my_collection) which will give you a counter and a collection element as you go.
• "I have made request to add PANDA and CSV" This is confusing to me: Pandas is a 3rd-party module, but csv is a Python built-in which should be available to you with import csv. As for the other restrictions, there's nothing stopping you from making one really big file that includes any code you like (though obviously trying to cram all of Pandas in would be silly, you probably can steal what you need from csvdiff and jam it into your file). – Tom Aug 2 '16 at 16:25