After profiling my current script I've found that nearly 100% of the run time (and I'm not surprised) comes from the following function:
def rem_asc_by_pos(xmlFile, fileOut,removeIndices): fileIn = xmlFile.replace('.xml', '.asc') with open(fileIn, 'r') as fin: with open(fileOut, 'w') as fout: #remove indices from each line and write to new file if len(removeIndices) > 0: for line in fin: lineList = list(line) checkLine = '' newLine =  for ind,char in enumerate(lineList): if ind not in removeIndices: newLine.append(char) lineOut = ''.join(newLine) fout.write(lineOut)
The files themselves can have anywhere from a 15 thousand lines to 2.5 million at the most, each line is usually from 300-600 characters.
I was thinking that the write operation may be causing an increase in run-time so I could write multiple lines at a time but wouldn't know a good way to determine when it should write the lines (so as to not run out of memory).
Is there anything obvious that I might be able to change to decrease the overall run-time? To go through all the files currently it takes about 20 hours and any decrease from that is appreciated!
edit: Adding an example of input and expected output!
Input: I have a bunch of data files that look similar to this:
511111 12 1.000000 1.000000 1.000000 .750000 511112 12 1.000000 .666667 .500000 .555556
The removeIndicies list contains the indicies to remove from each line, for this example we want to remove the first three and indicies 19-28.
111 12 1.000000 1.000000 .750000 112 12 1.000000 .500000 .555556