The code adds up numbers from two different text files. It opens an input file 'DEPTH.dat' and reads the numbers after some headers as follows:
SCALAR ND 3 ST 0 TS 1000 1.0 1.0 1.0 SCALAR ND 3 ST 0 TS 2000 3.3 3.4 3.5 SCALAR ND 3 ST 0 TS 3000 1.7 1.8 1.9
It then adds to them numbers which have been read from a smaller text file 'ELEVATION.dat' which looks as follows:
SCALAR ND 3 ST 0 TS 0 10.0 10.0 10.0
The result is written in a text file 'output.txt' like this:
SCALAR ND 3 ST 0 TS 1000 11.0 11.0 11.0 SCALAR ND 3 ST 0 TS 2000 13.3 13.4 13.5 SCALAR ND 3 ST 0 TS 3000 11.7 11.8 11.9
Here's the code in question. As I have learned from 200_success in my last code review that I should write the code using functions, I tried to write this new code using some functions.
For a small example, the code works properly but for bigger input data, it does not work. (For a 350 MB file with 30 million lines, there is no response after two hours.) How can the code be improved?
from itertools import zip_longest def grouper(iterable, n, padvalue=None): return zip_longest(*[iter(iterable)]*n, fillvalue=padvalue) def writing_ND(f1): for line1 in f1: if line1.startswith('ND'): ND = float(line1.split()[-1]) return ND def writing_TS(f): with open(f, 'r') as f: for line1 in f: if line1.startswith('TS'): x = float(line1.split()[-1]) TS.append(x) return TS TS =  ND =  n = 0 add_numbers =  TS = writing_TS('DEPTH.dat') with open("ELEVATION_LHP.dat") as f, open("DEPTH.dat") as f1,\ open('output.txt', 'w') as out: ND = writing_ND(f) n = int(ND)+4 f.seek(0) for lines in grouper(f, int(n)): for item in lines[4:]: add_numbers.append(float(item)) i = 0 for l in grouper(f1, n): data_numbers =  for line in l[4:]: data_numbers.append(float(line.split()[-1].strip())) result_numbers = [x + y for x, y in zip(data_numbers, add_numbers)] del data_numbers out.write('SCALAR\nND %d\nST 0\nTS %0.2f\n' % (ND, TS[i])) i += 1 for item in result_numbers: out.write('%s\n' % item)