The code adds up numbers from two different text files. It opens an input file 'DEPTH.dat' and reads the numbers after some headers as follows:
SCALAR ND 3 ST 0 TS 1000 1.0 1.0 1.0 SCALAR ND 3 ST 0 TS 2000 3.3 3.4 3.5 SCALAR ND 3 ST 0 TS 3000 1.7 1.8 1.9
It then adds to them numbers which have been read from a smaller text file 'ELEVATION.dat' which looks as follows:
SCALAR ND 3 ST 0 TS 0 10.0 10.0 10.0
The result is written in a text file 'output.txt' like this:
SCALAR ND 3 ST 0 TS 1000 11.0 11.0 11.0 SCALAR ND 3 ST 0 TS 2000 13.3 13.4 13.5 SCALAR ND 3 ST 0 TS 3000 11.7 11.8 11.9
Here's the code in question. As I have learned from 200_success in my last code review that I should write the code using functions, I tried to write this new code using some functions.
For a small example, the code works properly but for bigger input data, it does not work. (For a 350 MB file with 30 million lines, there is no response after two hours.) How can the code be improved?
from itertools import zip_longest
def grouper(iterable, n, padvalue=None):
return zip_longest(*[iter(iterable)]*n, fillvalue=padvalue)
def writing_ND(f1):
for line1 in f1:
if line1.startswith('ND'):
ND = float(line1.split()[-1])
return ND
def writing_TS(f):
with open(f, 'r') as f:
for line1 in f:
if line1.startswith('TS'):
x = float(line1.split()[-1])
TS.append(x)
return TS
TS = []
ND = []
n = 0
add_numbers = []
TS = writing_TS('DEPTH.dat')
with open("ELEVATION_LHP.dat") as f, open("DEPTH.dat") as f1,\
open('output.txt', 'w') as out:
ND = writing_ND(f)
n = int(ND)+4
f.seek(0)
for lines in grouper(f, int(n)):
for item in lines[4:]:
add_numbers.append(float(item))
i = 0
for l in grouper(f1, n):
data_numbers = []
for line in l[4:]:
data_numbers.append(float(line.split()[-1].strip()))
result_numbers = [x + y for x, y in zip(data_numbers, add_numbers)]
del data_numbers
out.write('SCALAR\nND %d\nST 0\nTS %0.2f\n' % (ND, TS[i]))
i += 1
for item in result_numbers:
out.write('%s\n' % item)