I have written a simple code to extract different time steps from a big text file. It works for small text files, but not for bigger text files as input.
n=0
y=[]
with open ("DEPTH.dat", "r") as input:
for x in input:
x = x.strip()
if x.startswith('SCALAR'):
n += 1
with open("TS_"+ str(n) + ".dat", "a") as subfile:
subfile.write("{0}\n".format(x))
The input file is like the following:
SCALAR
ND 5
ST 0
TS 10.00
0.0022
0.0022
0.0022
0.0020
0.4881
SCALAR
ND 5
ST 0
TS 100.00
0.1
0.2
0.12
0.32
0.15
SCALAR
ND 5
ST 0
TS 200.00
0.34
0.25
1.1
1.0020
1.4381
In this example file, the number of nodes ND=5
, so it works very well. But when I have 1,000,000 nodes, it does not work. It does not deliver any result even after 20 days. So I know I have to write the program as a function and return the result and not to upload the whole data into memory. I have no I idea how to do that.
csplit
utility. \$\endgroup\$ – hjpotter92 Nov 17 '17 at 17:51