I have a text file composed of several subsequent tables. I need to get certain values from certain tables and save them in an output file. Every table has a header which contains a string that can be used to find specific tables. The size of these text files can vary from tenths of MB to some GB. I have written the following script to do the job:
string = 'str' index = 20 n = 2 in_file = open('file.txt') out_file = open("out.txt", 'w') current_line = 0 for i in range(-index,index+1): for j in range(-index,index+1): for line in in_file: if string in line: En = line.split().pop(4) for line in in_file: current_line += 1 if current_line == 2*(n+1)+2: x = line.split().pop(10) elif current_line == 3*(n+1)+2: y = line.split().pop(10) elif current_line == 4*(n+1)+2: z = line.split().pop(10) current_line = 0 break print i, j, En, x, y, z data = "%d %d %s %s %s %s\n" % (i,j,En,x,y,z) out_file.write(data) break in_file.close() out_file.close()
The script reads the file line by line searching for the specified string ('str' in this example). When found, it then extracts a value from the line containing the string and continue reading the lines that form the data table itself. Since all the tables in the file have the same number of lines and columns, I've used the variable current_line to keep track of which line is read and to specify which line contains the data I need. The first two for-loops are just there to generate a pair of indexes that I need to be printed in the output file (in this case they are between -20 and 20).
The script works fine. But since I've been learning python by myself for about one month, and the files I have to handle can be very big, I'm asking for advices on how to make the script more efficient, and overall, better.
Also, since the tables are regular, I can know beforehand which are the lines that contain the values I need. So I was wondering, instead of reading all the lines in the file, is it possible to specify which lines have to be read and then jump directly between them?
Here's a sample input file. I've included just some tables so you can have an idea how it's organized. This file is composed by two blocks with three tables each. In this sample file, the string "table #" is what is used to find the data to be extracted.
And here's a sample output file. Keep in mind that these two files are not equivalent! This output was created by my script using an input file containing 1681 blocks of 16 tables. Each table had 13 lines just as in the sample input file.