# Importing of big files:right approach?

I want to import in python some ascii file ( from tecplot, software for cfd post processing). Rules for those files are (at least, for those that I need to import): -The file is divided in several section -Each section has two lines as header like:

VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen"
ZONE T="Window(s) : E_W_Block0002_ALL",  I=29,  J=17,  K=25, F=BLOCK

• Each section has a set of variable given by the first line. When a section ends, a new section starts with two similar lines.

• For each variable there are I*J*K values.

• Each variable is a continous block of values.
• There are a fixed number of values per row (6).
• When a variable ends,the next one start in a new line.
• Variables are "IJK ordered data".The I-index varies the fastest; the J-index the next fastest; the K-index the slowest. The I-index should be the inner loop, the K-index shoould be the outer loop, and the J-index the loop in between.

Here an example of data:

VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen"
ZONE T="Window(s) : E_W_Block0002_ALL",  I=29,  J=17,  K=25, F=BLOCK
-3.9999999E+00 -3.3327306E+00 -2.7760824E+00 -2.3117116E+00 -1.9243209E+00 -1.6011492E+00
[...]
0.0000000E+00 #fin first variable
-4.3532482E-02 -4.3584235E-02 -4.3627592E-02 -4.3663762E-02 -4.3693815E-02 -4.3718831E-02 #second variable, 'y'
[...]
1.0738781E-01 #end of second variable
[...]
[...]
VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen" #next zone
ZONE T="Window(s) : E_W_Block0003_ALL",  I=17,  J=17,  K=25, F=BLOCK


I am quite new at python and I have written a code to import the data to a dictionary, writing the variables as 3d numpy.array . Those files could be very big, (up to Gb). How can I make this code faster?(or more generally, how can I import such files as fast as possible)?.

import numpy as *
from numpy.linalg import norm
import re

def vectorr(I, J, K):
arr = empty((I*J*K, 3), int)
arr[:,0] = tile(arange(I), J*K)
arr[:,1] = tile(repeat(arange(J), I), K)
arr[:,2] = repeat(arange(K), I*J)
return arr

def all_data(pathfilename,NumberCol = 6):
with open(pathfilename) as a:
Count = 0
data = dict()
leng = len(filelist)
Countzone = 0
while Count < leng:
strVARIABLES = re.findall('VARIABLES', filelist[Count])
variables = re.findall(r'"(.*?)"',  filelist[Count])
Countzone = Countzone+1
data[Countzone] = {key:[] for key in variables}
Count = Count+1
strI = re.findall('I=....', filelist[Count])
strI = re.findall(r'\d+', strI[0])
I = int(strI[0])
strJ = re.findall('J=....', filelist[Count])
strJ = re.findall(r'\d+', strJ[0])
J = int(strJ[0])
strK = re.findall('K=....', filelist[Count])
strK = re.findall(r'\d+', strK[0])
K = int(strK[0])
data[Countzone]['indmax'] = array([I, J, K])
pr = prod(data[Countzone]['indmax'])
lin = pr // NumberCol
if pr % NumberCol != 0 :
lin = lin+1
vect = vectorr(I, J, K)
for key in variables:
init = zeros((I, J, K))
for ii in range(0, lin):
Count = Count+1
temp = [ float(x.replace('D', 'E')) for  x in filelist[Count].split() ]
for iii in range(0, len(temp)):
init.itemset(tuple(vect[ii*NumberCol+iii]), temp[iii])
data[Countzone][key] = init
Count = Count+1
return data

• Not enough for an answer but you could/should use list comprehension in vectorr and the with keyword when you open your file (you forfot to close it) – SylvainD Jan 7 '14 at 12:53

There are two algorithmic aspects to your code which concern me. The most significant problem is that you read the entire file in to memory before you start processing the data:

filelist = a.readlines()


This is an inefficient way to do things with large input files.

You should instead be reading the data 1 line at a time and processing each line as you get it. This allows you to have a much smaller memory 'footprint', and it allows you to discard data that you do not need.... instead of your while loop while count < leng consider the approach:

for line in a:
....


This should reduce your memory significantly.

(Asker updated question....) The second issue you should consider changing is using numpy arrays for the vectorr method as well. This array is about the same size as the final array and you are incrementally allocating memory for it. Doing it with a single initializer (perhaps vect = empty((I*J*K, 3)) ) would be worth considering.

• I think I have uploaded the wrong version of the code (I am very sorry!!)-I going to edit it right now – Pierpaolo Jan 7 '14 at 13:30