Importing of big files:right approach?

Question

I want to import in python some ascii file ( from tecplot, software for cfd post processing). Rules for those files are (at least, for those that I need to import): -The file is divided in several section -Each section has two lines as header like:

VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen" 
ZONE T="Window(s) : E_W_Block0002_ALL",  I=29,  J=17,  K=25, F=BLOCK

• Each section has a set of variable given by the first line. When a section ends, a new section starts with two similar lines.

• For each variable there are I*J*K values.

• Each variable is a continous block of values.
• There are a fixed number of values per row (6).
• When a variable ends,the next one start in a new line.
• Variables are "IJK ordered data".The I-index varies the fastest; the J-index the next fastest; the K-index the slowest. The I-index should be the inner loop, the K-index shoould be the outer loop, and the J-index the loop in between.

Here an example of data:

VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen" 
ZONE T="Window(s) : E_W_Block0002_ALL",  I=29,  J=17,  K=25, F=BLOCK
-3.9999999E+00 -3.3327306E+00 -2.7760824E+00 -2.3117116E+00 -1.9243209E+00 -1.6011492E+00
[...]
0.0000000E+00 #fin first variable
-4.3532482E-02 -4.3584235E-02 -4.3627592E-02 -4.3663762E-02 -4.3693815E-02 -4.3718831E-02 #second variable, 'y'
[...]
1.0738781E-01 #end of second variable
[...]
[...]
VARIABLES = "x" "y" "z" "ro" "rovx" "rovy" "rovz" "roE" "M" "p" "Pi" "tsta" "tgen" #next zone
ZONE T="Window(s) : E_W_Block0003_ALL",  I=17,  J=17,  K=25, F=BLOCK

I am quite new at python and I have written a code to import the data to a dictionary, writing the variables as 3d numpy.array . Those files could be very big, (up to Gb). How can I make this code faster?(or more generally, how can I import such files as fast as possible)?.

import numpy as *
from numpy.linalg import norm
import re

def vectorr(I, J, K):
    arr = empty((I*J*K, 3), int)
    arr[:,0] = tile(arange(I), J*K)
    arr[:,1] = tile(repeat(arange(J), I), K)
    arr[:,2] = repeat(arange(K), I*J)
    return arr

def all_data(pathfilename,NumberCol = 6):
    with open(pathfilename) as a:
        filelist = a.readlines()
        Count = 0
        data = dict()
        leng = len(filelist)
        Countzone = 0
        while Count < leng:
            strVARIABLES = re.findall('VARIABLES', filelist[Count])
            variables = re.findall(r'"(.*?)"',  filelist[Count])
            Countzone = Countzone+1
            data[Countzone] = {key:[] for key in variables}
            Count = Count+1
            strI = re.findall('I=....', filelist[Count])
            strI = re.findall(r'\d+', strI[0]) 
            I = int(strI[0])
            strJ = re.findall('J=....', filelist[Count])
            strJ = re.findall(r'\d+', strJ[0])
            J = int(strJ[0])
            strK = re.findall('K=....', filelist[Count])
            strK = re.findall(r'\d+', strK[0])
            K = int(strK[0])
            data[Countzone]['indmax'] = array([I, J, K])
            pr = prod(data[Countzone]['indmax'])
            lin = pr // NumberCol
            if pr % NumberCol != 0 :
                lin = lin+1
            vect = vectorr(I, J, K)
            for key in variables:
                init = zeros((I, J, K))
                for ii in range(0, lin):
                    Count = Count+1
                    temp = [ float(x.replace('D', 'E')) for  x in filelist[Count].split() ]
                    for iii in range(0, len(temp)):
                        init.itemset(tuple(vect[ii*NumberCol+iii]), temp[iii])
                        data[Countzone][key] = init
            Count = Count+1
    return data

Answer 1

There are two algorithmic aspects to your code which concern me. The most significant problem is that you read the entire file in to memory before you start processing the data:

filelist = a.readlines()

This is an inefficient way to do things with large input files.

You should instead be reading the data 1 line at a time and processing each line as you get it. This allows you to have a much smaller memory 'footprint', and it allows you to discard data that you do not need.... instead of your while loop while count < leng consider the approach:

for line in a:
    ....

This should reduce your memory significantly.

(Asker updated question....) The second issue you should consider changing is using numpy arrays for the vectorr method as well. This array is about the same size as the final array and you are incrementally allocating memory for it. Doing it with a single initializer (perhaps vect = empty((I*J*K, 3)) ) would be worth considering.