Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

Is there better way to read from a file and connect all data in one big data than to use generators?

At the moment, I do the following:

  1. use generators to read data from files.
  2. use NumPy to pack all files in 3D array.
  3. use pandas to stack it to 2D array so readable for next operations (e.g. plotting).

In my example:

I split reading from file operations generators in 2 modules, and have a third module for reading names (file is not connected with others generators_module). In the last separate file I import that generators_modules, and make arrays with NumPy, then pandas.

In the code of modules I use os.walk to read from files, regex to read only that data which is needed.

In the code of making arrays with NumPy, pandas:

  • I enter variable parameters needed to get data with generator_modules
  • I pass from generators to array with code:

    xdata=np.array( [(float(line['PRESSURE']), float(line['CURVE'])) for line in dt_cols] )

I have four python files, as below:

module1 (gen_enter):

import re, matplotlib as mpl, matplotlib.pyplot as plt, os, fnmatch

def gen_find(filepat,top):
    for path, dirlist, filelist in os.walk(top):
        for name in fnmatch.filter(filelist,filepat): yield os.path.join(path,name)

def gen_open(filenames):
    for name in filenames:
        if name.endswith(".XL"): yield open(name)
        else: pass

def gen_get(sources):
    for s in sources:
        for item in s:  yield item

def gen_grep(pattern, fileparse):
    for line in fileparse:
        inputlines = line[:].strip().replace(';',' ')
        if pattern.search(inputlines):  yield inputlines
        else: pass

def field_map(dictseq,name,func):
    for d in dictseq:
        d[name] = func(d[name])
        yield d

module2 (gen_returnlines):

from gen_enter import *

def lines_from_dir(filepat, dirname):
    findnames = gen_find(filepat, dirname)
    openfiles = gen_open(findnames)
    getlines = gen_get(openfiles)
    #patlines = gen_grep(pattern, getlines)
    return getlines

module3 (gen_shownames):

import re, matplotlib as mpl, matplotlib.pyplot as plt, os, fnmatch

def gen_shownames(filepat,top):
    for path, dirlist, filelist in os.walk(top):
        for name in fnmatch.filter(filelist,filepat): yield name

The main code:

import numpy as np, pandas as pd, os, fnmatch, re
from pylab import *
from gen_returnlines import *
from gen_shownames import *

def dict_cols(lines):
        groups = (patlines.match(line) for line in getlines)
        tuples  = (group.groups() for group in groups if group)
        colnames = ('PRESSURE','CURVE')
        line = (dict(zip(colnames,t)) for t in tuples)
        line = (field_map(line,"PRESSURE", lambda s: float(s)))
        line = (field_map(line,"CURVE",float))
        return line

dir='C:\\Users\\REDHOOD\\workspace\\Politechnika_python\\\silniki\\files' #note: all files from ../files/
pattern = re.compile(r'(\d{3}\.\d{1})\D*(\d{3}\.\d{1})\D*')
pats = '(\d{3}\.\d{1})\D*(\d{3}\.\d{1})\D*'
patlines = re.compile(pats)

if __name__ == '__main__':
    names = gen_shownames('*', dir)
    getlines = lines_from_dir('*',dir)
    dt_cols = dict_cols(getlines)

xdata = np.array( [(float(line['PRESSURE']), float(line['CURVE'])) for line in dt_cols] )
xrdata = np.reshape(xdata,(17,360,2))

datapanel = pd.Panel(
                  items=[k for k in names],

datapaneldf = datapanel.to_frame()       #pressure na curve -> 360 stopni
share|improve this question

migrated from stackoverflow.com Dec 20 '12 at 17:21

This question came from our site for professional and enthusiast programmers.

This all looks pretty good to me. Do you have a specific concern, or are you merely looking for review? –  Marcin Dec 20 '12 at 15:06
You should be able to use read_csv which will probably be faster (especially in pandas 0.10). –  hayden Dec 20 '12 at 15:26
read_csv --- its for reading csv_files, right? so i should first rewrite file to csv format, if its in txt format files? –  Robert Grzelka Dec 21 '12 at 15:12
is it possible to write code, what will automaticly choose the dimensions of array, by the number of catalogues, files, lines, columns and rows? –  Robert Grzelka Jan 8 '13 at 13:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.