Multiline Input into Python Arrays

Question

I've knocked together a few lines of python to read in stats for a service (haproxy), and store them in an array (to do some analysis on later). It basically takes a multiline output, and splits it into multiple subarrays. This is how I've done it - can anyone offer improvements for me?

def build_array():
        services=[]
        for line in data.split('\n'):           # split out each line of raw input
                holding=[]                      # start a temp array
                for var in line.split(','):     # for each value append it to the temp array
                        holding.append(var)
                services.append(holding)        # append the temp array to the services array
        return services

The raw data is in the format:

data="""web,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,
    mysql,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,
    web-https,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,4,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,
    web,web2-NEW,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,0,0,19,0,,1,5,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
    web,web1-OLD,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,19,0,,1,5,2,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
    web,BACKEND,0,0,0,0,0,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,19,0,,1,5,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,"""

Answer 1

This looks like CSV. You are probably better off using the python csv module.

            holding=[]                      # start a temp array
            for var in line.split(','):     # for each value append it to the temp array
                    holding.append(var)
            services.append(holding)        # append the temp array to the services arra

Can be written as

services.append( line.split(',') )

line.split() returns a list already, there is no need to copy the elements into another list.

Answer 2

build_array uses a global variable, which I don't see any good reason for. Why not just pass the data in as a parameter?

def build_array(data):
   # ...whatever

Also, you don't remove the leading whitespace from the first element on each line - remember that python """ quotes will leave indentation spaces in the string (even though they get removed from docstrings).

It might be better just to remove leading and trailing whitespace from every string immediately after you split it, unless whitespace is ever significant in your application:

holding.append(var.strip())

Answer 3

I think NumPy library exists exactly for use cases like this. It is a pretty well known, established Python library which provides numerical capabilities in Python. It has a very active community with frequent releases. In numpy I would just do this:

from numpy import genfromtxt
from StringIO import StringIO

genfromtxt(StringIO(data), delimter=',', dtype=None)

Very concise and more readable. Much easier to maintain since fewer lines of code and hence fewer bugs :)

Also, I tried executing your code and genfromtxt code and it looks like for the given data genfromtxt is slower than your code by almost an order of magnitude, but I think specifying the datatype of the columns in the data will improve the performance of genfromtxt a bit. Also, if you can give the nature of the analysis you are trying to perform, what columns you want specifically for this analysis, the code can be made much faster.

Over and above all these, Python emphasises on readability because developer time is more expensive than CPU time today!