1
\$\begingroup\$

I've knocked together a few lines of python to read in stats for a service (haproxy), and store them in an array (to do some analysis on later). It basically takes a multiline output, and splits it into multiple subarrays. This is how I've done it - can anyone offer improvements for me?

def build_array():
        services=[]
        for line in data.split('\n'):           # split out each line of raw input
                holding=[]                      # start a temp array
                for var in line.split(','):     # for each value append it to the temp array
                        holding.append(var)
                services.append(holding)        # append the temp array to the services array
        return services

The raw data is in the format:

data="""web,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,
    mysql,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,
    web-https,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,4,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,
    web,web2-NEW,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,0,0,19,0,,1,5,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
    web,web1-OLD,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,19,0,,1,5,2,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
    web,BACKEND,0,0,0,0,0,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,19,0,,1,5,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,"""
\$\endgroup\$
4
\$\begingroup\$

This looks like CSV. You are probably better off using the python csv module.

            holding=[]                      # start a temp array
            for var in line.split(','):     # for each value append it to the temp array
                    holding.append(var)
            services.append(holding)        # append the temp array to the services arra

Can be written as

services.append( line.split(',') )

line.split() returns a list already, there is no need to copy the elements into another list.

\$\endgroup\$
  • \$\begingroup\$ I was trying something like that originally, but it kept failing on me. Works now! Cheers. \$\endgroup\$ – Coops Jun 22 '11 at 11:00
2
\$\begingroup\$

build_array uses a global variable, which I don't see any good reason for. Why not just pass the data in as a parameter?

def build_array(data):
   # ...whatever

Also, you don't remove the leading whitespace from the first element on each line - remember that python """ quotes will leave indentation spaces in the string (even though they get removed from docstrings).

It might be better just to remove leading and trailing whitespace from every string immediately after you split it, unless whitespace is ever significant in your application:

holding.append(var.strip())
\$\endgroup\$
1
\$\begingroup\$

I think NumPy library exists exactly for use cases like this. It is a pretty well known, established Python library which provides numerical capabilities in Python. It has a very active community with frequent releases. In numpy I would just do this:

from numpy import genfromtxt
from StringIO import StringIO

genfromtxt(StringIO(data), delimter=',', dtype=None)

Very concise and more readable. Much easier to maintain since fewer lines of code and hence fewer bugs :)

Also, I tried executing your code and genfromtxt code and it looks like for the given data genfromtxt is slower than your code by almost an order of magnitude, but I think specifying the datatype of the columns in the data will improve the performance of genfromtxt a bit. Also, if you can give the nature of the analysis you are trying to perform, what columns you want specifically for this analysis, the code can be made much faster.

Over and above all these, Python emphasises on readability because developer time is more expensive than CPU time today!

\$\endgroup\$
  • \$\begingroup\$ numpy is awesome, although I don't know if its useful here. It all depends on what you want to do with the data afterwards. \$\endgroup\$ – Winston Ewert Jun 21 '11 at 22:00
  • \$\begingroup\$ I've briefly used numpy before. The only reason for me probably not using it is that this script will be deployed on a number of boxes, and I'd prefer to avoid having to install the numpy library on each machine. Thanks anyhow. \$\endgroup\$ – Coops Jun 22 '11 at 11:00
  • \$\begingroup\$ I am not trying to convince you to use numpy, but installing any Python package has never been a problem to me. It has always been just a matter of easy_install numpy. \$\endgroup\$ – Madhusudan.C.S Jun 22 '11 at 12:44
  • \$\begingroup\$ Yes, but the script will probably be used across 100s of servers. Each one needing numpy installed, and each one being installed at a different time. It's not a problem with numpy, or it's install process, it's that I can't guarantee a server installed X years ago will have the same version of python/numpy as a server installed last week. The less I depend on external requirements, the more the script becomes portable. \$\endgroup\$ – Coops Jun 22 '11 at 20:00
  • \$\begingroup\$ Ok. Makes sense. Have to trade-off at some point :) \$\endgroup\$ – Madhusudan.C.S Jun 23 '11 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.