#!/usr/bin/python3
import sys
import re
scripttitle="PYTHON SIMPLE PROCESSOR (regex parsing)"
Python convention is to put constants in ALL_CAPS
linecount=0
commentary=0
unknownline=0
DATA={}
This isn't a constant, so it really shouldn't be all caps.
pattern_data=re.compile(r"^(\d+)\s+(\S+)\s+(\S+)\s+(\d+(?:\.\d+)?)")
print("#",scripttitle,"\n#TRANSFORMED INPUT",sep='')
Its best to put all your actual logic inside a function rather then at the main level of a script. It'll run a bit faster that way.
for line in sys.stdin:
linecount+=1
if line.startswith("#"):
commentary+=1
continue
I find code is almost always more readable when you put thing in the else block rather then use continue
line=line.replace(',','')
m=re.match(pattern_data,line)
if m:
Typically we'd explicit check for none with if m is not None
i,k1,k2,value = m.group(1,2,3,4)
Actually you could use m.groups() here. I'd also avoid such unhelpnames as i, k1, and k2
i=int(i)
I'm not sure why you bother doing this if you are just going to print it out anyways
value=float(value)
try:
DATA[k1][k2]+=value
except KeyError:
if k1 not in DATA: # Can't automaticaly create missing key and do the insert?
DATA[k1]={}
if k2 not in DATA[k1]:
DATA[k1][k2]=value
else:
DATA[k1][k2]+=value
Python has a useful class called defaultdict. It lets you provide the default value for a dictionary. It also has a class called Counter for counting things So you could do this:
DATA = collections.defaultdict(collections.Counter)
Then
DATA[k1][k2] += value
will always work because the default cases are handled.
print("{0},{1:.0f},{2},{3}".format(i,value,k2,k1))
It'd probably be easier to follow using sep=',' rather then what you've done here
else:
unknownline+=1
print("#DATADUMP")
keystat=0
for k1 in sorted(DATA):
Instead use for k1, items in sorted(DATA.items): Then items will be DATA[k1] and you can relooking up the data
print(k1,':',sep='',end='')
for k2 in sorted(DATA[k1]):
Same here, use the .items() to fetch keys and values together
keystat+=1
print(' (',k2,':',int(DATA[k1][k2]),')',sep='',end='')
print()
report="#{0}\n#{1}\nparsed line: {2}, commentary line: {3}, unknown line: {4}, keystat: {5}.".format(
scripttitle, sys.version.replace("\n"," "), linecount, commentary, unknownline, keystat)
print("#REPORT\n"+report,file=sys.stdout)
print(report,file=sys.stderr)
As for performance, remember that Perl is the practical extraction and report language. This kinda thing is perl's bread and butter, so its gonna be hard for python to win. Doesn't mean I'm not gonna try though.
EDIT: Performance
I've played with improving performance, a few points:
m=re.match(pattern_data,line)
A better way is to use
m = pattern_data.match(line)
They both do the same thing, but the first has a speed penalty associated with it.
print(' (',k2,':',int(DATA[k1][k2]),')',sep='',end='')
The print function is expensive, probably due to its versatility. Rewriting your code to use sys.stdout.write() directly gave much better performance.
try:
DATA[k1][k2]+=value
except KeyError:
if k1 not in DATA: # Can't automaticaly create missing key and do the insert?
DATA[k1]={}
if k2 not in DATA[k1]:
DATA[k1][k2]=value
else:
DATA[k1][k2]+=value
Replacing this with defaultdict or counter harmed performance. I rewrote it as
try:
row = DATA[k1]
except KeyError:
row = DATA[k1] = {}
try:
row[k2] += value
except KeyError:
row[k2] = value
Which gave me a speed boost because it avoids looking up the same values in the dictionary more then once.
With those changes I was able to get within one second of the speed of the perl script. But I was still slower. My semi-educated guess is that perl wins due to builtin support for sorting the keys of a hash during iteration. In python the sorting is done in an seperate function and may not be able to take advantage of the same things the perl version can.
FURTHER PERFORMANCE
Put everything in a function. Python optimizes functions more then other code outside of functions.
Replace
for key, value in sorted(data.items()):
with
for key in sorted(data):
value = data[key]
The first looks nicer, but it requires python to sort a list of tuples rather then a list of strings which ends up more expensive.
Replace
sys.stdout.write(' ({}: {})'.format(k1, math.trunc(v)))
With
sys.stdout.write(''.join([' (', k1, ': ', str(math.trunc(v)), ')']))
String formatting is expensive since python has to parse through the string every time to find the formatting positions.
You can add
write = sys.stdout.write
And then use write instead of sys.stdout.write for a bit of a speed boost. See my tweaked version here: http://pastebin.com/wmaR2Bmx.
