3
\$\begingroup\$

I have a string containing around 1 million sets of float/int coords, i.e.:

    '5.06433685685, 0.32574574576, 2.3467345584, 1,,,'

They are all in one large string and only the first 3 values are used. The are separated by 3 commas. The code i'm using right now to read through them and assign them to a list is below:

    def getValues(s):
        output = []
        while s:
            v1, v2, v3, _, _, _, s = s.split(',', 6)
            output.append("%s %s %s" % (v1.strip(), v2.strip(), v3.strip()))         
        return output
    coords = getValues(tempString)

The end results need to be a list of the three floats in each set separated by a space, i.e.:

    ['5.06433685685 0.32574574576 2.3467345584']

The code I defined above does work for what I want, it just takes much longer than I would like. Does anyone have any opinions or ways I could speed it up? I would like to stay away from external modules/modules i have to download.

Btw I don't think it's my computer slowing it down, 32gb ram and 350mb/s write speed, the program that creates the original string does it in about 20 secs, while my code above gets that same string, extracts the 3 values in around 30mins to an hour.

P.s. Using python 2.6 if it matters

EDIT: Tried replacing the while loop with a for loop as I did some reading and it stated for loops were faster, it might have shaved off an extra minute but still slow, new code:

    def getValues(s, ids):
        output = []
        for x in range(len(ids)):
            v1, v2, v3, _, _, _, s = s.split(',', 6)
            output.append("%s %s %s" % (v1.strip(), v2.strip(), v3.strip()))         
        return output
\$\endgroup\$
1
  • \$\begingroup\$ Could a map perform faster then a for loop? What about multiprocessing.pool.map? Could anyone provide an example that I could apply the above too? \$\endgroup\$
    – cbarnett
    Oct 21 '12 at 2:48
1
\$\begingroup\$

How about this?

import random

def build_string(n):
    s = []
    for i in range(n):
        for j in range(3):
            s.append(random.random())
        for j in range(3):
            s.append(random.randint(0, 10))
    s = ','.join(map(str, s))+','
    return s

def old_getvalues(s):
    output = []
    while s:
        v1, v2, v3, _, _, _, s = s.split(',', 6)
        output.append("%s %s %s" % (v1.strip(), v2.strip(), v3.strip()))         
    return output

def new_getvalues(s):
    split = s.split(",")
    while not split[-1].strip():
        del split[-1]
    outputs = [' '.join(split[6*i:6*i+3]) for i in range(len(split)//6)]
    return outputs

I get (using 2.7 here, but I get similar times on 2.6):

In [13]: s = build_string(3)

In [14]: s
Out[14]: '0.872836834427,0.151510882542,0.746899728365,1,5,2,0.908901266489,0.92617820935,0.686859068595,1,0,1,0.0773422174111,0.874219587245,0.473976008481,7,9,2,'

In [15]: old_getvalues(s)
Out[15]: 
['0.872836834427 0.151510882542 0.746899728365',
 '0.908901266489 0.92617820935 0.686859068595',
 '0.0773422174111 0.874219587245 0.473976008481']

In [16]: new_getvalues(s)
Out[16]: 
['0.872836834427 0.151510882542 0.746899728365',
 '0.908901266489 0.92617820935 0.686859068595',
 '0.0773422174111 0.874219587245 0.473976008481']

In [17]: s = build_string(10001)

In [18]: old_getvalues(s) == new_getvalues(s)
Out[18]: True

and times of

1000 old 0.00571918487549 new 0.00116586685181
2000 old 0.0169730186462 new 0.00192594528198
4000 old 0.0541620254517 new 0.00387787818909
8000 old 0.240834951401 new 0.00893807411194
16000 old 3.2578599453 new 0.0209548473358
32000 old 16.0219330788 new 0.0443530082703

at which point I got bored waiting for the original code to finish. And it seems to work nicely on your full case, taking about 2s on my notebook:

In [32]: time s = build_string(10**6)
CPU times: user 13.05 s, sys: 0.43 s, total: 13.48 s
Wall time: 13.66 s

In [33]: len(s), s.count(',')
Out[33]: (51271534, 6000000)

In [34]: s[:200]
Out[34]: '0.442266619899,0.54340551778,0.0973845441797,6,9,9,0.849183222984,0.557159614938,0.95352706538,10,7,2,0.658923388772,0.148814178924,0.553198811754,1,0,8,0.662939105945,0.343116945991,0.384742018719,9,'

In [35]: time z = new_getvalues(s)
CPU times: user 1.14 s, sys: 0.75 s, total: 1.89 s
Wall time: 1.89 s

In [36]: len(z)
Out[36]: 1000000

In [37]: z[:3]
Out[37]: 
['0.442266619899 0.54340551778 0.0973845441797',
 '0.849183222984 0.557159614938 0.95352706538',
 '0.658923388772 0.148814178924 0.553198811754']
\$\endgroup\$
6
  • \$\begingroup\$ Hm This looks very promising thanks for the respongs, I am having one thing that is a bit odd, every once in a while i get a result like this: '-5.46000003815 \n\t\t\t\t\t\t\t\t6.95499992371 -0.194999933243' Which is why i originally had the strip command in my above code, i see you have it in this too, any idea why it's not stripping? \$\endgroup\$
    – cbarnett
    Oct 21 '12 at 3:55
  • 1
    \$\begingroup\$ .strip() only affects the start and the end of a line, not anything in the middle. I only put the .strip() in as a safety measure. If you want to get rid of that whitespace (I'm assuming you've read an entire file in), then either .split() the string or .strip() the individual elements. \$\endgroup\$
    – DSM
    Oct 21 '12 at 3:58
  • \$\begingroup\$ ah ok I'm pretty new to python and wasn't aware of that for the .strip(), they are apparently already in my string, possibly that way from the source, any ideas on how to include a way to remove the newlines and tabs from within the script you provided, i.e without creating to much more inefficiency? \$\endgroup\$
    – cbarnett
    Oct 21 '12 at 4:06
  • 1
    \$\begingroup\$ Probably the fastest would be to add s = ''.join(s.split()) at the start of new_getvalues() -- that will split the string by whitespace and then join it back together. That might seem like it's doing more work, but it's actually faster than .strip()-ping each of the elements, because there's only one call to a builtin function. The secret to writing fast Python code is to push as much of the work into the library C code as possible. \$\endgroup\$
    – DSM
    Oct 21 '12 at 4:14
  • \$\begingroup\$ You sir are truly a gentlemen and a scholar, I can't even believe how much faster it is, seriously (Its a sequence of frames, the last frame = the most data) before i was attempting at half way in the sequence, about 50,000 coords and it was taking 20-30mins, now i did the last frame, 1.5 million coords and it took barely 30 secs! I never realized how much more efficient C code is, very eye opening thank you so much for the time and help, you sir have given me hope for this world :p \$\endgroup\$
    – cbarnett
    Oct 21 '12 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.