Timeline for Efficiently processing large (~100 MB) structured binary data in Python 3
Current License: CC BY-SA 3.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 7, 2017 at 15:07 | comment | added | 0x2b3bfa0 | Maybe you can use kaitai struct | |
| Feb 16, 2016 at 23:39 | vote | accept | glS | ||
| Jan 21, 2016 at 13:24 | answer | added | Michael Urman | timeline score: 3 | |
| Jan 21, 2016 at 9:08 | comment | added | oliverpool |
@glS if you have 5 timestamps within the window, only the first branch of the if timestamps_in_window will be taken (4 channels will be appended) and 5 elements will be in the channels list. After you quit the window, the other branch is taken and len(channels) equals 5: hence if len(channels) == 4: is not correct
|
|
| Jan 21, 2016 at 8:09 | comment | added | glS |
@oliverpool you are right for the bug making some fourfold coincidences not being detected, it has also been pointed out by JS1 in the related question on the C implementation, thanks for noticing. If more than 4 timestamps are within the window I do not have to save the corresponding channels so I think in this case the if len(channels) == 4: is correct.
|
|
| Jan 20, 2016 at 16:40 | comment | added | oliverpool |
And it will fail if 5 timestamps are within the window : if len(channels) >= 4: will efficiently correct this.
|
|
| Jan 20, 2016 at 16:17 | comment | added | oliverpool |
You code does not exactly match your description: if you have a timestamp sequence of 100, 150, 170, 220, 230, 500, 600, the 150, 170, 220, 230 sequence will not be detected (because the 220 will be the triggering_timestamp). It might no matter for your actual problem, but it's a corner case that may influence the code.
|
|
| Jan 20, 2016 at 14:35 | comment | added | glS | @MichaelUrman I've done as you suggested, and it actually seems to improve things. On a ~50MB file it now takes ~9.1s against ~11.5s, after simply putting the whole thing (I'm now using this version of the code) into a function and calling the function. If you find the time to write an answer I'd be really curious to understand why does this happen. | |
| Jan 20, 2016 at 14:15 | comment | added | Phrancis | @glS please do not modify the code in your question (or add more code) after answers have been posted. See What should I do when someone answers my question? | |
| Jan 20, 2016 at 14:13 | history | rollback | Phrancis |
Rollback to Revision 4
|
|
| Jan 20, 2016 at 14:08 | history | edited | glS | CC BY-SA 3.0 |
added 2743 characters in body; edited title
|
| Jan 20, 2016 at 13:40 | history | edited | glS | CC BY-SA 3.0 |
deleted 2239 characters in body
|
| Jan 20, 2016 at 13:16 | comment | added | Michael Urman | I hope to write an actual review later, but for now I wanted to suggest that you check your performance if you stuff everything into a function and call that function. (CPython has an optimization for function locals that doesn't help at the global scope.) | |
| Jan 17, 2016 at 20:18 | answer | added | vnp | timeline score: 6 | |
| Jan 17, 2016 at 19:01 | history | edited | glS |
edited tags
|
|
| Jan 17, 2016 at 18:38 | history | edited | glS | CC BY-SA 3.0 |
added 1 character in body
|
| Jan 17, 2016 at 18:34 | review | First posts | |||
| Jan 17, 2016 at 18:59 | |||||
| Jan 17, 2016 at 18:32 | history | asked | glS | CC BY-SA 3.0 |