Timeline for High performance parsing for large, well-formatted text files
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 16, 2015 at 18:44 | vote | accept | isosceleswheel | ||
| Jul 27, 2015 at 18:44 | history | edited | Eric | CC BY-SA 3.0 |
added 136 characters in body
|
| Jul 27, 2015 at 18:37 | history | edited | Eric | CC BY-SA 3.0 |
added 60 characters in body
|
| Jul 27, 2015 at 18:37 | comment | added | Eric |
@isosceleswheel: This doesn't directly give a performance improvement, but it allows you to pipeline your processing, and maybe do parts in parallel (ie, move sections you want to keep into separate files, and then use multiprocess)
|
|
| Jul 27, 2015 at 13:46 | comment | added | isosceleswheel |
Thanks for your code, it has given me some better intuition for iterator/generator processing. Although I find your solution more elegant than the function I posted, is there any major performance gain here or are you just suggesting stylistic and "best practice" improvements? My understanding is that the f in for line in f is already an iterator, so I am not loading anything excessive into RAM from the file, and I flush out all of the counters, flags etc. for each segment. Also, could you elaborate on what you meant by "business logic"? Thanks again for your help :)
|
|
| Jul 26, 2015 at 22:00 | history | answered | Eric | CC BY-SA 3.0 |