Timeline for Breaking a 3GB gz file into chunks
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 22, 2021 at 19:55 | comment | added | Setris | @JohnDoe Thanks, I'm happy to know that you found it helpful! | |
| Feb 22, 2021 at 14:30 | comment | added | Abercrombie | @Setris I am amazed at the way you have given this review. This comment might be against the rules of Stack Exchange but I would like to thank you for this answer. There was a lot for me to learn from it. | |
| Feb 10, 2021 at 22:12 | comment | added | Setris | @KellyBundy That's a good catch! Yeah a chunk size of 1 MB for me was much faster (~8-9 seconds). I'd imagine the optimal value varies depending on one's system, so a nice enhancement to make to this script would be to allow the user to configure the chunk size to use via a command-line option. | |
| Feb 10, 2021 at 21:08 | comment | added | Kelly Bundy | What's the speed with chunk size 1 MB? That's what I usually use and I think 4 KB was slower for me in previous benchmarks. | |
| Feb 10, 2021 at 19:47 | comment | added | Setris |
@AhlamAIS Nice, I'm glad it worked well for you! My 9GB file has 120,795,955 lines. I don't think yield has much of an effect on time execution here compared to the original, because in the original it is also iterating in a memory-efficient way (line by line).
|
|
| Feb 10, 2021 at 13:22 | comment | added | Ahlam AIS | Thank you so much for both the review and your suggestion! This took less than 2 minutes on my laptop, I am very curious about how chunks of bytes instead of lines made this gap of performance between the two programs, also one question that I am curious about too, how many lines does your 9GB file contain? Mine has about 162 000 000 lines, and does yield have any effect on time execution here? | |
| Feb 10, 2021 at 13:19 | vote | accept | Ahlam AIS | ||
| Feb 10, 2021 at 12:30 | vote | accept | Ahlam AIS | ||
| Feb 10, 2021 at 12:30 | |||||
| Feb 10, 2021 at 8:49 | history | edited | Setris | CC BY-SA 4.0 |
Add timings/tests, expand on note about `run()`
|
| Feb 10, 2021 at 3:33 | history | answered | Setris | CC BY-SA 4.0 |