Timeline for Code for creating combinations taking a long time to finish
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 1, 2017 at 18:31 | history | edited | 301_Moved_Permanently | CC BY-SA 3.0 |
added 288 characters in body
|
| Mar 1, 2017 at 18:26 | comment | added | 301_Moved_Permanently | @Patthebug Check updated answer | |
| Mar 1, 2017 at 18:23 | history | edited | 301_Moved_Permanently | CC BY-SA 3.0 |
Added a note about memory usage
|
| Mar 1, 2017 at 18:14 | comment | added | Patthebug |
@MathiasEttingerI tried that earlier. But the generate_combination_of_questions(IncorrectQuestions) is the part that hogs up the RAM, and I think understandably so, because it is working on the entire dataframe with all the memberid values. Because of this reason itself, I moved to the code I had pasted.
|
|
| Mar 1, 2017 at 17:49 | comment | added | 301_Moved_Permanently |
@Patthebug try for pair in generate_combination_of_questions(IncorrectQuestions): writer.writerow(pair) instead.
|
|
| Mar 1, 2017 at 17:29 | comment | added | Patthebug |
Here's what I finally wrote: for memberid in IncorrectQuestions['memberid'].unique(): for pair in generate_combination_of_questions(IncorrectQuestions[IncorrectQuestions['memberid']==memberid]): writer.writerow(pair)
|
|
| Mar 1, 2017 at 16:52 | vote | accept | Patthebug | ||
| Mar 1, 2017 at 16:52 | comment | added | Patthebug | Ahh, I see. Thanks a lot for your advice. I was thinking about doing that, without necessarily knowing that that's what a generator is. Thanks for confirming, and also for the amazing, and extremely detailed answer :). | |
| Mar 1, 2017 at 16:43 | comment | added | Patthebug | @PeterTaylor: You are absolutely right. I tried this solution (which is extremely well written, Thanks to @MathiasEttinger) and it hogged all the memory and left my system in a somewhat unresponsive state. I'm clearly not a Python developer and don not really know how to code up generators. So I'm going to have to research more in that area. | |
| Mar 1, 2017 at 16:34 | comment | added | Peter Taylor | Given that OP has just answered my question in comments stating that the average size of a group is about 20 elements, there are going to be on the order of 400 million pairs, and each pair of 24-element strings uses at least 50 bytes. So we're talking on the order of 20 gigabytes of memory with a list. Generators all the way. | |
| Mar 1, 2017 at 15:20 | history | edited | 301_Moved_Permanently | CC BY-SA 3.0 |
added 7 characters in body
|
| Mar 1, 2017 at 14:59 | history | answered | 301_Moved_Permanently | CC BY-SA 3.0 |