Thanks so much. I updated my code as per your recommendation. It is still going on for an hour or so. Guess I will just have to wait for it to finish, but I'm sure it's better than my method.

@Graipher: Added the imports

@MathiasEttingerI tried that earlier. But the generate_combination_of_questions(IncorrectQuestions) is the part that hogs up the RAM, and I think understandably so, because it is working on the entire dataframe with all the memberid values. Because of this reason itself, I moved to the code I had pasted.

Here's what I finally wrote: for memberid in IncorrectQuestions['memberid'].unique(): for pair in generate_combination_of_questions(IncorrectQuestions[Incorre‌​ctQuestions['memberi‌​d']==memberid]): writer.writerow(pair)