I have the following code that I'm using to create combinations of elements in my dataset:
start_time = time.time()
question_pairs = []
import itertools as iter
for memberid in IncorrectQuestions.memberid.unique():
combinations = IncorrectQuestions[IncorrectQuestions.memberid == memberid].groupby('memberid').questionid.apply(
lambda x: list(iter.combinations(x, 2)))
for elem in combinations:
for el in elem:
question_pairs.append(list(el))
print("--- %s seconds ---" % (time.time() - start_time))
the DataFrame IncorrectQuestions has about 40 Million records. Here's the sample dataset:
memberid created firstencodedid questionid
0 9 2016-01-18 05:10:44 MAT.CAL.110 5696d0248e0e0869c96357d3
1 9 2016-01-18 05:10:44 MAT.CAL.110 5696cbc45aa413444ffd9973
2 9 2016-01-18 05:10:44 MAT.CAL.110 5696cf86da2cfe6f21d09879
3 34 2016-11-10 04:24:14 MAT.ARI.300 51d8cd415aa41337ec50425a
4 34 2016-11-10 04:24:14 MAT.ARI.300 559a84505aa4136cb37be676
The piece of code above takes way too long. It has been an hour and it is still running. Is there a way to optimize this piece of code so it takes less time?
The desired output would create all possible combinations of questionid for each memberid. For example, the combinations for memberid = 9 would be:
['5696d0248e0e0869c96357d3','5696cbc45aa413444ffd9973']
['5696d0248e0e0869c96357d3','5696cf86da2cfe6f21d09879']
['5696cbc45aa413444ffd9973','5696cf86da2cfe6f21d09879']
The combinations for memberid = 34 would be:
['51d8cd415aa41337ec50425a','559a84505aa4136cb37be676']
And finally, the combined output would be both the lists combined, i.e.:
['5696d0248e0e0869c96357d3','5696cbc45aa413444ffd9973']
['5696d0248e0e0869c96357d3','5696cf86da2cfe6f21d09879']
['5696cbc45aa413444ffd9973','5696cf86da2cfe6f21d09879']
['51d8cd415aa41337ec50425a','559a84505aa4136cb37be676']
I hope this makes things clearer.
Any pointers would be appreciated.
