I am generating co-occurrence matrix (2000X2000) in Python.
- Vocab is a list of the 2000 top words in corpus. It does not contain all words in corpus.
Corpus is a Pandas dataframe of 30000 (rows) sentences.
An example of a single row in the corpus:
the energy classroom high the room filled scholars eager learn they embarking learning journey unlike no these scholars asked learn time rapidly changing technology communication as teacher job give tools successful our school special always looking ways give students best education possible my 4th grade students counting provide tools necessary success i want keep love learning alive giving students access technology critical success 21st century learning environment our school 1 3 ratio ipads kids we use variety apps allow kids collaborate build critical thinking skills even though ipads keyboard ipad cumbersome difficult kids use therefore i asking 15 keyboards plug ipads having wired keyboard allows kids practice keyboarding skills standard collaborate others effectively utilize technology accesskeyboards enhance collaboration choice critical thinkin
My code is currently taking 120s per iteration and 364 hours to complete run. I am running this code in Google Colab with 25GB RAM.
The code works fine with small data like 10 row corpus and 5 word vocab. Is there a better way to complete this task or speed up this process?
tqdm is required because if the code runs for longer than 12 hours, which it is, then it gets disconnected from Google Colab.
coocur_matrix = np.zeros((len(voca), len(voca)),np.float64)
#python from tqdm import tqdm window_size=5 corpus=corpu1 vocab = voca for word1 in tqdm(vocab): for word2 in vocab: for sent in corpus: doc_tokens =  doc_tokens=sent.split() p1= [i for i in range(len(doc_tokens)) if doc_tokens[i] == word1] p2= [i for i in range(len(doc_tokens)) if doc_tokens[i] == word2] for k in p1: print(9) for l in p2: if (abs(l-k)<=window_size): if(word1!=word2): coocur_matrix[vocab.index(word1),vocab.index(word2)] += 1 print(coocur_matrix)