# Applying Scikit-Learn Linear Kernel with a generator for low memory pressure

matrix = # a sparse matrix of TFIDF vectors - 28458x3218988


My aim was to create a similarity matrix comparing all documents to all documents using sklearn.metrics.pairwise.cosine_similarity but the doc notes indicated that with normalized values, such as TFIDF vectors, that linear_kernel' was equivalent with better performance.

I tried linear_kernel(matrix,matrix) which I believe is valid but was clearly going to take a long time and was resulting in high memory pressure. My solution which uses a generator and writing to disk to alleviate memory pressure, functions, but I wondered if there was any way to speed it up.

def cos_compare(corpus_matrix):
for i in range(0, corpus_matrix.shape[0]):
x = linear_kernel(corpus_matrix[i],corpus_matrix)
yield i,x


My generator cos_compare iterates through the matrix row by row, yielding the index and the result of applying linear kernel to measure the similarity between document at index i and all documents.

try:
with open('cosine_data.csv','a') as f:
for i,x in cos_compare(matrix):
if i%100 == 0:
print('Saving Row {} of {}:'.format(i, matrix.shape[0]))

The script opens 'cosine_data.csv' for appending and then initializes cos_compare() generator. The single row array that the generator yields as x is then converted to a pandas DataFrame for convenience when writing a line to disk (np.savetxt() was throwing format errors).
Currently works fine but am I overlooking any optimizations from numpy,pandas or scikit` that I could use?