matrix = # a sparse matrix of TFIDF vectors - 28458x3218988

My aim was to create a similarity matrix comparing all documents to all documents using sklearn.metrics.pairwise.cosine_similarity but the doc notes indicated that with normalized values, such as TFIDF vectors, that `linear_kernel' was equivalent with better performance.

I tried linear_kernel(matrix,matrix) which I believe is valid but was clearly going to take a long time and was resulting in high memory pressure. My solution which uses a generator and writing to disk to alleviate memory pressure, functions, but I wondered if there was any way to speed it up.

def cos_compare(corpus_matrix):
    for i in range(0, corpus_matrix.shape[0]):
        x = linear_kernel(corpus_matrix[i],corpus_matrix)
        yield i,x

My generator cos_compare iterates through the matrix row by row, yielding the index and the result of applying linear kernel to measure the similarity between document at index i and all documents.

    with open('cosine_data.csv','a') as f:
        for i,x in cos_compare(matrix):
            if i%100 == 0:
                print('Saving Row {} of {}:'.format(i, matrix.shape[0]))
            pd.DataFrame(x).to_csv(f, header=False,index=False,mode='a')

    cosine_data = pd.read_csv('cosine_data.csv', header=None)

The script opens 'cosine_data.csv' for appending and then initializes cos_compare() generator. The single row array that the generator yields as x is then converted to a pandas DataFrame for convenience when writing a line to disk (np.savetxt() was throwing format errors).

Currently works fine but am I overlooking any optimizations from numpy,pandas or scikit that I could use?

  • \$\begingroup\$ I am looking for a good solution to this small problem too. I found a doc that may enlighten you a little bit: medium.com/@AgenceSkoli/… \$\endgroup\$ – Z.Wei Mar 17 at 23:38
  • \$\begingroup\$ BTW, the solution you mention runs quite slow. I think it is because the frequent I/O R/W because you use "append". A possible solution could be to use bulk dot product instead of iterating line by line. \$\endgroup\$ – Z.Wei Mar 18 at 0:24
  • \$\begingroup\$ Hi Z.Wei thanks for the input. That article was interesting. I'd also be interested to see an implementation using bulk dot product as you suggest as I don't think I fully understand what this would look like. If you have the time please do add an answer. \$\endgroup\$ – James Allen-Robertson Mar 18 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.