I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.
The formula for tf-idf I am using is:
log(1 + tf) * log(N / (1 + df)) # N is the number of coloumns of the matrix # tf is the value at a cell of the matrix # df is the number of non-zero elements in a row
So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:
csr[i,j] = log(1 + csr[i, j]) * log(csr.shape / (1 + sum(csr[i] != 0))
Since I have a large matrix, I am using sparse matrices from
scipy.sparse. Is it possible to do the tf-idf computation more efficiently?
import numpy as np import scipy.sparse import scipy.io csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata')) for iter1 in xrange(csr.shape) : # Finding indices of non-zero data in the matrix tmp,non_zero_indices = csr[iter1].nonzero() # dont need tmp df = len(non_zero_indices) if df > 0 : # This line takes a long time... csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((csr.shape)/(1.0+df))