SciPy sparse: optimize computation on non-zero elements of a sparse matrix (for tf-idf)

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1 + tf) * log(N / (1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1 + csr[i, j]) * log(csr.shape / (1 + sum(csr[i] != 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

for iter1 in xrange(csr.shape) :

# Finding indices of non-zero data in the matrix
tmp,non_zero_indices = csr[iter1].nonzero()
# dont need tmp

df = len(non_zero_indices)
if df > 0 :
# This line takes a long time...
csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((csr.shape)/(1.0+df))
• What are the approximate dimensions of csr and how dense is it? I'd like to be able to run this on my computer to test but I need to be able to mock thedata. – Veedrac Dec 29 '14 at 3:50
• The size of the matrix is (1M X 500K), and the density is 0.0002. I was testing this code with a random matrix of smaller size, like : coo_mat = scipy.sparse.rand(100000, 50000, density=0.0002, format='coo') – Avisek Dec 29 '14 at 8:04