4 added 1 character in body
source | link

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1 + tf) * log(N / (1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1 + csr[i, j]) * log(csr.shape[1] / (1 + sum(csr[i] != 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docscsr.shape[1])/(1.0+df))

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1 + tf) * log(N / (1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1 + csr[i, j]) * log(csr.shape[1] / (1 + sum(csr[i] != 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1 + tf) * log(N / (1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1 + csr[i, j]) * log(csr.shape[1] / (1 + sum(csr[i] != 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((csr.shape[1])/(1.0+df))
3 improved formatting
source | link

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1+tf1 + tf)*log * log(N / (1+df1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1+csr[i1 + csr[i, j])*log * log(csr.shape[1] / (1+sum1 + sum(csr[i] !=0= 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1+tf)*log(N/(1+df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1+csr[i,j])*log(csr.shape[1]/(1+sum(csr[i]!=0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1 + tf) * log(N / (1 + df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1 + csr[i, j]) * log(csr.shape[1] / (1 + sum(csr[i] != 0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))
2 deleted 4 characters in body; edited tags; edited title
source | link

Scipy SciPy sparse  : Optimizeoptimize computation on non-zero elements of a sparse matrix (for tf-idf)

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is  :

log(1+tf)*log(N/(1+df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute  :

csr[i,j] = log(1+csr[i,j])*log(csr.shape[1]/(1+sum(csr[i]!=0))

Since I have a large matrix, I am using sparse matrices from scipy.sparsescipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))

Scipy sparse  : Optimize computation on non-zero elements of a sparse matrix (for tf-idf)

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is  :

log(1+tf)*log(N/(1+df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute  :

csr[i,j] = log(1+csr[i,j])*log(csr.shape[1]/(1+sum(csr[i]!=0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))

SciPy sparse: optimize computation on non-zero elements of a sparse matrix (for tf-idf)

I have a sparse matrix (term-document) containing integers (word counts/tf) and I am trying to compute the tf-idf, for every non-zero value in the sparse-matrix.

The formula for tf-idf I am using is:

log(1+tf)*log(N/(1+df))   # N is the number of coloumns of the matrix
# tf is the value at a cell of the matrix
# df is the number of non-zero elements in a row

So for a matrix csr, at an index [i,j] with a non-zero value, I want to compute:

csr[i,j] = log(1+csr[i,j])*log(csr.shape[1]/(1+sum(csr[i]!=0))

Since I have a large matrix, I am using sparse matrices from scipy.sparse. Is it possible to do the tf-idf computation more efficiently?

import numpy as np
import scipy.sparse
import scipy.io

csr = scipy.sparse.csr_matrix(scipy.io.mmread('thedata'))

for iter1 in xrange(csr.shape[0]) :

    # Finding indices of non-zero data in the matrix
    tmp,non_zero_indices = csr[iter1].nonzero()
    # dont need tmp

    df = len(non_zero_indices)
    if df > 0 :
        # This line takes a long time...
        csr[iter1,non_zero_indices] = np.log(1.0+csr[iter1,non_zero_indices].todense())*np.log((num_of_docs)/(1.0+df))
1
source | link