6
\$\begingroup\$

I have binary files containing sparse matrices. Their format is:

number of rows int
    length of a row int
        column index int
        value float

Reading each row with a single struct call instead of looping through each row with single struct calls gave me roughly a 2-fold speedup. I'm parsing 1 GB sized matrices and I would like to speed this proces up even further.

from scipy.sparse import coo_matrix
import struct
def read_sparse_matrix(handle):
    cols = []
    rows = []
    weights = []
    numrows = struct.unpack('i' , handle.read(4))[0]
    shape = numrows
    for rownum in range(numrows):
        rowlen = struct.unpack('i', handle.read(4))[0]
        row = list(struct.unpack("if" * rowlen, handle.read(8 * rowlen)))
        cols += row[::2]
        weights += row[1::2]
        rows += [rownum] * rowlen
    return coo_matrix((weights, (rows, cols)), shape=(shape, shape))

A file contains multiple of these matrices, and other informatinon, so the size of the file is not informative about the structure of the matrix.

\$\endgroup\$
  • 1
    \$\begingroup\$ Can you change the format? If so (and you don't need it to be human readable), you should probably just serialize the data. That should be faster. \$\endgroup\$ – Oscar Smith Mar 12 at 15:07
  • \$\begingroup\$ Have you tried reading (or mmapping) the entire file into a buffer, and using struct.unpack_from to decode the data? \$\endgroup\$ – Austin Hastings Mar 12 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.