I have binary files containing sparse matrices. Their format is:
number of rows int length of a row int column index int value float
Reading each row with a single struct call instead of looping through each row with single struct calls gave me roughly a 2-fold speedup. I'm parsing 1 GB sized matrices and I would like to speed this proces up even further.
from scipy.sparse import coo_matrix import struct def read_sparse_matrix(handle): cols =  rows =  weights =  numrows = struct.unpack('i' , handle.read(4)) shape = numrows for rownum in range(numrows): rowlen = struct.unpack('i', handle.read(4)) row = list(struct.unpack("if" * rowlen, handle.read(8 * rowlen))) cols += row[::2] weights += row[1::2] rows += [rownum] * rowlen return coo_matrix((weights, (rows, cols)), shape=(shape, shape))
A file contains multiple of these matrices, and other informatinon, so the size of the file is not informative about the structure of the matrix.