Reading sparse matrix from binary file

I have binary files containing sparse matrices. Their format is:

number of rows int
length of a row int
column index int
value float


Reading each row with a single struct call instead of looping through each row with single struct calls gave me roughly a 2-fold speedup. I'm parsing 1 GB sized matrices and I would like to speed this proces up even further.

from scipy.sparse import coo_matrix
import struct
cols = []
rows = []
weights = []
shape = numrows
for rownum in range(numrows):

• Have you tried reading (or mmapping) the entire file into a buffer, and using struct.unpack_from to decode the data? – Austin Hastings Mar 12 at 22:58