Reading cell array of vectors with h5py

I have a data stored in .mat files, each .mat contains a cell array of vectors of length ~70. I'm trying to read these .mat files in to Python with h5py, but reading just 1000 cells takes about 7 seconds, and I have in total about 10^8 cells to read...

Is there any way I could simplify this process? I'm pretty sure the bottleneck lies in reading the mat file, and all the dereferences implicit in the h5 file structure.

# normalize_mat(mat_file)
#
# takes in a .mat file containing a cell vector 'seg',
#   which contains references to vectors representing one pulse
#
# returns a matrix of equal length, 1-normaled, segment vectors
#
def normalize_mat(mat_file, RESAMPLED_LENGTH=128):
segments = []
# reading from .mat version 7.3 files is super weird..
# we start off by opening the file with h5py
with h5py.File(mat_file) as mat_file:
# then we get the cell array 'seg'
seg_data = mat_file['seg']
for segment_index in xrange(1001):        #xrange(seg_data.shape[0]):
if (segment_index%1000) == 0:
print "Evaluating segment", segment_index

# then we get a reference to the actual sample array for one segment
# this is stored in [0] of the dereferenced segment_index. ([1] is the type)
segment_ref = seg_data[segment_index][0]

# the data is stored as [[val1],[val2],[val[3]]], so lets get rid of that
segment = [x[0] for x in mat_file[segment_ref]]
segment = normalize_segment(segment, RESAMPLED_LENGTH)
# above statement provides little performance hit.
segments.append(segment)

return segments

• Could you add the normalize_segment method for context? It would also be useful if you could check how the time taken varies with the number of segments (say, with 500, 1000, 2000 segments)---that would characterise the rough complexity of your process. If you could determine the difference between reading the first 1000 segments and reading the whole of a 1000-segment file, that would also be interesting information. – David Morris Feb 21 '16 at 21:04
• Have you tried the hdf5storage package's loadmat? It might simplify this considerably. – TheBlackCat Apr 22 '16 at 17:50