I have a data stored in .mat files, each .mat contains a cell array of vectors of length ~70. I'm trying to read these .mat files in to Python with h5py, but reading just 1000 cells takes about 7 seconds, and I have in total about 10^8 cells to read...

Is there any way I could simplify this process? I'm pretty sure the bottleneck lies in reading the mat file, and all the dereferences implicit in the h5 file structure.

# normalize_mat(mat_file)
# takes in a .mat file containing a cell vector 'seg',
#   which contains references to vectors representing one pulse
# returns a matrix of equal length, 1-normaled, segment vectors
def normalize_mat(mat_file, RESAMPLED_LENGTH=128):
    segments = []
    # reading from .mat version 7.3 files is super weird..
    # we start off by opening the file with h5py
    with h5py.File(mat_file) as mat_file:
        # then we get the cell array 'seg'
        seg_data = mat_file['seg']
        for segment_index in xrange(1001):        #xrange(seg_data.shape[0]):
            if (segment_index%1000) == 0:
                print "Evaluating segment", segment_index

            # then we get a reference to the actual sample array for one segment
            # this is stored in [0] of the dereferenced segment_index. ([1] is the type)
            segment_ref = seg_data[segment_index][0]

            # the data is stored as [[val1],[val2],[val[3]]], so lets get rid of that
            segment = [x[0] for x in mat_file[segment_ref]]
            segment = normalize_segment(segment, RESAMPLED_LENGTH)
            # above statement provides little performance hit. 

    return segments
  • \$\begingroup\$ Could you add the normalize_segment method for context? It would also be useful if you could check how the time taken varies with the number of segments (say, with 500, 1000, 2000 segments)---that would characterise the rough complexity of your process. If you could determine the difference between reading the first 1000 segments and reading the whole of a 1000-segment file, that would also be interesting information. \$\endgroup\$ – David Morris Feb 21 '16 at 21:04
  • 1
    \$\begingroup\$ Have you tried the hdf5storage package's loadmat? It might simplify this considerably. \$\endgroup\$ – TheBlackCat Apr 22 '16 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.