3
\$\begingroup\$

I have N(>1000) .npy files, each has ~80MB, each contains a vector of high dimension (P).

I have to deal with these data with the code below:

G = np.zeros((N, N))
for i in range(N):
    vi = np.load(str(i)+'.npy')
    for j in range(i, N, 1):
        vj = np.load(str(i)+'.npy')
        G[i, j] = process(vi, vj)
        G[j, i] = G[i, j]

post_process(G)

It takes a lot of time depending on reading files and I'd like to accelerate it.

Is there any suggestion?

\$\endgroup\$
1
\$\begingroup\$

Try to cache as much as possible.

I suppose buying more RAM and read all .npy files in advance is out of the question.

But, for instance, if it is possible to keep the data of 20 files in memory, you could read 10 files for vi-data and 10 files for vj-data, now you can fill a block of 10x10 in matrix G at the cost of 20 file reads instead of 10 + 10x10 = 110 file reads. Looping and indexing will get a bit more complicated of course.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.