6
\$\begingroup\$

I need to manipulate data found in multiple data files (~100,000 files). One single data file has ~60,000 rows and looks something like this:

 ITEM: TIMESTEP
300
ITEM: NUMBER OF ATOMS
64000
ITEM: BOX BOUNDS xy xz yz pp pp pp
7.1651861306114756e+02 7.6548138693885244e+02 0.0000000000000000e+00
7.1701550555416179e+02 7.6498449444583821e+02 0.0000000000000000e+00
7.1700670287454318e+02 7.6499329712545682e+02 0.0000000000000000e+00
ITEM: ATOMS id mol mass xu yu zu 
1 1 1 731.836 714.006 689.252 
5 1 1 714.228 705.453 732.638 
6 2 1 736.756 704.069 693.386 
10 2 1 744.066 716.174 708.793 
11 3 1 715.253 679.036 717.336 
.  . .  .       .       .
.  . .  .       .       .
.  . .  .       .       .

I need to extract the x coordinate of the first 20,000 lines and group it together with the x coordinates found in the other data files.

Here is the working code:

import numpy as np
import glob
import natsort
import pandas as pd

data = []


filenames = natsort.natsorted(glob.glob("CoordTestCode/ParticleCoordU*"))
for f in filenames:
    files = pd.read_csv(f,delimiter=' ', dtype=float, skiprows=8,usecols=[3]).values
    data.append(files)

lines = 20000

x_pos = np.zeros((len(data),lines))

for i in range(0,len(data)):
    for j in range(0,lines):
        x_pos[i][j]= data[i][j]


np.savetxt('x_position.txt',x_pos,delimiter=' ')

The problem is of course the time it will take to do this for all the 100,000 files. I was able to significantly reduce the time by switching from np.loadtxt to pandas.read_csv, however is still too inefficient. Is there a better approach to this? I read that maybe using I/O streams can reduce the time but I am not familiar with that procedure. Any suggestions?

\$\endgroup\$
1
  • \$\begingroup\$ what do you mean by too inefficient? How long does it take and how much time do you have? How close to the hardware (read limit of your harddrive) are you? \$\endgroup\$
    – lalala
    Aug 28, 2021 at 12:06

3 Answers 3

10
\$\begingroup\$

Avoid iterating over rows of a dataframe or array. Avoid copying data.

Process one file at a time...read then write data for each file. There is no need to build a list of all the data.

Perhaps use np.loadtxt() instead of pd.read_csv(). Use skip and max_rows to limit the amount of data read and parsed by np.loadtxt(). Use unpack=True and ndmin=2 so it returns a row instead of a column. Then np.savetxt() will append a '\n' after each row.

Something like this (untested):

import numpy as np
import glob
import natsort

with open('x_position.txt', 'a') as outfile:

    filenames = natsort.natsorted(glob.glob("CoordTestCode/ParticleCoordU*"))

    for f in filenames:
        data = np.loadtxt(f, skiprows=9, max_rows=20000, usecols=3, unpack=True, ndmin=2)

        np.savetxt(outfile, data, fmt="%7.3f")

Presuming the data is stored on hard drives, the average rotational latency for a 7200 rpm hard drive is 4.17ms. 100k files at 4.17ms each is about 417 seconds = almost 7 minutes just to seek to the first sector of all those files. Perhaps using concurrent.futures.ThreadPoolExecutor would let you overlap those accesses and cut down that 7 minutes.

\$\endgroup\$
3
  • \$\begingroup\$ Thank you so much for the useful reply! However, I read in multiple forums that pandas is way faster than numpy.loadtxt due to the fact that the padas library is built in C. Any specific reason you recommend for switching back to np.loadtxt ? \$\endgroup\$ Aug 27, 2021 at 15:25
  • \$\begingroup\$ @AlessandroPerego Mostly, I was trying to avoid the need tp copy the data from a DataFrame to an ndarray. Try it both ways and see which is faster. \$\endgroup\$
    – RootTwo
    Aug 27, 2021 at 16:20
  • 1
    \$\begingroup\$ @Seb The idea is that when one thread is waiting for IO, another thread can do some non-IO processing. For example, when one thread is waiting for the hard drive to return the next chunk of data, another thread can be parsing a chunk of data it read from a different file. Generally, use threads for IO-bound jobs and processes for CPU-bound jobs, but it depends on the actual workload. Try threads and processes and see which is faster. Another possibility is async using something like the aiofile library, but I haven't used it. \$\endgroup\$
    – RootTwo
    Aug 27, 2021 at 16:30
6
\$\begingroup\$

Some quick thoughts:

In pd.read_csv you skip the first 8 rows. But you could also use the nrows parameter to limit to the 20000 first lines in order not to read more lines than necessary. Or an offset of 20000 + 8 if you want.

The second loop feeds an array, and you save the array to a file at the end (np.savetxt). But why not open the file x_position.txt in write mode at the beginning of the loop, and then write to it directly as you loop on the rows. Feeding the array is overhead. If you're dealing with large files this could make a difference. And if an error occurs in the middle, then you could always discard the file and start again.

As for the first loop, I would get rid of data.append(files). Just process the files one by one. Don't build huge lists.

\$\endgroup\$
0
\$\begingroup\$

On Windows we can speed up iterating over many files by typically 100x by doing a sleep() after every 10 to 100 file reads and allowing the garbage collection time to close unneeded files and free file handles.

\$\endgroup\$
1
  • 5
    \$\begingroup\$ This may be true and helpful, but it would be more of a code review if you refocused it on the code at hand. I.e. point out that it doesn't allow time for garbage collection (preferably with a link to a reference explaining how that works) and show how the code would have to change. \$\endgroup\$
    – mdfst13
    Aug 27, 2021 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.