Reading 100,000 data files in Python

Question

I need to manipulate data found in multiple data files (~100,000 files). One single data file has ~60,000 rows and looks something like this:

 ITEM: TIMESTEP
300
ITEM: NUMBER OF ATOMS
64000
ITEM: BOX BOUNDS xy xz yz pp pp pp
7.1651861306114756e+02 7.6548138693885244e+02 0.0000000000000000e+00
7.1701550555416179e+02 7.6498449444583821e+02 0.0000000000000000e+00
7.1700670287454318e+02 7.6499329712545682e+02 0.0000000000000000e+00
ITEM: ATOMS id mol mass xu yu zu 
1 1 1 731.836 714.006 689.252 
5 1 1 714.228 705.453 732.638 
6 2 1 736.756 704.069 693.386 
10 2 1 744.066 716.174 708.793 
11 3 1 715.253 679.036 717.336 
.  . .  .       .       .
.  . .  .       .       .
.  . .  .       .       .

I need to extract the x coordinate of the first 20,000 lines and group it together with the x coordinates found in the other data files.

Here is the working code:

import numpy as np
import glob
import natsort
import pandas as pd

data = []


filenames = natsort.natsorted(glob.glob("CoordTestCode/ParticleCoordU*"))
for f in filenames:
    files = pd.read_csv(f,delimiter=' ', dtype=float, skiprows=8,usecols=[3]).values
    data.append(files)

lines = 20000

x_pos = np.zeros((len(data),lines))

for i in range(0,len(data)):
    for j in range(0,lines):
        x_pos[i][j]= data[i][j]


np.savetxt('x_position.txt',x_pos,delimiter=' ')

The problem is of course the time it will take to do this for all the 100,000 files. I was able to significantly reduce the time by switching from np.loadtxt to pandas.read_csv, however is still too inefficient. Is there a better approach to this? I read that maybe using I/O streams can reduce the time but I am not familiar with that procedure. Any suggestions?

Answer 1

Avoid iterating over rows of a dataframe or array. Avoid copying data.

Process one file at a time...read then write data for each file. There is no need to build a list of all the data.

Perhaps use np.loadtxt() instead of pd.read_csv(). Use skip and max_rows to limit the amount of data read and parsed by np.loadtxt(). Use unpack=True and ndmin=2 so it returns a row instead of a column. Then np.savetxt() will append a '\n' after each row.

Something like this (untested):

import numpy as np
import glob
import natsort

with open('x_position.txt', 'a') as outfile:

    filenames = natsort.natsorted(glob.glob("CoordTestCode/ParticleCoordU*"))

    for f in filenames:
        data = np.loadtxt(f, skiprows=9, max_rows=20000, usecols=3, unpack=True, ndmin=2)

        np.savetxt(outfile, data, fmt="%7.3f")

Presuming the data is stored on hard drives, the average rotational latency for a 7200 rpm hard drive is 4.17ms. 100k files at 4.17ms each is about 417 seconds = almost 7 minutes just to seek to the first sector of all those files. Perhaps using concurrent.futures.ThreadPoolExecutor would let you overlap those accesses and cut down that 7 minutes.

Answer 2

Some quick thoughts:

In pd.read_csv you skip the first 8 rows. But you could also use the nrows parameter to limit to the 20000 first lines in order not to read more lines than necessary. Or an offset of 20000 + 8 if you want.

The second loop feeds an array, and you save the array to a file at the end (np.savetxt). But why not open the file x_position.txt in write mode at the beginning of the loop, and then write to it directly as you loop on the rows. Feeding the array is overhead. If you're dealing with large files this could make a difference. And if an error occurs in the middle, then you could always discard the file and start again.

As for the first loop, I would get rid of data.append(files). Just process the files one by one. Don't build huge lists.

Answer 3

On Windows we can speed up iterating over many files by typically 100x by doing a sleep() after every 10 to 100 file reads and allowing the garbage collection time to close unneeded files and free file handles.