Context:
I have a program which stores data to the disk. The data is then reprocessed during some of the iterations. Thus, it needs to store, search and load set of data.
Let's consider the following class Signal
which defines a multiphasic signal:
class Signal:
def __init__(self, amp, fq, phases):
self.amp = amp
self.fq = fq
self.phases = phases
# List of signal objects:
signals = [Signal(0.2, 50, [20, 30]), Signal(10, 200, [20, 30]), Signal(20, 20, [20, 90])]
Based on the list of signals, a file_name is computed:
def file_name(signals):
amplitudes = tuple([S.amp for S in signals])
frequencies = tuple([S.fq for S in signals])
phases = tuple([S.phases for S in signals])
return "A{}_F{}_P{}.pkl".format(amplitudes, frequencies, phases)
For the example above, it would return:
"A(0.2, 10, 20)_F(50, 200, 20)_P([20, 30], [20, 30], [20, 90]).pkl"
As you can see, I'm pickling the files (with _pickle
). Let's now believe that hundreds of files have been stored to the folder: folder
. To check if a specific combination of signals has been computed I'm using:
import itertools
def is_computed(files, signals):
"""
Check if the signals are already computed
"""
return any(file_name(elt) in files for elt in itertools.permutations(signals))
I'm using itertools
since the permutations are relevant, i.e.:
signals = [Signal(0.2, 50, [20, 30]), Signal(10, 200, [20, 30]), Signal(20, 20, [20, 90])]
# IS THE SAME AS:
signals = [Signal(10, 200, [20, 30]), Signal(20, 20, [20, 90]), Signal(0.2, 50, [20, 30])]
Issue:
To get the list of files past to is_computed()
, I'm using: files = os.listdir(folder)
which becomes fairly inefficient as the number of files grows up.
# Folder of 26K files with the size from 1 kB to hundreds of MBs
In: %timeit os.listdir(folder)
Out: 3.75 s ± 842 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question:
How could I make a similar system but efficient?
Thanks for the help!