I have the following fully working code that:
- Walks through all sub-folders in a given directory;
- Captures dirpath and filename for the compressed file (i.e., Dir/*.json.bz2);
- Creates a decompressed file with the same dirpath and new file ending (i.e., Dir/*.json). Notably, each newly created decompressed file has a unique numbered filename (i.e., 1.json, 2.json, ..., n.json);
- Decompresses BZ2 archives and writes them on disk.
I tested the code using 1,380 BZ2 archives (~1.5GB) and it takes substantial time to process them. Considering that my goal is to decompress 40,000 BZ2 archives (~45Gb) per run, please suggest if there is a plausible way to optimize the code in order to increase its speed. Thanks!
import os
file_counter = 0
for dirpath, dirname, files in os.walk('/Users/mymac/Documents/Dir'):
for filename in files:
file_counter += 1
if filename.endswith('.json.bz2'):
filepath = os.path.join(dirpath, filename)
newfilepath = os.path.join(dirpath, "{0}.json".format(file_counter))
with open(newfilepath, 'wb') as new_file, bz2.BZ2File(filepath, 'rb') as file:
for data in iter(lambda : file.read(100 * 1024), b''):
new_file.write(data)
Additionally, I am curious which option will be more efficient: (A) run the code and then delete all processed BZ2's, or (B) add some lines in the code and delete each decompressed BZ2 while running the code?
find . -iname '*.bz2' -exec bzip2 -d {} \;
\$\endgroup\$