1
\$\begingroup\$

I have the following fully working code that:

  1. Walks through all sub-folders in a given directory;
  2. Captures dirpath and filename for the compressed file (i.e., Dir/*.json.bz2);
  3. Creates a decompressed file with the same dirpath and new file ending (i.e., Dir/*.json). Notably, each newly created decompressed file has a unique numbered filename (i.e., 1.json, 2.json, ..., n.json);
  4. Decompresses BZ2 archives and writes them on disk.

I tested the code using 1,380 BZ2 archives (~1.5GB) and it takes substantial time to process them. Considering that my goal is to decompress 40,000 BZ2 archives (~45Gb) per run, please suggest if there is a plausible way to optimize the code in order to increase its speed. Thanks!

import os

file_counter = 0
for dirpath, dirname, files in os.walk('/Users/mymac/Documents/Dir'):
    for filename in files:
        file_counter += 1
        if filename.endswith('.json.bz2'):
            filepath = os.path.join(dirpath, filename)
            newfilepath = os.path.join(dirpath, "{0}.json".format(file_counter))
            with open(newfilepath, 'wb') as new_file, bz2.BZ2File(filepath, 'rb') as file:
                for data in iter(lambda : file.read(100 * 1024), b''):
                    new_file.write(data)

Additionally, I am curious which option will be more efficient: (A) run the code and then delete all processed BZ2's, or (B) add some lines in the code and delete each decompressed BZ2 while running the code?

\$\endgroup\$
5
  • 2
    \$\begingroup\$ Why use Python? If this is all you want to do, it's neater to use a shell script. And quite probably faster as well. \$\endgroup\$
    – kyrill
    Mar 31, 2017 at 0:38
  • \$\begingroup\$ kyrill, thank you for response. I have started my programming endeavors just recently and was not aware of "shell script" approach. I appreciate your suggestion and will explore it in more detail. \$\endgroup\$
    – kiton
    Mar 31, 2017 at 2:37
  • 2
    \$\begingroup\$ You seem to be working on a Mac, not sure if this works there, but you can try find . -iname '*.bz2' -exec bzip2 -d {} \; \$\endgroup\$
    – ChatterOne
    Mar 31, 2017 at 8:05
  • \$\begingroup\$ The command suggested by @ChatterOne worked faster than the Python code. How can I modify it, so that each of the decompressed files (regardless of the sub-folder they are in) gets a unique (non-duplicate) name--e.g., 1.json, 2.json, 3, json, ,.. ,N.json? (similar to what I have in the Python code above). Thanks for help! \$\endgroup\$
    – kiton
    Apr 4, 2017 at 23:36
  • \$\begingroup\$ You should ask that question on stackoverflow, lots of people that know more than me about shell scripting over there. \$\endgroup\$
    – ChatterOne
    Apr 5, 2017 at 6:47

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.