3
\$\begingroup\$

I have zip bundle, for example, abcd.zip, contains more zips like 1.zip, 2.zip etc. Inside of each child zip there is a .jpg file like 1.jpg, 2.jpg etc. There are so many other files but I need only .jpeg.

I need to extract the .jpeg's and and create a zip it again with same parent name like 1.zip.

This works fine, but just wanted to know if I can make it faster. There will be approx 30,000 zip I need to process.

def fjpeg(file):
    base = os.path.basename(file)
    jp = base[:-4]+".jpg"
    return jp    

def process(bundle):
    z1 = zp.ZipFile(bundle, 'r')
    for z1file in z1.namelist():
        if z1file[-4:] == '.zip':
            z2 = zp.ZipFile(z1.extract(z1file, "tmp"), 'r')
            z3 = os.path.basename(z2.extract(fjpeg(z1file)))
            process_path = "processed" + os.path.sep + os.path.basename(z1file)
            with zp.ZipFile(process_path, 'w', mode) as final:
                final.write(z3)
            z2.close()    
            os.unlink(os.path.join("tmp", z1file))
            os.unlink(z3)
        else:
            continue
    z1.close()
\$\endgroup\$
2
\$\begingroup\$

It is not necessary to create temporary files on disk, as zipfile.ZipFile can work in-memory.

  • Use a cStringIO.StringIO instance to hold a zip file in memory.
  • Use ZipFile.read to read a jpeg file into a str variable.
  • Use ZipFile.writestr to write the jpeg back.

This Stack Overflow question may be useful to you: Unzip nested zip files in python. As mentioned, decompressing zip files requires random access to the archive. If the "bundle" zip stores its contents uncompressed (which is an option), in theory it should be possible to have random access into the files.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Single zip file is as big as 2 gigs \$\endgroup\$ – DevC Dec 16 '14 at 16:49
  • \$\begingroup\$ @DevC If you mean the "bundle" file, I'm not suggesting to pull that into memory, only the temporary files. \$\endgroup\$ – Janne Karila Dec 17 '14 at 6:46
  • \$\begingroup\$ Bundle is approx 10G, and has other zip approx 1-2 gig each... Which at last has jpg approx 25 mb... I can read jpg in memory though \$\endgroup\$ – DevC Dec 17 '14 at 6:57
2
\$\begingroup\$

If @JanneKarila's suggestion to unzip in memory instead of disk didn't seem to work for you, I don't know how to make it faster. But in any case, there's some room for improvement.

Instead of this:

z1 = zp.ZipFile(bundle, 'r')
# ...
z1.close()

It would be better to use with, and you can omit 'r':

with zp.ZipFile(bundle) as z1:
    # ...

The same goes for z2.

It's confusing that z1 and z2 are zip files but z3 is not. So you should rename z3 to something better.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.