6
\$\begingroup\$

I'm working with thousands of large image files in a regularly updated library. The following script does the job (on average reduces my file size ~95%) but costs me around 25 seconds to compress one image. Obviously, I can just let the script run overnight, but it would be cool if I can shave some time off this process. I'm mostly looking for any unnecessary redundancies or overhead in the script that can be trimmed out to speed up the process. I'm still new to Python, so go easy on me.

from PIL import Image
from pathlib import Path
import os, sys
import glob

root_dir = "/.../"

basewidth = 3500

for filename in glob.iglob(root_dir + '*.jpg', recursive=True):
    p = Path(filename)
    img = p.relative_to(root_dir)
    new_name = (root_dir + 'compressed/' + str(img))
    print(new_name)
    im = Image.open(filename)
    wpercent = (basewidth/float(im.size[0]))
    hsize = int((float(im.size[1])*float(wpercent)))
    im = im.resize((basewidth,hsize), Image.ANTIALIAS)
    im.save(new_name, 'JPEG', quality=40)
\$\endgroup\$
  • \$\begingroup\$ basewidth = 3500 quality=40 can you describe what visual effect you are aiming for ("70%-80% is considered low quality")? What is the expected lifetime of the images, the resolution of the viewports/output devices? Will zoom in be required? \$\endgroup\$ – greybeard Jan 21 at 22:40
  • \$\begingroup\$ What version of python are you using? \$\endgroup\$ – Linny Jan 21 at 22:44
  • \$\begingroup\$ @Linny I'm using Spyder 3.3 through Anaconda \$\endgroup\$ – cburchie Jan 22 at 0:47
  • \$\begingroup\$ @greybeard I'm using this compression to create a library of smaller jpegs that I can quickly index. All of the RAW files are kept and these compressed files maintain their names. That way I can easily find the photo I need in my compressed library instead of overloading my computer searching through the RAW files. This level of compression gets me to a very reasonable image quality given the enormous size of the originals \$\endgroup\$ – cburchie Jan 22 at 0:51
  • \$\begingroup\$ Well, I don't know about PIL (which doesn't seem active lately), but I'd give a lower width (1920 coming to mind) and a more common quality (default 75?) a try. \$\endgroup\$ – greybeard Jan 22 at 1:25
7
\$\begingroup\$

I think what greybeard is getting at in the comments is that you could get away with squishing these images much more than you presently are. It sounds like you're basically using the reduced versions as thumbnails, but these "thumbnails" are almost twice the width (over three times the area) of a standard HD monitor.

Dropping basewidth to 1920 (or possibly much lower still) seems like a good idea. Contrary to greybeard, I think your JPEG "quality" setting is fine, but you could play around with different variations of smaller-vs-larger and crisp-vs-compressed.

A minute of googling suggests that PIL is in fact the normal choice for image handling. Maybe there's something better, but I'll suppose not. Given that the originals are large (50MB? more?), it may simply be that there's a lot of work to be done. That said, there are some things you can try.

  • General hardware: Given that this is a heavy task, you may have some luck just running the same code on a fancier computer. This is an expensive option; even if you feel like throwing money at the problem still do plenty of research first.
  • Storage hardware: One possible bottleneck is the hard drive you're reading from and writing to. If you can easily try the task against a different (faster or slower) harddrive while keeping everything else the same, that may be informative.
  • Memory: Does you computer have a lot of RAM? Is that memory available for this python script to use? If you have a way of making half the currently available memory unavaliable, check if that makes the process twice as slow.
  • Memory leaks: I see the line im = Image.open(filename), and looking at the docs suggests that you should probably have a call to load or more likely close someplace.
  • Parallelism: Your computer probably has multiple processor cores. It's hard to say if or how well this script is using all of those cores. If you try running your script twice at the same time (against non-overlapping target sets), how much slower is each concurrent process?
  • It's not clear if ANTIALIAS is a valid option to the resize method in the 3.x version. Try NEAREST.
  • Oh hey there's a thumbnail function that affects the way the file is read from disk!. Playing around with the underlying draft function may also be helpful.

And because this is code review:

  • Move your format and quality constants up to the same place as basewidth.
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Thank you! You've left some really useful tips. I have a few followup questions: 1) Is there any way you can think of to do this compression without read/write? The originals and compressed files are actually both stored on a cloud server, so I imagine that is causing a significant hindrance. 2) I realized my code strips metadata. Do you have any suggestions on how it can be maintained? \$\endgroup\$ – cburchie Jan 23 at 17:41
  • \$\begingroup\$ 1) You can't work on data without reading it and you can't save your work without writing it. You could move the data locally for processing, or you could see if the cloud in question has some way for you to run your script on their machines. Or you could see if some permutation of Image.[load | thumbnail | draft] makes a difference. \$\endgroup\$ – ShapeOfMatter Jan 23 at 19:18
  • \$\begingroup\$ 2) I don't. Searching in the PIL docs left me unclear if PIL has awareness of metadata (and could therefore help) or not. But am I right that what you really want is for the "thumbnails" to be metadata of the originals? \$\endgroup\$ – ShapeOfMatter Jan 23 at 19:20
  • \$\begingroup\$ A few things. First of all, using NEAREST instead of ANTIALIAS helped immensely. I want to maintain as much resolution as possible to allow for maximum zoom on the compressed jpeg, so this has actually allowed me to bump UP my basewidth to ~5,000. Additionally, it's not clear in the PIL documentation but there is a very easy solution to copying exif data. I'm putting my new script as an answer \$\endgroup\$ – cburchie Jan 23 at 20:29
  • \$\begingroup\$ There are at least 3 alternatives to PIL as far as I am aware: Pillow, PIL2, Pillow2. Couldn't say which to prefer though. \$\endgroup\$ – ades Apr 2 at 10:10
3
\$\begingroup\$

Here is my updated script. Notable changes include the use of NEAREST instead of ANTIALIAS, as well as the inclusion of an EXIF copy and paste. I think the major hang on the original script was the inefficiency of ANTIALIAS, as this script gives me around 95% compression in about 2 seconds per image.

from PIL import Image
from pathlib import Path
import os, sys
import glob

root_dir = "/.../"

basewidth = 5504 #sets base width of new images

for filename in glob.iglob(root_dir + '*.jpg', recursive=True): #creates for loop to refeence all .jpg files in root directory
    p = Path(filename) #converts filename into Path object
    img = p.relative_to(root_dir) #uses Path function to parse out Path into components, then uses all components following that equal to the root_dir path name (in this case, our jpeg file names)
    new_name = (root_dir + 'compressed/' + str(img)) #creates new path to save compressed files to in subfolder "compressed" (note: must create subfolder before running)
    print(new_name)

    #resize and reduce
    im = Image.open(filename) #sets filename as object
    wpercent = (basewidth/float(im.size[0])) #uses the base width to establish aspect ratio
    hsize = int((float(im.size[1])*float(wpercent))) #scales image height using aspect ratio
    im = im.resize((basewidth,hsize), Image.NEAREST) #sets new resolution using basewidth and hsize
    exif = im.info['exif'] #copy EXIF data
    im.save(new_name, 'JPEG', exif = exif, quality=40)

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.