1
\$\begingroup\$

I have an app written with node.js, I have a function which takes an image file name and generates a few variations of this image, code below. The issue I am having is that when I have to generate resized images from multiple original images(20+) it sucks up all of the resources on my developing machine, CPU bumps to 100% as well as RAM. While If I run this on the server the process is killed since it sucks up all resources. I am using gm npm module which works on top of GraphicsMagick and ImageMagick. The way I use them is pretty identical to the example on their page.

My developing machine is macbook air with 8gb ram and 1.7GHz i7 processor. My server is 10$ droplet on digitalocean with 1 GB / 1 CPU

On both I have

  • Node v6.10.1
  • NPM v4.2.0
  • GraphicsMagick v1.3.25
  • ImageMagick v6.9.8-3

I know that generating images is not an effortless task, but I am sure this should not be taking so much resources and break the app each time I have to generate images from 20+ originals on the server. I tried other npm packages like jimp, node-imagemagick-native, imagemagick and lwip. But the outcome is the same, my question is there something I can do to optimize the resources usage of the machine it is running on?

'use strict';
const path = require('path');
const gm = require('gm');
const debug = require('debug');

module.exports = (imgName) => {

  let sizes = [{
    width: 150,
    height: 150
  }, {
    width: 150,
    height: 150,
    crop: true,
    shortName: 'thumb'
  }, {
    width: 300,
    height: 300
  }, {
    width: 300,
    height: 300,
    crop: true,
    shortName: 'grid-300'
  }, {
    width: 500,
    height: 500
  }, {
    width: 500,
    height: 500,
    crop: true,
    shortName: 'grid-500'
  }, {
    width: 900,
    height: 600
  }, {
    width: 1400,
    height: 900
  }];

  sizes.forEach((size) => {

    if (imgName === undefined) return;

    let w = size.width;
    let h = size.height ? size.height : size.width;
    let ext = path.extname(imgName);
    let name = path.basename(imgName, ext);
    let suffix = `${w}x${h}`;
    let readPath = path.join(__dirname, '../uploads/' + imgName);

    if (typeof size.shortName === 'string') suffix = size.shortName;

    let relativePath = `../public/img/place/${name}-${suffix}.jpg`;
    let newPath = path.join(__dirname, relativePath);

    if (size.crop === true) {

      gm(readPath)
        .resize(w, h, '^')
        .gravity('Center')
        .crop(w, h)
        .quality(90)
        .write(newPath, (err) => {
          if (err) debug(err);
          return;
        });

    } else {

      gm(readPath)
        .resize(w, h)
        .quality(90)
        .write(newPath, (err) => {
          if (err) debug(err);
          return;
        });

    }

  });

};

Update:

  • in the uploads folder I have 40 images from 500kb to 2mb with resolution around 2000x3000px. The sizes for generated images are listed in the array.
\$\endgroup\$
  • \$\begingroup\$ Since this is not really working code (causes your server process to be shut-down), this probably belongs in stack overflow for troubleshooting/debugging. \$\endgroup\$ – jfriend00 Apr 9 '17 at 0:47
  • \$\begingroup\$ it works since I upscaled the droplet, but I am sure this can be less resources hungry \$\endgroup\$ – AntK Apr 9 '17 at 0:49
  • \$\begingroup\$ You also need to specify what the resolution is of the starting images (pixel size W x H) and what size you're trying to resize to. The memory consumed depends upon both of those image dimensions. Also, how many are you trying to do in a row? \$\endgroup\$ – jfriend00 Apr 9 '17 at 0:49
  • \$\begingroup\$ The used resources going up is to be expected. Re-calculating big images on the CPU is ressource hungry. Computers and software are usually expected to get a task done as fast as possible, so a 100% CPU spike is a good thing. Same applies to RAM. The problem I see is rather the fuzzy limits your hosting provider applies. Find out more about what exactly is causing the processes to be killed. If it is a rule like CPU usage > 90% for over a minute, then try with a break between the images or every X seconds. \$\endgroup\$ – Hubert Grzeskowiak Apr 9 '17 at 1:47
  • \$\begingroup\$ thanks @HubertGrzeskowiak, I failed to mention that I did try to set up the generation of the images with intervals, one image usually takes about 5 seconds to generate, I wrapped the generation in setTimeout with a step of 10 seconds in between resizing the files , but after your comment I tried this with larger interval and the server did not crush while performing the generation of new images, perhaps this can be a practical solution. Please post your comment as an answer so I can mark it as a correct one. \$\endgroup\$ – AntK Apr 9 '17 at 3:22
0
\$\begingroup\$

The used resources going up is to be expected. Re-calculating big images on the CPU is ressource hungry. Computers and software are usually expected to get a task done as fast as possible, so a 100% CPU spike is a good thing. Same applies to RAM. The problem I see is rather the fuzzy limits your hosting provider applies. Find out more about what exactly is causing the processes to be killed. If it is a rule like CPU usage > 90% for over a minute, then try with a break between the images or every X seconds.

\$\endgroup\$
1
\$\begingroup\$

Rather than using timing intervals, you may want to consider using promises/deferreds to process your queue. By doing this, you don't have wasted time gaps where you are not doing work. You could also experiment with the number of items you process of the queue at a given time to find the right balance between execution time and resource consumption.

\$\endgroup\$
  • \$\begingroup\$ Sounds interesting, but I can't think if a way to implement with what I have can you link an example of using promises within loops. Sorry if this is trivial question, I haven't worked with promises much \$\endgroup\$ – AntK Apr 9 '17 at 16:18
  • \$\begingroup\$ @AntK You might use something like this npmjs.com/package/promise-queue either directly or as inspiration. Here is a SO question on this topic as well - stackoverflow.com/questions/36672566/… \$\endgroup\$ – Mike Brant Apr 9 '17 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.