Efficient image generation with nodejs

Question

I have an app written with node.js, I have a function which takes an image file name and generates a few variations of this image, code below. The issue I am having is that when I have to generate resized images from multiple original images(20+) it sucks up all of the resources on my developing machine, CPU bumps to 100% as well as RAM. While If I run this on the server the process is killed since it sucks up all resources. I am using gm npm module which works on top of GraphicsMagick and ImageMagick. The way I use them is pretty identical to the example on their page.

My developing machine is macbook air with 8gb ram and 1.7GHz i7 processor. My server is 10$ droplet on digitalocean with 1 GB / 1 CPU

On both I have

• Node v6.10.1
• NPM v4.2.0
• GraphicsMagick v1.3.25
• ImageMagick v6.9.8-3

I know that generating images is not an effortless task, but I am sure this should not be taking so much resources and break the app each time I have to generate images from 20+ originals on the server. I tried other npm packages like jimp, node-imagemagick-native, imagemagick and lwip. But the outcome is the same, my question is there something I can do to optimize the resources usage of the machine it is running on?

'use strict';
const path = require('path');
const gm = require('gm');
const debug = require('debug');

module.exports = (imgName) => {

  let sizes = [{
    width: 150,
    height: 150
  }, {
    width: 150,
    height: 150,
    crop: true,
    shortName: 'thumb'
  }, {
    width: 300,
    height: 300
  }, {
    width: 300,
    height: 300,
    crop: true,
    shortName: 'grid-300'
  }, {
    width: 500,
    height: 500
  }, {
    width: 500,
    height: 500,
    crop: true,
    shortName: 'grid-500'
  }, {
    width: 900,
    height: 600
  }, {
    width: 1400,
    height: 900
  }];

  sizes.forEach((size) => {

    if (imgName === undefined) return;

    let w = size.width;
    let h = size.height ? size.height : size.width;
    let ext = path.extname(imgName);
    let name = path.basename(imgName, ext);
    let suffix = `${w}x${h}`;
    let readPath = path.join(__dirname, '../uploads/' + imgName);

    if (typeof size.shortName === 'string') suffix = size.shortName;

    let relativePath = `../public/img/place/${name}-${suffix}.jpg`;
    let newPath = path.join(__dirname, relativePath);

    if (size.crop === true) {

      gm(readPath)
        .resize(w, h, '^')
        .gravity('Center')
        .crop(w, h)
        .quality(90)
        .write(newPath, (err) => {
          if (err) debug(err);
          return;
        });

    } else {

      gm(readPath)
        .resize(w, h)
        .quality(90)
        .write(newPath, (err) => {
          if (err) debug(err);
          return;
        });

    }

  });

};

Update:

• in the uploads folder I have 40 images from 500kb to 2mb with resolution around 2000x3000px. The sizes for generated images are listed in the array.

Answer 1

The used resources going up is to be expected. Re-calculating big images on the CPU is ressource hungry. Computers and software are usually expected to get a task done as fast as possible, so a 100% CPU spike is a good thing. Same applies to RAM. The problem I see is rather the fuzzy limits your hosting provider applies. Find out more about what exactly is causing the processes to be killed. If it is a rule like CPU usage > 90% for over a minute, then try with a break between the images or every X seconds.

Answer 2

Rather than using timing intervals, you may want to consider using promises/deferreds to process your queue. By doing this, you don't have wasted time gaps where you are not doing work. You could also experiment with the number of items you process of the queue at a given time to find the right balance between execution time and resource consumption.