1
\$\begingroup\$

I have a piece of code written in node and I wanted to know if I am in the right direction. Basically I have a site where the homepage lists 6 vehicle cards. Every time the page is requested I do the following:

  1. Take 16 vehicles off neo4j (skip x amount if not first page)
  2. For each of the 16th vehicle objects, check if there exist a directory, if yes then read its content (i.e. vehicle images, max 5 per directory)
  3. Return jade template with all data to client

Initially I was using fs.readdirSync and I have just updated the code to use fs.readdir async version.

My worry is how would the following piece of code that read directory and lists its images including finding out its dimension perform in real life for hundreds of concurrent requests?

vehicles.forEach(function(vehicle) {
  var images = [];
  fs.readdir(__dirname + '/photos/' + v.vehicle_id, function(err, files) {
    if(err && err.code == 'ENOENT') {
      logger.info('No images found for vehicle.');
      modifiedVehicles.push(v);
    } else {
      files.forEach(function(file) {
        if(file.indexOf("thumbnail") != 0) {
          var dimension = sizeOf(__dirname + '/photos/' + v.vehicle_id +'/' + file);
          var temp = {
            filename: file,
            width: dimension.width,
            height: dimension.height
          };
          images.push(temp);
        }
      });
      v.images = images;
      modifiedVehicles.push(v);
    }
  });
});
callback(modifiedVehicles);

As a single request to a page of 16 vehicle cards with only 8 of those 16 vehicles having at least 1 image the time it took is:

benchmark took 5135013 nanoseconds (5.1 milliseconds)

Assuming that I host with VM that runs SSD, is the above the right solution? I am thinking of the following options:

Method 1

I could return the response to the user request without list of images but, then as soon as the page is loaded I send a subsequent req to retrieve list of images for all vehicles

Method 2

Retrieve list of images only when the vehicle card is viewed. Ajax request.

Method 3

When the app is started, for each of most recent vehicle (latest 500) read its images and insert them as key (vehicle_id) and value (array of images) to Redis and then retrieve values from Redis for each http request. Not sure how much of a difference Redis instead of Disk I/O can be.

Method 4

Store filename of of vehicles images in mongodb when the vehicle is created/updated. For each http request query mongodb for all images with vehicle ids IN (...)

Any other method that you think I should consider? please share.

\$\endgroup\$
  • \$\begingroup\$ I think Method 3 is going to be your most performant solution. In fact, Redis might not even be necessary, as opposed to simply storing an array in memory in some shared object. Even if 500 vehicles have 50 photos each, a simple array in memory can easily hold that data and very quickly respond with it without ever touching the file system. If your system will periodically have vehicles added, modified, or deleted, you can synchronize this array to your database at a specified interval, or just update the array whenever this happens (if the app is on the same process). \$\endgroup\$ – M Miller Feb 1 '16 at 1:12
  • \$\begingroup\$ If the app crash due to uncaughtException or any other reason then whatever in the memory will be lost ... I would be using pm2 or forever but, I wonder how long it is going to take to initialize redis with all those file names. I wonder if I store image names of all vehicles in mongodb when they are created/updated, I bet querying mongodb would be faster than i/o. ? \$\endgroup\$ – Raf Feb 1 '16 at 1:18
  • \$\begingroup\$ Welcome to Code Review! Good job on your first question \$\endgroup\$ – SirPython Feb 1 '16 at 1:20
  • \$\begingroup\$ Yes, you certainly should not rely on the data being in-memory. It should reside in a permanent database, and probably something more queryable than Redis (like a RDBMS or MongoDB). I only meant to keep them in an array in memory for speed, and on app startup, load them from the database into memory. \$\endgroup\$ – M Miller Feb 1 '16 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.