Parsing a CSV file in node.js and express.js

Question

I want to import a CSV file and do some operation on it.

The following is the code I'm using and it is working fine:

 return function (req, res) {
    var rl = require('readline').createInterface({
        input: fs.createReadStream(myFile.csv)
    });

    var lineNumber = 1;
    rl.on('line', function (line) {

        // Validate the first line
        if (lineNumber === 1) {
            if (firstLineIsInvalid()) {
                res.send({
                    status: 'error',
                    message: 'This file deos not have a proper header'
                });
                rl.close();
            }
            lineNumber++;
            return;
        }

        // Validate the 2nd line
        if (isSecondLineInvalid()) {
            res.send({
                status: 'error',
                message: '2nd line must be blank'
            });
            rl.close();
            lineNumber++;
            return;
        }

        // Skip 3rd line
        if (lineNumber === 3) {
            lineNumber++;
            return;
        }

        // Skip the last line, we do not need aggregations
        if (line.indexOf('Total') > -1) {
            lineNumber++;
            return;
        }

        // Start processing
        processLine(line);

        lineNumber++;
    });

    rl.on('close', function () {
        if (!res.headersSent) {
            fs.rename(myFile.csv, archivedFile.csv, function (err) {
                if (err) {
                    res.send({status: 'error',  message: 'Archiving failed, '+ JSON.stringify(err)});
                } else {
                    res.send({status: 'ok'});
                }
            })
        } else {
            fs.rename(myFile.csv, rejectedFile.csv)
        }
        res.end();
    });
}

But I am wondering if there is a better approach for:

• Counting the line numbers and validating the lines based on their position in the file.
• Reacting to the errors, especially those I get in the validation of the first and second lines.

Answer 1

Your code is fine and the approach will work. However, I do have the following comments based on my own experience.

1. In the case of error response, why not use the error object to capture the error details and pass return next(err), something to the effect of:

   //handling errors in your code woud look like
   if(somethingisntright()){
   
       err.message = 'Something is wrong';
      next(err);
   }
   
   //error handler presumably in your app.js
   app.use(function(err, req, res){
   //if you have multiple errors you could articulate based on assigning err.status
       res.json({
           status: 'error'
           message: err.message,
       });
   });
   

   This will reduce the amount of error handling code in the current module and centralize error handling thus reducing maintenance effort in the future.

2. I'm not sure if it is worth it but you may want to consider short circuiting your logic so that the code handling line numbers >= 3 are handled first. It saves some cycles having to evaluate the first two scenarios for every other line regardless.

Answer 2

My answer assumes that myFile.csv's content is not changing.

Currently, you read and parse this file every single time a valid request is made. Now, if the file changes a lot, that is fine. Although you might want to process the changes as it changes, and let Express send back the already-processed data.

If the content of the file does not or does not frequently change, I would highly recommend you process this file whenever it becomes available; probably when you start the server.