3
\$\begingroup\$

I recently bought a PSP and wanted to know the best ISO files and wrote a scraper to retrieve games ISOs titles that received a high rating and send them to a csv. Any recommendations as to readability, and/or better code optimizations?

var request = require('request');
var cheerio = require('cheerio');
var json2csv = require('json2csv');
var fs = require('fs');

var url = 'http://goo.gl/cc4HRc';
var pspISOs = [];

request(url, function (error, response, html) {
  if (!error && response.statusCode === 200) {
    var $ = cheerio.load(html);
    $('.gamelist').each(function () {
    var titles = $(this).text();
    var ratings = +parseFloat($(this)
      .parents('tr').find('td')
      .last().text()).toFixed(2);
    pspISOs.push({ title: titles, rating: ratings });
  });
  var topISOs = pspISOs.filter(function (iso) {
    return iso.rating >= 4.81;
  });
  var columns = Object.keys(topISOs[0]);
  json2csv({ data: topISOs, fields: columns }, function (err, csv) {
    if (err) console.log(err);
      fs.writeFile('psp_iso.csv', csv, function (err) {
      if (err) throw err;
        console.log('Saved.');
      });
    });
  }
});
\$\endgroup\$
1
\$\begingroup\$

After revisiting this, and even though I prefer to handle tasks like this with node, it is possible to write a bash script using curl, and/or xpath that accomplishes this same task. The ratings could filtered with a regex pattern within grep just to lay down a few ideas. Again, this is just an alternative solution.

\$\endgroup\$
  • 1
    \$\begingroup\$ Also this : github.com/dinedal/textql \$\endgroup\$ – konijn Jun 26 '14 at 19:11
  • \$\begingroup\$ OMG that is insane! \$\endgroup\$ – user27606 Jun 26 '14 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy