I'm building a program to scrape data about hockey games in order to compile statistics. One of the run modes will process all of the games in a single season (1230). Processing a single game consists of scraping, parsing, calculating and persisting gathered data and needs to be performed synchronously, but multiple games can be processed in parallel. The first version of the program would simply try and process all available games at the same time. But this would start to get bogged down trying process any more than about 250 games at once. So, to limit the number of games being processed at any time I came up with a promise based worker pool.

Here is a sample usage:

function collectData(itemId) {
    //snip -- returns promise

pool(10).process(items, collectData);

The pool() function accepts the number of worker threads to create and returns an object with a single function process().

The process() function accepts two parameters: a set of data to be processed and a function which processes a single element from the data set. The worker function must return a promise and will be invoked once for each element of the given data set. The execute function returns a promise which is resolved when all of the elements in the data set have been processed.

Here is the pool implementation:

var Q = require('q');
var _ = require('lodash');

function doWork(dataSet, worker) {

    if (dataSet.length === 0) {
        return Q();

    return worker(dataSet.shift())
    .then(function () {
        return doWork(dataSet, worker);

function buildPool(numWorkers) {

    return {
        process: function (dataSet, worker) {
            var workers = [];

            var dataSetCopy = _.clone(dataSet);
            for (var i = 0; i < numWorkers; i++) {
                workers.push(doWork(dataSetCopy, worker));

            return Q.all(workers);

module.exports = buildPool;

While I've written a lot of client side JS I'm relatively new to CommonJS modules and would appreciate any comments on the module's organization. I would also like to hear thoughts on the efficiency of the implementation and whether Promises are a good way to go about this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.