3
\$\begingroup\$

I wrote a node script to traverse a folder of hour-long mp3s and upload them to Mixcloud via their API. It works, but I suspect it's fairly inefficient - the computer it's going to run on at our radio station is an old white Macbook. Would appreciate any insight for how to improve it.

  const restler = require('restler');
  const fs = require('fs');
  const readDir = require('readdir');
  const powerOff = require('power-off');

  const options = {
    folder: 'files',
    completefolder: 'complete',
    accesstoken: 'xxxxxxxxx'
  }

  // shut down computer
  const shutDown = () => {powerOff((err, stderr, stdout) => {
        if(!err && !stderr) {
            console.log(stdout);
        }
    })
  };

  // uploadFile uploads file with restler to mixcloud, if api returns rate limiting object, try again in x seconds
  const uploadFile = (folder, filename) => {
    const filepath = `./${folder}/${filename}`
    fs.stat(`./${folder}/${filename}`, function(err, stats) {
        const size = stats.size;
        restler.post(`https://api.mixcloud.com/upload/?access_token=${options.accesstoken}`, {
            multipart: true,
            data: {
                "mp3": restler.file(`./${folder}/${filename}`, null, size, null, 'audio/mpeg'),
                "name": filename,
                // "unlisted": true

                // more data can be added here depending on changes in workflow, automate images etc
            }
        }).on("complete", function(data) {
            const returned = JSON.parse(data);
            if (returned.error) {
              if (returned.error.type == "RateLimitException") {
                // try again in x seconds
                console.log(`uploading too fast, retrying upload of ${filename}after ${returned.error.retry_after} seconds`);
                setTimeout(() => uploadFile(folder, filename), returned.error.retry_after*1000);
              }
              else {
                console.log('non-rate-limiting error');
                console.log(returned);
              }
            }
            else {
              console.log('Success!');
              console.log(returned);
              // move uploaded files into completed folder
              fs.rename(`./${folder}/${filename}`, `./${options.completefolder}/${filename}`, (err) => {
                if (err) {
                  console.log(err)
                }
                else {
                  counter += 1;
                  console.log(counter);
                  if (counter === files.length) {
                    console.log('done');
                    shutDown();
                  }
                }
              })
            }
        });
    });
  };

  // get all mp3s and upload all of them
  const files = readDir.readSync(`./${options.folder}`, ['**.mp3'] );
  let counter = 0;
  for (var i = 0; i < files.length; i++) {
    uploadFile(options.folder, files[i])
  };
\$\endgroup\$
  • \$\begingroup\$ Do you really need to upload these files over and over again such that script performance is that big of an issue, or is this just a quick and dirty script to do this one time and then future uploads are incremental? \$\endgroup\$ – Mike Brant Oct 25 '16 at 21:36
  • \$\begingroup\$ it is will be run at the end of each day of live broadcasts of our radio station - so anywhere from 6 to 10 hours of content. \$\endgroup\$ – Conan Oct 26 '16 at 0:24
  • \$\begingroup\$ The code is too simple to be suspected of inefficiency, but instead of guessing you should measure the performance. There should be many articles showing how to do that. \$\endgroup\$ – wOxxOm Oct 26 '16 at 3:00
  • 1
    \$\begingroup\$ You should first check what is your limiting factor. Is it really js or is your network connection already working at max ? Are you sending files one at a time or several at the same time? Is the sending the file being compressed or not? \$\endgroup\$ – juvian Oct 26 '16 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.