Node.js – How to Write a For Loop With Callbacks

Let’s say you have 10 files that you need to upload to your web server. 10 very large files. You need to write an upload script because it needs to be an automated process that happens every day.

You’ve decided you’re going to use Node.js for the script because, hey, it’s cool.

Let’s also say you have a magical upload function that can do the upload:

upload('myfile.ext', function(err){
  if( err ) {
    console.log('yeah, that upload didn't work: '+err)
  }
})

This upload function uses the callback pattern you’d expect from Node. You give it the name of the file you want to upload and then it goes off and does its thing. After while, when it is finished, it calls you back, by calling your callback function. It passes in one argument, an err object. The Node convention is that the first parameter to a callback is an object that describes any errors that happened. If this object is null, then everything was OK. If not, then the object contains a description of the error. This could be a string, or a more complex object.

I’ll write a post on that innards of that upload function – coming soon!

Right, now that you have your magical upload function, let’s get back to writing a for loop.

Are you a refugee from Javaland? Here’s the way you were thinking of doing it:

var filenames = [...]

try {
for( var i = 0; i < filenames.length; i++ ) { upload( filenames[i], function(err) { if( err ) throw err }) } } catch( err ) { console.log('error: '+err) } Here's what you think will happen: 1. upload each file in turn, one after the other 2. if there's an error, halt the entire process, and throw it to the calling code Here's what you just did: 1. Started shoving all 10 files at your web server all at once 2. If there is an error, good luck catching it outside that for loop – it's gone to the great Event Loop in the sky Node is asynchronous. The upload function will return before it even starts the upload. It will return back to your for loop. And your for loop will move on to the next file. And the next one. Is your website a little unresponsive? How about your net connection? Things might be a little slow when you push all those files up at the same time. So you can't use for loops any more! What's a coder to do? Bite the bullet and recurse. It's the only way to get back to what you actually want to do. You have to wait for the callback. When it is called, only then do you move on to the next file. That means you need to call another function inside your callback. And this function needs to start uploading the next file. So you need to create a recursive function that does this. It turns out there's a nice little recursive pattern that you can use for this particular case:

var filenames = [...]

function uploader(i) {
  if( i < filenames.length ) {
    upload( filenames[i], function(err) {
      if( err ) {
        console.log('error: '+err)
      }
      else {
        uploader(i+1)
      }
    })
  }
}
uploader(0)

Do you see the pattern?

repeater(i) {
  if( i < length ) {
     asyncwork( function(){
       repeater( i + 1 )
     })
  }
}
repeater(0)

You can translate this back into a traditional for(var i = 0; i < length; i++) loop quite easily: repeater(0) is var i = 0,
if( i < length ) is i < length, and
repeater( i + 1 ) is i++

When it comes to Node, the traditional way of doing things can mean you lose control of your code. Use recursion to get control back.