On the Vitals Choice team, we have divided our product into a number of apps. Each of those apps has one running instance per environment , and we have several environments. So as you might imagine, making sure updated code gets propagated through all those places at the right time can be quite the task. Luckily, we have a Hubot instance which does most of the heavy lifting, but even issuing all the Hubot commands for every single app makes you wonder: shouldn’t there be a way to automate this better?
Considering the problem, I realized that callbacks were the way to go. Hubot comes with an evented system, which we could utilize to force Hubot to only launch one app at a time. Here’s what I came up with, and what I learned along the way.
[Note: code examples have been simplified for readability, and proprietary secrets have been removed.]
Step 1: Implement callbacks for the deploy script
Initially, groundwork needed to be laid for post-deploy callbacks. This was relatively simple to implement (all examples in CoffeeScript):
1 2 3 4 5 6
Step 2: Create a function for each app
After quickly setting up a
robot.respond function (which is how Hubot
responds to a particular chat command), calling a
deployAll function, I
turned to the hard problem of setting up a series of functions to deploy all
apps, one after the other. I had a variable accessible as
robot.brain.PROJECTS, which was a collection of all the app names, so I
thought to iterate through all the apps, each time capturing the previous
function as a callback of the new function. This would effectively create
a stack of functions that would be executed one by one, exactly as I described.
1 2 3 4 5 6 7 8 9 10 11
It all seemed to make sense, until I actually tested it. For some reason, it just tried to deploy the last app again and again. What gives?
Step 2.5: Grokking Scope and Lazy Function Evaluation
It turns out that in the loop, when I redefined
callback, it associated the
callback variable with a new function, but within the function, it didn’t
yet do anything with the line
deploy(app, env, callback). This is because
functions exist within a particular closure; they will be evaluated as though
they were run in the place where they were defined, with access to all local
variables, but those variables are accessed when the function is actually run,
NOT when the function is defined.
Here’s a simpler example:
1 2 3 4 5 6 7
We can see that
i is defined in the outer scope, modified in the inner scope,
and that change persists in the outer scope.
Let’s now look at an example closer to our situation:
1 2 3 4 5
As we can see, the
i inside the function is not fixed, but rather depends on
what happens in its closure at any point before the function is actually
Here’s one last example, with a nested function like we had:
1 2 3 4 5 6 7 8 9 10 11 12 13
onePrinter in the outer scope, we changed what
This is because inside
onePrinter is referenced, that variable
is evaluated only when
cbRoP is actually called. The first time,
prints out 1, but the second time, it prints out “one”. Even within
onePrinter has been redefined.
Step 3: Create a Custom Scope
The solution to our problem was to create a scope where the callback wouldn’t change. Here’s the code:
1 2 3 4 5 6 7 8 9 10 11 12
This is a bit of a mind-bender, so let’s explain piece by piece.
deployFunctionFactorytakes in a reference to an app and a function to use as a callback. It returns a function which, when called, will deploy the app passed in, and use the callback that is passed in. Since
callbackis one of
deployFunctionFactory’s arguments, it has been captured in the scope of the function, and nothing outside can change it in the future.
callbackeach time to a new function which is produced on the spot by
deployFunctionFactory. The right side of the equals sign is evaluated immediately, so
callbackis passed into
deployFunctionFactory, a new function is returned, and that new function is assigned to
The cycle repeats for each app, ultimately generating what is effectively a stack of functions to be called, one after the other.
When the stack of functions is resolved on the last line, it starts by running the anonymous function returned by
deployFunctionFactoryfor the last app in
robot.brain.PROJECTS, since that’s the last function that has been added to the virtual stack. That function calls
deploywith the second-to-last function (which deploys the second-to-last app) as a callback. When the first app deployed is done deploying, this callback is run, deploying the next app in line.
The logic is pretty complex, so here’s a visual representation of what all this code is accomplishing. We will consider a case of 3 apps to keep it simple. First, we build the function inside out:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Now, when we call
callback, the functions will be run outside in. First,
app3 will be deployed. When that’s done, our
deploy function knows to call
the callback, i.e. the next function, deploying app2. When app2 finishes being
deployed, app1 will be deployed. At the end, a message will be sent letting
you know all the apps have been deployed.
Using closures and callbacks properly can be a mentally exhausting endeavor. However, when these factors are properly considered and utilized, you can accomplish some pretty powerful stuff.
I personally had to try a few iterations before I came up with a workable
solution in this case, but the results were quite satisfying, and it got the
job done. In the future, if we add apps to
deployAll function won’t have to be changed; it will just add more layers
to the nested function we’ve built.
Using closures and callbacks, we’ve managed to build a function that will run an arbitrary number of tasks synchronously. Sweet!