Note: Part III assumes familiarity with the idea of feature flags, and their practical implementation using the rollout gem. For the theoretical background, see Part I. For more info about rollout, see Part II.
You have feature flags in your application, letting you turn off calls to external services as necessary. How can we make this happen automatically?
Degrade: An Amputation Tool
Degrade exists to cut off error-prone external services when they fail. Let’s go through a demo of how we might use it.
Before starting, we should note that redis is absolutely required to use degrade.
Install the gem
gem install degrade or add
gem "degrade" to your Gemfile and
Next, assuming you have a
redis variable which is an instance of
1 2 3 4 5 6 7 8 9 10 11
Whoa, that’s a lot of options! Most pieces are optional and should be
fine-tuned to the needs of your application. The important things are: you need
to pass in a
redis object, give your degrade object a
name, and feed it a
failure_strategy lambda which will run if the error threshold is reached.
Let’s discuss how to use your shiny new degrade object!
Wrap a call
degrade object comes with a
#perform method that takes a block, thusly:
1 2 3
It will run the block without impacting the return value, only pausing to mark
whether the block raised one of the specified errors. As you can see from
the source code for
1 2 3 4 5 6 7 8 9
It simply marks that a request is being made, and yields to your passed-in
block. If all goes well,
perform will return the return value of the block.
If an error occurs, and it’s one of the errors you’ve selected to track, degrade
will rescue, mark that this request ended in failure, and re-raise the error.
It’s important to understand that degrade won’t automatically turn off a feature
for you. All it does is run the
failure_strategy you’ve given it. You can
use something like rollout to handle the failure situation. Here is an example:
1 2 3 4 5 6 7 8
You can, of course, cut off the request earlier in your code. You might turn off a controller action entirely if the external service it needs is down.
failure_strategy you pass to degrade is just a plain old lambda, so you
can pass in any code that you want. Let’s say you create an
module which can send an email/text when the external service goes down. Just
call that code from the
failure_strategy. You can also set up a Sidekiq
worker that will reactivate the service (i.e., try again) after a specified
period of time.
Here’s how you might do all that in one
1 2 3 4 5 6 7 8 9
There is one catch to be aware of when reactivating. Degrade does not
automatically reset the sample when it runs the
failure_strategy. It has a
private method to do that (after it hits the sample size max), but we’ll respect
that method’s privacy. Here’s what that method looks
1 2 3 4 5 6
If we dive deeper, we’ll see that
1 2 3 4 5 6 7
So if we copy those methods, all we have to do to reset the sample is copy that
code into our worker. Here’s a simplified Sidekiq worker, where
reset the sample and then reactivate the feature flag:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Now you can use the line of code:
to reactivate the feature after a 30-minute delay. If you’re really clever, you can even set it up to increase the delay each time. It all depends on the needs of your application.
In this series, we’ve learned
what feature flags are
how to use feature flags with the rollout gem
how to use degrade, building on rollout, to shut off external services as necessary
Let me know about your experiences in the comments!
To learn more about James’s life and the circumstances surrounding his untimely passing, see the links below the SpeakerDeck.