Application deployments have usually meant deploying a new version of an artifact to Production, with stops along the way in Test, Staging, etc. All the new features and bug fixes become active when you deploy this new version. Although you’ve thoroughly tested the application with automated unit tests, functional tests, load tests, and performance tests, you’ve only reduced the risk of something going wrong in Production. If and when something does go wrong, you may need to rollback to the previous version. But what if the fix wasn’t as dramatic as a rollback? What if you could revert the one mis-behaving feature? Even better, what if you could select which features to activate? In this article I’ll explain how you can control your deployments with feature flags.
What are Feature Flags?
Feature flags (also known as feature toggles, feature bits or feature flippers) are a design pattern. Martin Fowler is one of the earliest proponents of this pattern. At their core, feature flags are simple if-then constructs that will activate a feature or a behaviour based on a configuration setting. The setting can be in a property file, a database, a web service, or whatever. You change the configuration, and you change the behaviour of your application. The more sophisticated feature flag implementations can activate a feature based on time of day, a configurable proportion of incoming requests, the user’s IP address, or some attribute of the user. A big part of the value you get out of feature flags is the ability to turn on a feature, yet turn it off and revert to previous behaviour if the results are not what you expect.
Now feature flags do not in any way remove the need for thorough testing. You still need to have a full suite of unit, integration, functional, load and performance tests – all automated of course. What feature flags do is further mitigate the risk of the application failing due to a set of circumstances you could not possibly have predicted in the development and test phases.
Trunk-Based Development
Feature flags are a huge help with mainline or trunk-based software development. With feature branch based development, you create a new branch from the mainline for each new feature. This means you need to merge the feature branch into the mainline at some point in the future. This can be problematic with features that take weeks or months to implement. It also delays the feedback process when your build system is only triggered by changes in the mainline. With feature flags, you commit all code to the mainline branch (“main” or “master” in Git). You turn the feature flag off until the feature is ready for testing and Production deployment.
Continuous Delivery
In the words of Dave Farley, Continuous Delivery is the sum of all tools, processes and practices that lead to frequent and rapid deployment of high-quality code changes, from the time the developer pushes code to the central repository until it is delivered to production. An integral part of this is extensive use of automation.
It’s called Continuous Delivery rather than Continuous Deployment or Continuous Release for a reason. It’s because we reserve the decision whether or not to activate the changes. A human makes this decision. That is, the business customer or product owner decides if and when to activate all of or a subset of the new features. This activation can take the form of deploying a specific artifact version after it has completed testing and verification. It can also take the form of feature flags.
Feature flags complement Continuous Delivery quite nicely because you can have an automated deployment pipeline that reaches all the way to Production. It may not be feasible to activate certain features due to legislative compliance, end-user workload or training, or because you want to tie the availability them to a marketing campaign. This is a concept known as the separation of release from deployment. You deploy the new functionality to Production, but release it (make it available) only when you turn on the corresponding the feature flag. You control your deployments with feature flags.
Feature Flag Trade Offs
Like everything else in software architecture, there are some trade offs.
Complexity
Feature flags add complexity. You are adding more if-then logic to your codebase. Some implementations mitigate this by use of Java annotations (Togglz is one example). Regardless, it’s easy to fall into the trap of feature flag proliferation where the number of flags grows to the point where it’s hard to keep track of them all.
Mitigate this complexity by using feature flags only when you have to. If you can, break down the functionality so you can implement it on one short (two-week) release cycle. In addition, remove feature flags as soon as you no longer need them. This will take some team discipline to make sure they are removed, but it will keep your codebase clean.
Latency
Since you are adding logic to check a configuration setting, this adds some latency. Where performance and response time is critical, this can be a concern where the time needed for a file system read or database lookup can make a difference.
Togglz mitigates this by letting you put a caching mechanism between your application and your feature repository. An enterprise-grade feature flag implementation – LaunchDarkly – uses a world-wide content delivery network with streaming and caching mechanisms to ensure your application receives flag updates with minimal delay.
Testing
At first glance, it may look like the testing effort will dramatically increase just to cover all the variables. You would have to write test cases to cover every possible combination of feature flags.
Mitigate this by limiting the number of places where you have a given flag. For instance, if the only way to enter a new feature is through the UI, put the feature flag in the UI only. Furthermore, confine your testing to two scenarios: one where all the features expected for the next release have their flags turned on, and another where all the feature flags are turned on.
Parting Thoughts
You can control your deployments using feature flags. They give you fine-grained control over the behaviour of your application. You can turn on a new behaviour, turn it off instantly if the results are not what you expect, and remove the flag when you are confident the feature is working. Feature flags let you gradually roll out new functionality at a pace suitable for your team and your customers. Mitigating the downsides using a combination of team discipline, tooling, and testing strategy lets you get the best out of this pattern, and ultimately deliver high-quality code more frequently.