Suppose your client or your employer comes to you and says “We need you to help us understand how to set up a continuous delivery pipeline.” They assure you they are committed to the goals and benefits of Continuous Delivery, also known as Continuous Integration/Continuous Deployment (CI/CD):
- The software development teams want to reduce deployment risk.
- They like the idea of frequent small deployments during the workday instead of huge deployments once or twice a year.
- Teams no longer want to dedicate a full weekend to a massive deployment, with all the attendant stresses and risks that brings.
- The business is less tolerant of waiting months before they see even the smallest change; they want a faster time to market.
They have acquired all the tooling, and all they need is your guidance and leadership for how to set up a continuous delivery pipeline.
How would you approach this? What sequence of steps or phases would you use? In this article I’ll present some of my thoughts and suggestions.
What is CI/CD?
In the words of Jez Humble and Dave Farley, authors of the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, Continuous Delivery means “software is always in a releasable state”. CI/CD is the sum of all the tools, processes and engineering disciplines used to deploy high-quality software frequently and rapidly with a high degree of automation.
The idea being when a developer pushes code changes to the central repository, a whole suite of automated processes takes over. These processes run unit tests, build the application, deploy it to one or more test environments for end-to-end functional tests, performance tests, and functional tests. If all those tests pass, another automated process deploys to production. The ultimate goal is, when a developer pushes their code changes, the next human interaction is with the end user using the updated application.
Automation Baby!
As you noticed, CI/CD involves extensive automation. Automated tests run much faster than any human can possibly execute them, and they run consistently. If any test fails, the entire pipeline fails, and nothing is deployed to production. Automated deployment scripts perform the same steps they are told to perform. Every time. They are not subject to human error when they run.
Each time a human needs to perform some portion of a deployment, there is the potential for error. A mis-typed command, commands inadvertently entered in the wrong sequence, or even a command omitted altogether can cause problems with the deployment. With CI/CD you develop scripts or configure tooling to perform the application deployment, database configuration changes (if any), and configuration updates. After launch, these scripts run without human intervention. Every time a developer pushes code to the repository, the automation kicks off.
With this repeated execution of build, test and deploy you are also testing the automation itself. You get the feedback you need to tell you if it’s working. By the time you are ready to deploy all the way to production, the automation has proven itself by having been run hundreds of times deploying to Development, Test and Staging environments.
CI/CD is just as much about process as it is about tooling. It’s a discipline, a consistent approach to software development.
First Steps for Continuous Delivery
Before embarking on the CI/CD journey, it is important to understand why the team and the organization want to achieve CI/CD. If their reasons align with those I’ve written at the beginning of this article, that’s good. On the other hand, if they think it sounds cool based on what they read or heard at a conference, then the team should pause and reflect on whether this is an effort worth undertaking.
That said, begin by ensuring a few processes are in place. These are critical to achieving the benefits of Continuous Delivery. Here are the sequence of steps I follow to help my client achieve CI/CD. Some may be in place already; that’s fine. By going in the order I present here, you can make some steady progress toward the ultimate goal.
Version Control
The source code and configuration must be in version control. Many years ago it was common practice to have the source code on some network share. A developer would take a copy of all or part of the code base, make their changes, and overwrite the master copy on the network. Thankfully, today we have solutions that track who changed what and when. These solutions are also capable of merging changes from numerous copies of the codebase. Most source code repositories these days are based on Git. There are number of free and open source as well as commercial solutions that use Git. GitHub is a popular hosted repository, there are others such as GitLab and Bitbucket.
Having both source and configuration in version control lets you see what changed when, and who made the change. It also lets you rewind the application to a specific point in time whenever you need to debug a problem. This can be useful in situations where the application is under development, and you need to go back to the specific version that is in production in order to solve a problem.
Integrate Continuously
Developers need to be pushing their changes to the central repository frequently. Once a day is the minimum, preferably several times a day. This will minimize or even eliminate merge conflicts when two or more developers are working on the same part of the codebase. In the old days there would be an Integration phase on a project when the team would spend days or even weeks getting the application to compile, build, and run.
I mentioned earlier that the goal of Continuous Delivery is to always have software that is in a releasable state. A corollary to this is the code in the central repository is always in a working state. That way when a developer starts a new task, they can be confident the software was working before they begin. This eliminates so much wasted time figuring out why the application doesn’t build. It also ensures a new developer who joins the team can get up to speed quickly.
There’s a change in developer practice here. Ensure you only push code that passes unit tests and builds successfully. This means before every commit, the developer checks for and pulls any changes others have made into their local copy (git pull). Then run the command to build and run unit tests (mvn clean install or ./gradlew clean build). They deploy to the runtime environment on their local workstation and ensure the application behaves correctly. Then, and only then, does the developer push their changes to the central repository.
Automated Testing
You need solid unit test coverage. The general guideline I use for my teams is once you have achieved 80% to 85% coverage, then you can start thinking you might have enough. All documented requirements need to have unit tests that verify the behaviour of the individual components meet the requirements.
The team’s architect or lead developer needs to review the tests to make sure they are indeed testing for the expected behaviour. I have seen tests that mock out the class under test, or simply assert the class under test doesn’t throw an exception. Needless to say, neither of these are any good.
Automated Builds
You need a dedicated build server that will build your deployment artifacts. This eliminates the problem of “it worked fine on my machine” when you rely on one developer to make the build on their workstation. The build server listens for any commits to the code repository. Each time there is a new commit, it checks out a copy of the code, runs the command that compiles, runs unit tests, builds into a deployment unit (a Docker image, EAR or WAR file), and inserts the deployment unit into the artifact repository. If Maven is your build tool, this would be mvn deploy; with Gradle it is gradle build uploadArchives. The build server also provides a dashboard showing the status of the most recent builds, with links to the test reports. Jenkins, Atlassian’s Bamboo, and AWS CodeBuild are popular examples of build servers.
These build servers often provide for static code analysis. For example, with Jenkins you can install SonarQube’s SonarScanner for Jenkins plugin. Tools like this help immensely with code reviews by looking for common problems, leaving the reviewer to focus on readability of the code.
A bit of a cultural change happens here too. When a build breaks, whoever did the last commit is responsible for fixing it. If a build breaks, the team’s number one priority is to fix it.
From time to time, builds break. When they do, it means the team has discovered a defect or some other problem. This is good, because it means the error was caught at an early stage in the development process. It’s much easier to discover and fix it here than it is later on in Acceptance test.
Architecture
The architecture of your application and associated infrastructure needs to be one that lends itself to automation. Chiefly, this means you are able to use scripts or some sort of tooling to automate the deployments, configure the infrastructure, perform database structural changes and deploy configuration changes.
An example of an automation-ready infrastructure is having multiple instances of the application behind a load balancer. The load balancer itself needs to expose some sort of API or command line interface so the automated deployment can configure it. This gives you the option of a canary deployment, which I’ll talk about in a moment.
Monitoring and Observability
We’ll need some way for applications to report their health, whether it is good or there is a problem. Spring Boot has a number of production-ready features bundled into the library Spring Boot Actuator. Micronaut has a similar one they call Management & Monitoring. Both provide a simple way to achieve the same end: simply add a dependency to the application, and it automatically exposes a set of web endpoints. The one that is most useful is /health. If all is OK, and if the application can talk to its database, it returns an HTTP 200 OK. If not, it returns a 500-series HTTP status code. This is super easy to integrate into even the most basic monitoring tools.
Deployment scripts.
Rather than having a documented series of steps for a human to perform the deployment, automate it. You can write some home-grown scripts, use tools such as Chef or Puppet, or use your cloud provider’s managed services. Use this tooling for all environments, not just Production. Ultimately, you want to run the deployment tool and specify the application name, the version, and the target environment. Everything after that should be performed by the script, from pulling the deployment unit out of the artifact repository, grabbing the configuration appropriate to the target environment, deploying both of these, running any database migration scripts, and performing smoke tests to assert basic functionality. Do this for every environment, and by the time you deploy to Production, you have a tool that has proven itself over many prior deployments to UAT, Staging, Development, etc.
Canary Deployments
A canary deployment is so named because it refers to the practice of coal miners of old placing bird cages with canaries in them throughout the underground mine – the proverbial “canary in the coal mine”. Being more sensitive to toxic gases than humans, the presence of these gases would kill the cararies well before they harmed the humans. This served as an early learning system for the humans, albeit an inhumane one.
Here’s an example of a canary deployment. Suppose you have two Production instances of your application, and the load balancer does a simple round-robin to direct traffic equally between the two instances. With a canary deployment, you deploy the new version of the application to one of the instances; the other remains on the old version. You set the load balancer so that only a small number of users (say 10%) are directed to the new version. Through automation, you monitor the new application for errors. If those errors exceed a set threshold, send all the traffic back to the other instance while you rollback the bad instance to the previous known good version. Conversely, if the new version is working fine with no reported errors, you can have the load balancer gradually increase the traffic to the new instance.
Automated Functional, Performance and Load Testing
When you automate your functional tests, you will notice a dramatic improvement in the testing cycle time. One of my clients had a full suite of manual regression tests that took two or three QA people the better part of two weeks to execute. Thanks to the efforts of our QA lead, he used a test automation tool to bring that down to something like ten minutes. Same set of test cases, they just were automated. Talk about a game changer! Our team’s biggest obstacle to faster and more iterative deployments had been the significant time and effort to execute manual functional tests. No more.
There are several free and open source tools to help you automate your functional tests. Selenium is one tool that can automate web applications that have a user interface. Postman is great for testing APIs.
There are other free open source as well as commercial tools that will help you automate performance and load tests. For these tests to be meaningful, they need to be executed in environments that mimic that of production. These environments need to be sized as close as possible to production in terms of number of servers, memory allocation, storage, etc. Here is where Infrastructure As Code (IaC) and a virtualized infrastructure can help you. By having a set of templates or scripts that define your infrastructure, you can run them to create your production-like test environment, run your performance and load tests, then tear down the environment. AWS provides this IaC capability with AWS CloudFormation, and their Cloud Development Kit. I’m sure Microsoft Azure and Google Cloud Platform have something similar.
Putting the Pipeline Together
Up to this point you’ve automated a good portion of the deployment process. But you probably have a couple manual steps in there. For instance, you’ve automated everything from code push to the central repository up to where the built and unit-tested application is in your artifact repository. Another manual process initiates a deployment automation to the environment of your choice. Automated functional, performance and load tests run after a human initiates each of those. Finally, a human kicks off the automation to deploy to production. This is a very good start. Your team is already seeing the benefits of higher-quality, faster, and more consistent deployments.
Automate up to Functional Tests
To begin assembling your deployment pipeline, begin with your Development environment and functional tests. Once the automated build inserts the deployment unit into your artifact repository, automatically kick off a deployment process that deploys to your Development environment, and runs the automated functional tests. You can do this while your QA people are still working on building out the functional tests. Just take what they have so far and use that in Development. With repeated deploys you’ll get plenty of useful feedback on what works and what doesn’t.
After several repetitions, this automation will have earned your trust. Now extend this automation to deploy to the test environment (or QA, UAT, whatever you call it), and run the full suite of functional tests.
Any time any part of the build and test process fails, the whole deployment run comes to a full stop. All tests must pass before moving on to the next phase.
Automate up to Performance and Load Tests
Now move on to automatically executing the performance and load tests. By this point you will have numerous reports from each type of test so you can verify all the requirements have been satisfied.
So here we are, right before we deploy to Production. What some organizations do at this point is they establish a gate of sorts. That is, to comply with governance and compliance constraints, a human examines all the test reports. If they meet the criteria the organization established, and if the time is right for the business to accept a deployment, someone clicks on a button to begin the automated deployment.
The Last Step: Delivery to Production!
As time goes on, and with more of these successful gated production deployments, you can consider automating the last step. By now your team has a high degree of confidence in the deployment pipeline. There have been occasions where the backout automation I spoke of earlier activated due to a deployment problem. The test reports serve as an effective audit trail to assert the new version complies with governance and compliance. Production deployments have been pretty much a non-event. The time may be right to take this last step.
In Summary
This is one approach for how to set up a continuous delivery pipeline. Building out a pipeline like this takes months, even a year or two for some organizations. What I’ve written here is a sequence of steps I have implemented that have worked well for my clients. You can certainly pause at various points for a couple weeks so the team can reflect on what they have learned, and the lessons they can apply to the next step.
The effort is just as much about people and process as it is about tools and technology. You and your bosses need to cultivate and nurture an environment where learning and experimentation is encouraged and celebrated. Mistakes will happen; it’s important to treat mistakes as learning opportunities for everyone.
At the end of this you can have a deployment pipeline that delivers high-quality software more frequently and with much less stress. It’s a win-win for the development teams and their business customers.