Implementing CI/CD and continuous deployment
Earlier, we learned that one of the key DevOps practices is the process of continuous integration and continuous delivery, also known as CI/CD. In fact, behind the acronyms of CI/CD, there are three practices:
- Continuous integration (CI)
- Continuous delivery (CD)
- Continuous deployment
What does each of these practices correspond to? What are their prerequisites and best practices? Where are they applicable?
Let's look at each of these practices in detail, starting with continuous integration.
Continuous integration (CI)
In the following definition given by Martin Fowler, three key things are mentioned – members of a team, integrate, and as quickly as possible:
That is, CI is an automatic process that allows you to check the completeness of an application's code every time a team member makes a change. This verification must be done as quickly as possible.
We see DevOps culture in CI very clearly, with the spirit of collaboration and communication, because the execution of CI impacts all members in terms of work methodology and therefore collaboration; moreover, CI requires the implementation of processes (branch, commit, pull request, code review, and so on) with automation that is done with tools that have been adapted to the whole team (Git, Jenkins, Azure DevOps, and so on). Finally, CI must run quickly to collect feedback on code integration as soon as possible and hence be able to deliver new features more quickly to users.
Implementing CI
Therefore, to set up CI, it is necessary to have a Source Code Manager (SCM) that will centralize the code of all members. This code manager can be of any type: Git, SVN, or Team Foundation Version Control (TFVC). It's also important to have an automatic build manager (CI server) that supports continuous integration, such as Jenkins, GitLab CI, TeamCity, Azure Pipelines, GitHub Actions, Travis CI, and Circle CI.
Note
In this book, we will use Git as an SCM, and we will look a little more deeply into its concrete uses.
Each team member will work on the application code daily, iteratively, and incrementally (such as in agile and scrum methods). Each task or feature must be partitioned from other developments with the use of branches.
Regularly, even several times a day, members archive or commit their code and preferably with small commits (trunks) that can easily be fixed in the event of an error. This will be integrated into the rest of the code of the application, with the rest of the commits of the other members.
Integrating all the commits is the starting point of the CI process.
This process, which is executed by the CI server, needs to be automated and triggered at each commit. The server will retrieve the code and then do the following:
- Build the application package – compilation, file transformation, and so on
- Perform unit tests (with code coverage)
Note
It is also possible to enrich this process with static code and vulnerability analysis, which we will look at in Chapter 12, Static Code Analysis with SonarQube, which is dedicated to testing.
This CI process must be optimized as soon as possible so that it can run fast, and so that developers can gather quick feedback on the integration of their code. For example, code that has been archived and does not compile or whose test execution fails can impact and block the entire team.
Sometimes, bad practices can cause tests to fail during CI. To deactivate this test's execution, you must take it is not serious, it is necessary to deliver quickly, or the code that compiles it is essential as an argument.
On the contrary, this practice can have serious consequences when the errors that are detected by the tests are revealed in production. The time that's saved during CI will be lost on fixing errors with hotfixes and redeploying them quickly, which can cause stress. This is the opposite of DevOps culture as there's poor application quality for end users and no real feedback; instead of developing new features, we spend time correcting errors.
With an optimized and complete CI process, the developer can quickly fix their problems and improve their code or discuss it with the rest of the team and commit their code for a new integration. Let's look at the following diagram:
Figure 1.2 – The continuous integration workflow
This diagram shows the cyclical steps of continuous integration. This includes the code being pushed into the SCM by the team members and the build and test being executed by the CI server. The purpose of this process is to provide rapid feedback to members.
Now that we've seen what continuous integration is, let's look at continuous delivery.
Continuous delivery (CD)
Once continuous integration has been completed, the next step is to deploy the application automatically in one or more non-production environments, which is called staging. This process is called continuous delivery (CD).
CD often starts with an application package being prepared by CI, which will be installed based on a list of automated tasks. These tasks can be of any type: unzip, stop and restart service, copy files, replace configuration, and so on. The execution of functional and acceptance tests can also be performed during the CD process.
Unlike CI, CD aims to test the entire application with all of its dependencies. This is very visible in microservice applications composed of several services and APIs; CI will only test the microservice under development, while once deployed in a staging environment, it will be possible to test and validate the entire application, as well as the APIs and microservices that it is composed of.
In practice, today, it is very common to link CI to CD in an integration environment; that is, CI deploys at the same time in an environment. This is necessary so that developers can not only execute unit tests but also verify the application as a whole (UI and functional) at each commit, along with the integration of the developments of the other team members.
It is important that the package that's generated during CI, which will also be deployed during CD, is the same one that will be installed on all environments, and this should be the case until production. However, there may be configuration file transformations that differ, depending on the environment, but the application code (binaries, DLL, Docker images, and JAR) must remain unchanged.
This immutable, unchangeable character of the code is the only guarantee that the application that's verified in an environment will be of the same quality as the version that was deployed in the previous environment, and also the same one that will be deployed in the next environment. If changes (improvements or bug fixes) are to be made to the code following verification in one of these environments, once done, the modifications will have to go through the CI and CD cycle again.
The tools that are set up for CI/CD are often used with other solutions, as follows:
- A package manager: This constitutes the storage space of the packages generated by CI and recovered by CD. These managers must support feeds, versioning, and different types of packages. There are several on the market, such as Nexus, ProGet, Artifactory, and Azure Artifacts.
- A configuration manager: This allows you to manage configuration changes during CD; most CD tools include a configuration mechanism with a system of variables.
In CD, deploying the application in each staging environment is triggered as follows:
- It can be triggered automatically, following a successful execution in a previous environment. For example, we can imagine a case where the deployment in the pre-production environment is automatically triggered when the integration tests have been successfully performed in a dedicated environment.
- It can be triggered manually, for sensitive environments such as the production environment, following manual approval by the person responsible for validating the proper functionality of the application in an environment.
What is important in a CD process is that the deployment to the production environment – that is, to the end user – is triggered manually by approved users.
Figure 1.3 – The continuous delivery workflow
The preceding diagram clearly shows that the CD process is a continuation of the CI process. It represents the chain of CD steps, which are automatic for staging environments but manual for production deployments. It also shows that the package is generated by CI and stored in a package manager, and that it is the same package that is deployed in different environments.
Now that we've looked at CD, let's look at continuous deployment practices.
Continuous deployment
Continuous deployment is an extension of CD, but this time, with a process that automates the entire CI/CD pipeline from the moment the developer commits their code to deployment in production through all of the verification steps.
This practice is rarely implemented in enterprises because it requires a variety of tests (unit, functional, integration, performance, and so on) to be covered for the application. Successfully executing these tests is sufficient to validate the proper functionality of the application regarding all of these dependencies. However, it also allows you to automatically deploy to a production environment without any approval action required.
The continuous deployment process must also take into account all of the steps to restore the application in the event of a production problem.
Continuous deployment can be implemented by using and implementing feature toggle techniques (or feature flags), which involves encapsulating the application's functionalities in features and activating its features on demand, directly in production, without having to redeploy the code of the application.
Another technique is to use a blue-green production infrastructure, which consists of two production environments, one blue and one green. First, we deploy to the blue environment, then to the green one; this will ensure that no downtime is required.
Figure 1.4 – The continuous deployment workflow
Note
We will look at the feature toggle and blue-green deployment usage in more detail in Chapter 15, Reducing Deployment Downtime.
The preceding diagram is almost the same as that of CD, but with the difference that it depicts automated end-to-end deployment.
CI/CD processes are therefore an essential part of DevOps culture, with CI allowing teams to integrate and test the coherence of its code and to obtain quick feedback regularly. CD automatically deploys on one or more staging environments and hence offers the possibility to test the entire application until it is deployed in production.
Finally, continuous deployment automates the ability to deploy the application from commit to the production environment.
Note
We will learn how to implement all of these processes in practice with Jenkins, Azure DevOps, and GitLab CI in Chapter 7, Continuous Integration and Continuous Delivery.
In this section, we discussed the practices that are essential to DevOps culture, which are continuous integration, continuous delivery, and continuous deployment.
In the next section, we will look at another DevOps practice, known as IaC.