Continuous Integration is the name given to processes and practices that are involved in regularly integrating the work of several developers into a shared mainline/repository.
My colleague Martin Fowler has written a popular article (http://martinfowler.com/articles/continuousIntegration.html) in which he defines CI as follows:
"A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily, leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly."
There are some key practices that must be followed to have an effective CI:
Developers check in code to a common version-control repository. This happens regularly, at least once a day. Everything—source code, tests, database migrations scripts, build and release scripts, and so on—that is needed to get the application running is checked in to this common repository.
Automated builds run off the checked-in code. This is where CI servers such as TeamCity come into the picture. CI servers run automated builds whenever there are changes in the version-control repository. Every commit has to go through this process of automated builds.
The builds include the process of testing the checked-in code. This includes running unit tests, code coverage, functional tests, and code inspection among others. All the tests must be cleared for the build to be certified as a fully integrated build.
The automated builds should result in well-tested artifacts/binaries/executables, depending on the type of the project. The artifacts must be easily available for anyone to download, and generally it is the CI server that provides the artifacts. In many setups, an external artifact repository such as Nexus or NuGet is utilized as well.
The automated build process should be as quick as possible. The compilation, testing, and all the other steps to get the artifacts must be short to provide quick feedback.
The automated process involved in getting the application running on a developer's box must be the same as the one used to run the CI.
CI is about visibility. It must be clear what is happening to the builds and at what stage a particular commit is. Broken builds should be clearly highlighted and quickly acted upon. The CI server serves as a dashboard to this kind of activity. In TeamCity, users can quickly see the broken builds; the status of running builds and test failures, if any; communicate to the rest of the team that they are working on fixing a build; and pause the builds when needed.
CI is also about team discipline. The team should ensure that broken builds are fixed as soon as possible and they never check in to a broken build. Developers must make sure that they run the build locally before checking in. Many of these can be attacked by technical solutions. For example, many VCSes allow the setting up of pre-commit hooks, which can be used to see whether the build is broken, and if so, prevent check ins. However, primarily, these are people's issues and have to be fixed appropriately.
Automated deployments to a test environment against which we can run our automated functional tests is a key requirement in CI. The idea is to reuse the artifacts generated previously and deploy it into the environment without having to rebuild them again. The artifacts have to be agnostic of the environment they will be deployed into.
CI brings a lot of value and benefits to the teams practicing it. Some of these include the following:
Integrating code continuously leads to a more predictable and less tense integration process.
Issues and bugs are identified and fixed earlier in the process. The presence of automated tests means that more bugs are caught as soon as a developer checks in the code. Also, it is inevitable that bugs escape the testing net. In such cases, the automated testing process that has been put in place encourages us to add tests so that similar bugs don't escape as well.
One of the main benefits of CI is that you get reliable artifacts available to deploy at any point in time. The artifact has gone through lots of quality checks and can be deployed with more trust and less risk. This means that software can be delivered to the client/users frequently.
CI encourages automation and helps in removing manual processes. As you build the foundation for automated tests and automated deployments as part of your CI, you are enabled to work further on this foundation and improve upon it over time.
Continuous Delivery (CD) is the name given to the processes and practices through which applications are made available to be deployed into production at any time. CD is the natural extension to CI, which is more a developer team activity. CD is about making the built application ready to be deployed into production at any time. CD brings in the development team, the operations (ops) team, and the business together to ensure the application is released to production in a timely and appropriate way.
Continuous deployment is different from Continuous Delivery. The former is about deploying every build into production, while the latter is about making every build available to be deployed into production. The actual deployment depends on various business and ops factors, and hence not every build might end up being deployed into production. When we refer to CD in this book, we are talking about Continuous Delivery.
Note
CI can be seen as a subset of CD. The point where CI ends and where the parts that were introduced from the following CD begin is not clearly defined, and they can vary from one setup to another. It can be generally defined that CI ends with getting out the artifacts needed to deploy the application, and CD adds in the ability to deploy them into production in a reliable manner.
A key part of CD is what's called the build pipeline, which is what we will discuss in the next section.
The idea of a build pipeline is to have your build process separated into various stages so that multiple builds can run at the same time. Each build can be in a different stage of the pipeline, thereby leading to a better throughput of builds.
A build pipeline helps to get fast feedback for the team. The first stage of the pipeline generally does the compilation to produce the binaries and runs the unit tests. The artifacts from this stage are passed on to the later stages. The first stage is expected to be very fast in order to provide quick feedback to the team.
The later stages perform various kinds of testing such as acceptance/functional testing and performance testing. These stages are generally slower due to their nature (for example, functional tests that hit the UI of the application are expected to be slower than the unit tests) and may be sequential or parallel depending on the requirements and/or resources available. The later stages also involve deployments to various environments such as the testing environment, against which we run the aforementioned tests, and User Acceptance Testing (UAT) environments that might be used for manual testing.
The pipeline will culminate in the deployment of the artifacts to production-like environments such as staging and eventually production itself. Not every stage of a build pipeline is automatically triggered. Deployments to most of the environments outside the environments used for automated testing are probably done manually as and when required. The following diagram shows a high-level view of a build pipeline for a project:
The build stage does the compilation, runs the unit tests, and produces the binaries. The binaries are then deployed into a testing environment for CI purposes. Quick smoke tests are then run to verify that the build is stable. Then, longer acceptance/functional tests are run. All the stages so far were automatic. The pipeline branches into a set of stages performing deployments to various environments such as UAT, staging, and production on one side, and, on the other side, a couple of stages performing performance tests after the pipeline is deployed into a performance environment. Deployment to various environments such as UAT is generally manual and is done as needed. The exact implementation of a build pipeline will vary from project to project and team to team. Typically, there are other dependencies such as libraries that come into the picture, but the overall structure should be similar to this.
The build pipeline, apart from aiding in fast builds and quick feedback, also enables you to ensure that only the builds that have gone through the rigorous testing process are finally deployed into production.
The build pipeline is also called the deployment pipeline.