Continuous Integration (CI), Continuous Testing (CT), and Continuous Delivery (CD) are significant part of DevOps culture. CI includes automating builds, unit tests, and packaging processes while CD is concerned with the application delivery pipeline across different environments. CI and CD accelerate the application development process through automation across different phases, such as build, test, and code analysis, and enable users achieve end-to-end automation in the application delivery lifecycle:
Continuous integration and continuous delivery or deployment are well supported by cloud provisioning and configuration management. Continuous monitoring helps identify issues or bottlenecks in the end-to-end pipeline and helps make the pipeline effective.
Continuous feedback is an integral part of this pipeline, which directs the stakeholders whether are close to the required outcome or going in the different direction.
"Continuous effort – not strength or intelligence – is the key to unlocking our potential"
|-- Winston Churchill
The following diagram shows a mapping of different parts of an application delivery pipeline with the toolset for Java web applications:
We will use a sample Spring application throughout this book for demonstration purposes, which is why the toolset is related to Java.
An automated build helps us create an application build using build automation tools such as Apache Ant and Apache Maven. An automated build process includes the following activities:
Compiling source code into class files or binary files
Providing references to third-party library files
Providing the path of configuration files
Packaging class files or binary files into WAR files in the case of Java
Executing automated test cases
Deploying WAR files on local or remote machines
Reducing manual effort in creating the WAR file
Maven and Ant automate the build process and make it simple, repeatable, and less error prone as it is a create-once-run-multiple-times concept. Build automation is the base of any kind of automation in the application delivery pipeline:
Build automation is essential for continuous integration and the rest of the automation is effective only if the build process is automated. All CI servers, such as Jenkins, Atlassian, and Bamboo use build files for continuous integration and creating their application-delivery pipeline.
What is continuous integration? In simple words, CI is a software engineering practice where each check-in made by a developer is verified by either of the following:
Pull mechanism: Executing an automated build at a scheduled time
Push mechanism: Executing an automated build when changes are saved in the repository
This step is followed by executing a unit test against the latest changes available in the source code repository:
The main benefit of continuous integration is quick feedback based on the result of build execution. If it is successful, all is well; else, assign responsibility to the developer whose commit has broken the build, notify all stakeholders, and fix the issue.
Read more about CI at http://martinfowler.com/articles/continuousIntegration.html .
So why is CI needed? Because it makes things simple and helps us identify bugs or errors in the code at a very early stage of development, when it is relatively easy to fix them. Just imagine if the same scenario takes place after a long duration and there are too many dependencies and complexities we need to manage. In the early stages, it is far easier to cure and fix issues; consider health issues as an analogy, and things will be clearer in this context.
Continuous integration is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
CI is a significant part and in fact a base for the release-management strategy of any organization that wants to develop a DevOps culture.
Following are immediate benefits of CI:
Automated integration with pull or push mechanism
Repeatable process without any manual intervention
Automated test case execution
Coding standard verification
Execution of scripts based on requirement
Quick feedback: build status notification to stakeholders via e-mail
Teams focused on their work and not in the managing processes
Jenkins, Apache Continuum, Buildbot, GitLabCI, and so on are some examples of open source CI tools. AnthillPro, Atlassian Bamboo, TeamCity, Team Foundation Server, and so on are some examples of commercial CI tools.
We will now be looking at best practices that can be useful when considering a continuous integration implementation:
Maintain a code repository such as Git or SVN.
Check-in third-party JAR files, build scripts, other artifacts, and so on into the code repository.
Execute builds fully from the code repository: Use a clean build.
Automate the build using Maven or Ant for Java.
Make the build self-testing: Create unit tests.
Commit all changes at least once a day per feature.
Every commit should be built to verify the integrity of changes.
Authenticate users and enforce access control (authentication and authorization).
Use alphanumeric characters for build names and avoid symbols.
Keep different build jobs to maintain granularity and manage operations in a better way. A single job for all tasks is difficult when trying to troubleshoot. It also helps to assign build execution to slave instances, if that concept is supported by CI server.
homedirectory of the CI server regularly as it contains archived builds and other artifacts too, which may be useful in troubleshooting.
Make sure the CI server has enough free disk space available as it stores a lot of build-related details.
Do not schedule multiple jobs to start at the same time, or use a master-slave concept, where specific jobs are assigned to slave instances so that multiple build jobs can be executed at the same time.
Set up an e-mail, SMS, or Twitter notification to specific stakeholders of a project or an application. It is advisable to use customized e-mails for specific stakeholders.
It is advisable to use community plugins.
Cloud computing is regarded as a groundbreaking innovation of recent years. It is reshaping the technology landscape. With breakthroughs made in appropriate service and business models, cloud computing has expanded to its role as a backbone for IT services. Based on experience, organizations improved from dedicated servers to consolidation and then to virtualization and cloud computing:
Cloud computing provides elastic and unlimited resources that can be efficiently utilized at the time of peak load and normal load with a pay-per-use pricing model. The pay-as-you-go feature is a boon for development teams that have faced resource scarcity for years. It is possible to automate resource provisioning and configuration based on your requirements, which has reduced a lot of manual effort. For more information, refer to NIST SP 800-145, The NIST Definition of Cloud Computing at http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf .
It has opened various opportunities in terms of the availability of application—deployment environments, considering three service models and four deployment models as shown in the following diagram:
There are four cloud deployment models, each addressing specific requirements:
Public cloud: This cloud Infrastructure is available to the general public
Private cloud: This cloud Infrastructure is operated for and by a single organization
Community cloud: This cloud infrastructure is shared by specific community that has shared concerns
Hybrid cloud: This cloud infrastructure is a composition of two or more cloud models
Cloud computing is pivotal if we want to achieve our goals of automation to inculcate DevOps culture in any organization. Infrastructure can be treated similar to code while creating resources, configuring them, and managing resources using configuration-management tools. Cloud resources play an essential role in the successful adoption of DevOps culture. Elastic, scalable, and pay-as-you-go resource consumption enables organizations to use the same type of cloud resources in different environments. The major problems in all the environments are inconsistency and limited capacity. Cloud computing solves this problem as well as those of economic benefits.
Configuration management(CM) manages changes in the system or, to be more specific, the server runtime environment. Let's consider an example where we need to manage multiple servers with same kind of configuration. For example, we need to install Tomcat on each server. What if we need to change the port on all servers or update some packages or provide rights to some users? Any kind of modification in this scenario is a manual and, if so, error-prone process. As the same configuration is being used for all the servers, automation can be useful here. Automating installation and modification in the server runtime environment or permissions brings servers up to spec effectively.
CM is also about keeping track or versions of details related to the state of specific nodes or servers. It is a far better situation when we nee and update themselves. A centralized change can trigger this, or nodes can communicate with the CM server about whether they need to update themselves. CM tools make this process efficient when only changed behavior is updated, and the entire installation and modification isn't applied again to the server nodes.
There are many popular configuration management tools in the market, such as Chef, Puppet, Ansible, and Salt. Each tool is different in the way it works, but the characteristics and end goal are the same: to bring standardized behavior to the state changes of specific nodes without any errors.
Continuous delivery and continuous deployment are used interchangeably more often than not. However, there is a small difference between them. Continuous delivery is a process of deploying an application in any environment in an automated fashion and providing continuous feedback to improve its quality. Continuous deployment, on the other hand, is all about deploying an application with the latest changes to the production environment. In other words, we can say that continuous deployment implies continuous delivery, but the converse isn't true:
Continuous delivery is significant because of the incremental releases after short spans of implementation, or sprint in agile terms. To deploy a feature-ready application from development to testing may include multiple iterations in a sprint due to changes in the requirements or interpretation. However, at the end of a sprint, the final, feature-ready application is deployed to the production environment. Like we discussed about having multiple deployments in a testing environment even for a short span of time, it is advisable to automate such a thing. Scripts to create infrastructure and runtime environments for all environments are useful. It is easier to provision resources in such environments.
For example, to deploy an application in Microsoft Azure, we need the following resources:
The Azure web app configured with specific types of resources
A storage account to store BACPAC files to create the database
Then, we need to follow these steps:
Create a SQL Server instance to host the database.
Import BACPAC files from the storage account to create a new database.
Deploy the web application to Microsoft Azure.
In this scenario, we may consider to use a configuration file for each environment with respect to naming conventions and paths. However, we need similar types of resources in each environment. It is possible that the configuration of resources changes according to the environment, but that can be managed in a configuration file for each environment. Automation scripts can use configuration files based on the environment and create resources and deploy an application into it. Hence, repetitive steps can be easily managed by an automated approach, and this is helpful both in continuous delivery and continuous deployment.
The following are some common practices we should follow to implement continuous delivery:
Plan to automate everything in an application delivery pipeline: Consider a situation where just a single commit only is required to deploy an application in the target environment. It should include compilation, unit test execution, code verification, notification, instance provisioning, setting up runtime environment, and deployment. You must remember to automate:
Develop and test the newly implemented bug fixes in a production-like environment; it is possible now with pay-per-use resources provided by cloud computing.
Deploy frequently in the development and test environments to gain experience and consistency.
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation: http://martinfowler.com/books/continuousDelivery.html. Continuous Delivery vs Continuous Deployment: http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/. Continuous Delivery versus Continuous Deploy: http://devops.com/2015/10/30/continuous-delivery-versus-continuous-deploy/.
Continuous monitoring is a backbone of end-to-end delivery pipeline, and open source monitoring tools are like toppings on an ice cream scoop. It is desirable to monitor at almost every stage in order to have transparency about all the processes, as shown in the following diagram. It also helps us troubleshoot quickly.
Monitoring should be a well thought-out implementation of a plan and it should a part of each of the component mentioned in the following diagram. Consider monitoring practices for continuous integration to continuous delivery/deployment:
There is a likely scenario where end-to-end deployment is implemented in an automated fashion but issues arise due to coding problems, query-related problems, infrastructure related issues, and so on. We can consider different types of monitoring, as shown in the following diagram:
However, there is normally a tendency to monitor only infrastructure resources. The question one must ask is whether it is enough or whether we must focus on other types of monitoring as well. To answer this question, we must have a monitoring strategy in place in the planning stage itself. It is always better to identify stakeholders, monitoring aspects, and so on based on the culture and experience of an organization.
Continuous monitoring demystified: http://searchsecurity.techtarget.com/feature/Continuous-monitoring-demystified
Continuous feedback is the last important component in the DevOps culture and provides a means of improvement and innovation. Feedback always provides improvement if it comes from stakeholders who know what they need and how the outcome should be. Feedback from the customer after deployment activities can serve as inputs to developers for improvement, as shown in the following diagram, and its correct integration will make the customer happy:
Here, we are considering a situation where a feature implementation is provided to the stakeholders and they provide their feedback. In the waterfall model, the feedback cycle is very long and hence developers may not be aware about whether the end product is what the customer asked for or whether the interpretation of what needs to be delivered was changed somewhere. In agile or DevOps culture, a shorter feedback cycle makes a major difference as stakeholders can actually see the result of a small implementation phase, and hence, the outcome is verified multiple times. If customers are not satisfied, then feedback is available at a stage where it is not very tedious to change things. In the waterfall model, this would've been a disaster as feedback used to be received very late. With time and dependencies, the complexity increases and changes in such situations takes a long time. In addition to this, no one remembers what they wrote 2 months back. Hence, a faster feedback cycle improves the overall process and connects endpoints as well as finding patterns in mistakes, learning lessons, and using improved patterns. However, continuous feedback not only improves the technical aspects of implementation but also provides a way to assess current features and whether they fit into the overall scenario or there is still room for improvement. It is important to realize that continuous feedback plays a significant role in making customers happy by providing an improved experience.