The DevOps culture and the practices of CI/CD require modern tools for building, testing, packaging, deploying, and monitoring. There are many open source, licensed, and vendor-specific tools on the market with different prominent features. In this section, we will first categorize CI/CD tools and then present a guideline for choosing the appropriate ones.
DevOps tools can be categorized as follows, starting from source code to the application running in a production environment:
Version Control Systems: GitHub, GitLab, and Bitbucket
Continuous Integration: Jenkins, Travis CI, and Concourse
Test Frameworks: Selenium, JUnit, and pytest
Artifact Management: Maven, Docker, and npm
Continuous Delivery/Deployment: AWS Code pipeline, Codefresh, and Wercker
Infrastructure Provisioning: AWS, GCP, and Microsoft Azure
Release Management: Octopus Delivery, Spinnaker, and Helm
Log Aggregation: Splunk, ELK stack, and Loggly
Metric Collection: Heapster, Prometheus, and InfluxData
Team Communication: Slack, Stride, and Microsoft Teams
On the market, there are plenty of tools with robust features that will make the tool qualified for more than one of the preceding categories. To select an appropriate tool, considering the pros and cons of each is difficult, owing to the uniqueness of organizations and software requirements. Therefore, the following guidelines could help to evaluate the core features of the tools within a continuously evolving industry:
Note
No Silver Bullet—Essence and Accident in Software Engineering was written by Turing Award winner Fred Brooks in 1986. In the paper, it is argued that "There is no single development, in either technology or management technique, which by itself promises even one order of magnitude (tenfold) improvement within a decade in productivity, in reliability, in simplicity." This idea is still valid for most software development fields due to complexity.
Enhanced collaboration: To have a successful DevOps culture in place, all of the tools in the DevOps chain should focus on increasing collaboration. Although there are specific tools, such as Slack, that have cooperation as the main focus, it is crucial to select tools that improve collaboration for every step in software development and delivery. For instance, if you need a source code versioning system, the most basic approach is to set up a bare git server with a single line of code: sudo apt-get install git-core.
With that set up, all of the components will be required to use git command-line tools to interact with the git server. Also, the team will carry the discussions and code reviews to other platforms, such as emails. There are tools such as GitHub, GitLab, or Bitbucket that integrate code reviews, pull requests, and discussion capabilities. Everyone on the team can quickly check the latest pull requests, code reviews, and issues. This eventually increases collaboration. Usage experience differences can be checked out in the following screenshots, between the bare git server with a command-line interface and GitLab with the merge request screen. The crucial point is that both git server setup and GitLab solve the source code versioning problem, but they differ on the level of collaboration increase.
One of the key points while evaluating tools is taking the collaboration capabilities of the tools into consideration even if all the tools provide the same main functionality. The following screen shot shows how GitLab provides a better collaborative experience in comparison to a bare git server:
Figure 1.4: CLI for git server versus GitLab merge request
API integration: The DevOps toolchain and its operations need a high level of automation and configuration. It is unlikely that you should need to hire people to configure infrastructure for every run of the integration test. Instead, all stages of DevOps, from source code to production, are required to expose their APIs. It enables applications to communicate with each other, sending build results, configuration, and artifacts. Rich API functionality enables new tools to be plugged in to the toolchain and work without high customization. Therefore, exposing the APIs is not only crucial for the first setup of the DevOps toolchain, but is also crucial when new tools replace old ones. Accordingly, the second of the key points while evaluating tools is API exposure and the composition of APIs to create a value chain.
Learning curve: Cloud-native DevOps tools try to follow up the latest cloud industry standards and vendor-specific requirements. Besides, most tools focus on user experience and are open to extensions with custom plugins at the same time. These different features could make DevOps tools complicated for beginners. Although there might be experienced users in the team, it is vital to select tools that entail exponential learning curves, starting from zero experience. In other words, tools should allow users to gain competency in a short amount of time, as illustrated in the following diagram. There are three key points to check for a tool that is suitable for everyone in the team:
Documentation: Official documentation, example applications, and references are essential to learn and gain competency.
Community and Support: Online and offline communities and support are critical for solving problems, considering the broad scope of DevOps tools and cloud integrations.
Multiple Access: Selection of the appropriate tools and, having different methods of access, such as API, web interface, and CLI, is essential. This enables beginners to discover tools using the web interface, while experienced users can automate extensively and configure using the API and command line:
Figure 1.5: Exponential learning curve to a competence limit
DevOps practices for the cloud-native world can be viewed as a voyage in which both naive and experienced sailors are in the same boat. All parties need to get some competency, and selected tools should allow for this with their learning curves. Thus, this is the third crucial point while choosing a tool for an organization.
Three main points of enhanced collaboration, API exposure, and learning curves are the must-have features of DevOps tools and also important measures for comparing tools over each other. In the next section, the most popular cloud-native DevOps tools are evaluated using the guideline points mentioned.
Blog websites are popular in areas such as company websites, technology news, or sharing personal journeys. In this exercise, we aim to create a blog where the contents are written and kept in the source code repository, and the site is generated and automatically published by CI/CD pipelines in the cloud.
An up and running live blog website with its contents should appear as follows:
Figure 1.6: A sample blog with a single post
When a new blog post is added to the source code repository, the site should automatically be updated with the new material:
Figure 1.7: An updated blog with a new post added
Before we begin this exercise, ensure that the following prerequisites are satisfied:
The blog and CI/CD pipelines are running on managed cloud services. In this book, we are using GitLab. Thus, you need to ensure that you have a GitLab account to which the projects provided in the book can be forked. You can sign up for an account at https://gitlab.com/.
Website content, namely the HTML files, are generated by Hugo (https://gohugo.io/), which is a framework for building websites. Styling of the blog is handled with beautifulhugo (https://github.com/halogenica/beautifulhugo), which is a Hugo template repository.
The website's pipeline example repository (https://gitlab.com/TrainingByPackt/blog-pipeline-example) should be forked to your individual accounts.
Note
The code files for this exercise can be found here: https://bit.ly/2PBKisL
To successfully complete this exercise, we need to ensure that the following steps are executed:
Fork the project on GitLab to your namespace by clicking the forking icon, as shown in the following screenshot:
Figure 1.8: Forking the project to create a copy
Review the hierarchy of the files and their usages by running the tree command in the shell:
Figure 1.9: Tree view of the folder
As we can see, the following files appear on the tree: .gitignore is used for ignored files in the repository, while .gitlab-ci.yaml defines CI/CD pipeline on GitLab. The config.toml file defines the configuration for Hugo. The content folder is for the source content of the blog and contains another folder, post, and a _index.md file. The post folder contains another file called 2018-10-01-kubernetes-deployment.md. Details on these are provide as follows. _index.md is a Markdown style for the index page, and 2018-10-01-kubernetes-deployment.md is the only blog post live now.
Open the CI/CD pipeline defined in the .gitlab-ci.yaml file and check the following code:
image: registry.gitlab.com/pages/hugo:latest stages: - validate - pages validate: stage: validate script: - hugo pages: stage: pages script: - mkdir -p themes/beautifulhugo && git clone https://github.com/halogenica/beautifulhugo.git themes/beautifulhugo - hugo --theme beautifulhugo only: - master artifacts: paths: - publicThe image: block defines the Docker container image where the pipeline steps will be running, and the stages block defines the sequential order of the jobs that will run. In this pipeline, first, validate will run and if it is successful, then pages will run.
The validate block defines the required testing before publishing the changes. In this block, there is only one command: hugo. This command verifies whether the contents are correct for creating a website.
The pages block is for generating the website with its template and finally publishing it. In the script section, first, the template is installed, and then the site is generated with the installed theme. However, pages has an only block with a master. This implies that only for the master branch will the website be updated. In other words, the pipeline could run for other branches for validation, but the actual site will only be deployed from the master branch.
Check the CI/CD pipelines from the left-hand side of the menu bar by clicking Pipelines under the CI/CD tab, as shown in the following screenshot:
Figure 1.10: The CI/CD pipeline view
As expected, the page shows that there are no pipelines as we have not yet created them.
Click Run Pipeline on the top right-hand corner of the interface. It will redirect you to the following page; then, click Create pipeline:
Figure 1.11: Creating a pipeline on GitLab
You will be able to view the running pipeline instance. With a successful run, it is expected that you will see three successful jobs for Validate, Pages, and Deploy, as shown in the following screenshot:
Figure 1.12: Successful Validate, Pages, and Deploy jobs
Click on the pages tab, as displayed in the preceding screenshot. You will obtain the following output log:
Figure 1.13: A log of the jobs run in containers
In the log screen, all of the lines until Cloning repository... show the preparation steps for the build environment. After that, the source code repository is cloned, and the beatifulhugo template is retrieved. This part is essential since it enables us to combine blog and style at the build time. This approach makes it possible to change to another styling with a template in the future easily without source code change. Then, the HTML files are generated in the Building sites part, and finally, artifacts are uploaded to be served by GitLab.
Type the following URL in your browser window: https://<USERNAME>.gitlab.io/blog-pipeline-example/. The published website is shown as follows:
Figure 1.14: Screen shot of the live blog
Create another file with the name 2018-10-02-kubernetes-scale.md under the content/post folder and type in the following code:
- title: Scaling My Kubernetes Deployment date: 2018-10-02 tags: ["kubernetes", "code"] --- //[...] NAME READY STATUS RESTARTS AGE IP NODE kubernetes-bootcamp-5c69669756-9jhz9 1/1 Running 0 3s 172.18.0.7 minikube kubernetes-bootcamp-5c69669756-lrjwz 1/1 Running 0 3s 172.18.0.5 minikube kubernetes-bootcamp-5c69669756-slht6 1/1 Running 0 3s 172.18.0.6 minikube kubernetes-bootcamp-5c69669756-t4pcs 1/1 Running 0 28s 172.18.0.4 minikube '''We are doing this as we expect the blog to be updated with new content thanks to the CI/CD pipeline we activated with the .gitlab-ci.yaml file.
Note
The file and the complete code is available under the new-post branch of this project: https://bit.ly/2rz8O3Y.
Click on Pipelines under the CI/CD tab to check whether there are further instances of the pipeline. The CI/CD pipeline automatically runs and creates a new deployment as soon as a new post is added:
Figure 1.15: The pipeline runs whenever there is a new commit
Type https://<USERNAME>.gitlab.io/blog-pipeline-example/ and check whether the website is updated when the pipeline is finished:
Figure 1.16: The website is updated with the new post automatically
As you can see, compared to the previous output (as listed in step 7), the website has now been updated.
Thus, in this exercise, we created a blog where the contents were written in a repository, and we observed that the site was automatically generated and published by CI/CD pipelines. We will now summarize this chapter.