What this book covers
Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy, introduces Jenkins and discusses its strengths, along with a little bit of history and important keywords. The chapter describes the architecture of the Jenkins infrastructure that we will be building in the coming chapters, one for Jenkins on AWS and another for Jenkins inside a corporate firewall. It discusses the architecture of the controllers, reverse proxy, agents, and the Docker cloud by listing the required virtual machines, operating system, and software packages we’ll use, the ports that need to be opened, and other required components. The chapter continues to discuss frequently asked questions for the AWS infrastructure, such as EC2 instance types and sizes, Regions and Availability Zones, routing rules, and Elastic IPs. Then, the chapter discusses the TLS/SSL certificate choices and goes through the steps for using Let’s Encrypt in detail to create a free certificate. Finally, the chapter discusses the importance of storage backend choices. It discusses the different options by benchmarking performance and going through the pros and cons of the popular storage backend solutions.
Chapter 2, Jenkins with Docker on HTTPS on AWS and inside a Corporate Firewall, goes through the entire journey of setting up the Jenkins controller, reverse proxy for HTTPS connections, agents, and the Docker cloud. It shows a way to create a directory on the host machine and mount it in a Docker container running Jenkins, so that the state is preserved across container restarts. It also shows three different ways of terminating the TLS to provide HTTPS connections. Once Jenkins is running on HTTPS, the chapter goes through the basic configuration options for login methods, pipeline default speed, user permissions, and other useful default settings. It continues to show the steps for attaching agents and creating and attaching a Docker cloud, so that we end up with production-grade Jenkins.
Chapter 3, GitOps-Driven CI Pipeline with GitHub, shows the steps for creating premerge CI pipelines that are triggered from a GitHub pull request activity. It first creates four example users, then assigns them various permissions for the two example projects, adder and subtractor, to demonstrate the Jenkins permission model. It then goes through the steps for creating the CI pipelines in detail, demonstrating and discussing each step as we progress. It shows the two different ways to configure a CI pipeline, one for AWS Jenkins using push hooks and another for firewalled Jenkins using the GitHub Pull Request Builder plugin. It finishes by showing the optional steps to allow the CI pipeline to build an arbitrary branch, along with the steps to require a successful build for merging a pull request.
Chapter 4, GitOps-Driven CD Pipeline with Docker Hub and More Jenkinsfile Features, shows the steps for creating postmerge CD pipelines that are triggered from a GitHub pull request merge activity. Along the way, it discusses various Jenkinsfile techniques such as running external scripts, passing variables across steps, several ways of using Docker-outside-of-Docker (DooD), using bare-metal agents, using credentials, and interacting with GitHub and Docker Hub. Similar to Chapter 3, GitOps-Driven CI Pipeline with GitHub, it goes through the detailed steps for creating CD pipelines for both AWS Jenkins using push hooks and firewalled Jenkins using polling.
Chapter 5, Headfirst AWS for Jenkins, shows detailed instructions on using AWS. In the earlier chapters, we have skipped the details on most of the AWS operations in order to keep the focus on Jenkins, and in this chapter, we discuss them in full detail so that new users can follow the steps click by click while referring to the numerous screenshots. It starts by discussing the basics of logging into AWS, then continues to the steps for creating an SSH key pair, managing security groups, creating EC2 instances with Elastic IPs, using Let’s Encrypt to generate TLS/SSL certificates, creating and configuring Elastic Load Balancers (ELBs), using AWS Certificate Manager to generate TLS/SSL certificates, setting up routing rules, and finally configuring Route 53 to point the Jenkins URL to the controller.
Chapter 6, Jenkins Configuration as Code (JCasC), discusses JCasC in detail by creating a whole new Jenkins instance using a configuration file we generate throughout the chapter. It starts by installing the JCasC plugin and discussing the limitations and the boundaries of what JCasC can manage. It then continues to read the configuration details of the Jenkins we set up in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy, through Chapter 4, GitOps-Driven CD Pipeline with Docker Hub and More Jenkinsfile Features. It discusses each section of the configuration, and builds a new JCasC configuration file based on the entries from the existing Jenkins. Once the configuration file is built for the controller, agent, and Docker cloud, it creates a new Jenkins instance using the configuration file. It revisits each configuration item and discusses how well (or not) it was restored. Finally, it shows an optional way to use Groovy scripting to work around some of the issues found during the restoration.
Chapter 7, Backup and Restore and Disaster Recovery, discusses the backup strategies for different scenarios, and goes over the exact steps to set up an automated backup system. It first discusses the pros and cons of a disk snapshot backup and a file-level backup. Then it looks at the content of $JENKINS_HOME and identifies the files and folders that need to be backed up at a high frequency, as opposed to the ones that need to be backed up only once a day. Once we have determined which files to back up, the chapter goes through ThinBackup plugin configurations. It first provides an off-site backup solution using NFS and Docker volume mount, then goes into the specifics of configuring the ThinBackup plugin to generate the backup files effectively. Once backup files are generated, the chapter goes through a disaster scenario where we restore a pipeline that a user mistakenly deleted. It shows various ways to identify the correct backup snapshot to restore, then goes into deep discussions on how to restore a backup effectively. In addition to restoring the mistakenly deleted pipeline, it teaches the fundamental mechanism for backup and restore by demonstrating a way to restore a pipeline that didn’t exist. Finally, it goes through an infrastructure failure disaster scenario and provides a recovery playbook that you can follow.
Chapter 8, Upgrading the Jenkins Controller, Agents, and Plugins, discusses the upgrade strategies for both small and large Jenkins instances. It first discusses the pitfalls of upgrading plugins, and provides various ways of upgrading the plugins and controller effectively. In addition to the upgrade process, the chapter goes through an SRE runbook for an upgrade scenario where you are taught when and how to communicate with users about the upgrade. The runbook covers not only the success path but also the failure scenario and discusses the restore and rollback strategies.
Chapter 9, Reducing Bottlenecks, teaches you various ways to optimize your Jenkins, such as picking the right EC2 instance size, reducing the Jenkins memory footprint, not using periodic triggers in favor of webhook triggers, tracking the AWS costs, optimizing GitHub Pull Request Builder options, and removing the weather icon from the home page. It continues on to discuss the various Groovy scripts that terminate long-running pipelines, release stale locks, and clean up logs. It also discusses the best practices for writing Jenkins pipeline code, such as reducing the use of the echo step and using NonCPS code for faster execution. It then talks about reducing the agent startup time by baking the plugin archives into the EC2 AMI, and then finally discusses the ways to manage various logs effectively.
Chapter 10, Shared Libraries, starts by discussing the directory structure and the content of a shared library. Afterward, it creates an example shared library that uses many common features, then explains the differences between providing the shared library as a global shared library versus a folder-level shared library. Once the shared library is available to be used, the chapter teaches you several different methods of loading it and discusses the use case for each method. Afterward, the chapter goes through a hands-on example of creating shared library functions that use the Slack messenger app to provide standardized messaging wrappers. Finally, the chapter dives deeper into a more advanced use case of creating custom domain-specific languages (DSLs) using shared libraries.
Chapter 11, Script Security, starts with an explanation of the role of an administrator versus a non-administrator in Jenkins. It continues by explaining the concept of the Groovy sandbox, and discusses running outside and inside the sandbox. It teaches the dangers of running pipelines outside of the sandbox and provides a use case of using a global shared library to wrap dangerous method calls. It then continues to explain the Jenkins permission model by discussing running inside the sandbox and teaches you how to use method signature approvals effectively. The chapter takes a deep dive into explaining the approve assuming permissions check button, and explains the SYSTEM user and the dangers of the default Jenkins permission model. Finally, it discusses an alternate Jenkins design that doesn’t rely on the Script Security plugin’s protection.