DevOps Automation Cookbook

DevOps Automation Cookbook

By : Michael Duffy

Buy this Book

DevOps Automation Cookbook

By: Michael Duffy

Buy this Book

Overview of this book

There has been a recent explosion in tools that allow you to redefine the delivery of infrastructure and applications, using a combination of automation and testing to deliver continuous deployment. DevOps has garnered interest from every quarter, and is rapidly being recognized as a radical shift, as large as the Agile movement for the delivery of software. This book takes a collection of some of the coolest software available today and shows you how to use it to create impressive changes to the way you deliver applications and software. It tackles the plethora of tools that are now available to enable organizations to take advantage of the automation, monitoring, and configuration management techniques that define a DevOps-driven infrastructure. Starting off with the fundamental command-line tools that every DevOps enthusiast must know, this book will guide you through the implementation of the Ansible tool to help you facilitate automation and perform diverse tasks. You will explore how to build hosts automatically with the creation of Apt mirrors and interactive pre-seeds, which are of the utmost importance for Ubuntu automation. You will also delve into the concept of virtualization and creating and manipulating guests with ESXi. Following this, you will venture into the application of Docker; learn how to install, run, network, and restore Docker containers; and also learn how to build containers in Jenkins and deploy apps using a combination of Ansible, Docker, and Jenkins. You will also discover how to filter data with Grafana and the usage of InfluxDB along with unconventional log management. Finally, you will get acquainted with cloud infrastructure, employing the Heroku and Amazon AWS platforms. By tackling real-world issues, this book will guide you through a huge variety of tools, giving new users the ability to get up and running and offering advanced users some interesting recipes that may help with existing issues.

DevOps Automation Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Basic Command Line Tools

Introduction

Controlling network interfaces

Monitoring network details with the IP command

Monitoring connections using the ss command

Gathering basic OS statistics

Viewing historical resource usage with SAR

Installing and configuring a Git client

Creating an SSH key for Git

Using ssh-copy-id to copy keys

Creating a new Git repository

Cloning an existing Git repository

Checking changes into a Git repository

Pushing changes to a Git remote

Creating a Git branch

Ad Hoc Tasks with Ansible

Introduction

Installing an Ansible control node on Ubuntu

Installing an Ansible control node on CentOS

Creating an Ansible inventory

Using the raw module to install python-simplejson

Installing packages with Ansible

Restarting services using Ansible

Executing freeform commands with Ansible

Managing users with Ansible

Managing SSH keys with Ansible

Automatic Host builds

Introduction

Creating an Apt mirror using aptly

Automated installation using PXE boot and a Preseed file

Automating post-installation tasks

Virtualization with VMware ESXi

Introduction

Installing ESXi

Installing and using the vSphere Client

Allowing SSH access to ESXi

Creating a new guest

Allocating resources to a guest

Using the ESXi command line to start, stop, and destroy guests

Managing command-line snapshots

Tuning the host for guest performance

Automation with Ansible

Introduction

Installing Ansible

Creating a scaffold Playbook

Creating a common role

Creating a webserver using Ansible and Nginx

Creating an application server role using Tomcat and Ansible

Installing MySQL using Ansible

Installing and managing HAProxy with Ansible

Using ServerSpec to test your Playbook

Containerization with Docker

Introduction

Installing Docker

Pulling an image from the public Docker registry

Performing basic Docker operations

Running a container interactively

Creating a Dockerfile

Running a container in detached mode

Saving and restoring a container

Using the host only network

Running a private Docker registry

Managing images with a private registry

Using Jenkins for Continuous Deployment

Introduction

Installing Jenkins

Installing the Git plugin

Installing a Jenkins slave

Creating your first Jenkins job

Building Docker containers using Jenkins

Deploying a Java application to Tomcat with zero downtime using Ansible

Metric Collection with InfluxDB

Introduction

Installing InfluxDB

Creating a new InfluxDB database

Logging events with the InfluxDB REST API

Gathering host statistics with Telegraf

Exploring data with the InfluxDB data explorer

Installing Grafana

Creating dashboards with Grafana

Log Management

Introduction

Centralizing logs with Syslog

Using syslog templates

Managing log rotation with the Logrotate utility

Installing ElasticSearch, Logstash, and Kibana

Importing logs into Elasticsearch with Logstash

Using Kibana queries to explore data

Using Kibana queries to examine data

Monitoring with Sensu

Introduction

Installing a Sensu server

Installing a Sensu client

Installing check prerequisites

Finding community checks

Adding a DNS check

Adding a disk check

Adding a RAM check

Adding a process check

Adding a CPU check

Creating e-mail alerts

Creating SMS alerts

Using Ansible to install Sensu

IAAS with Amazon AWS

Introduction

Signing up for AWS

Setting up IAM

Creating your first security group

Creating your first EC2 host

Using Elastic Load Balancers

Managing DNS with route53

Using Ansible to create EC2 hosts

Application Performance Monitoring with New Relic

Introduction

Signing up for a New Relic account

Installing the New Relic Java agent

Using the performance overview

Locating performance bottlenecks with Transaction Traces

Observing database performance with New Relic

Release performance monitoring with New Relic

Server Monitoring with New Relic

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Gathering basic OS statistics

One of the most basic responsibilities of a DevOps engineer is to know how the various server instances under their control are performing. It forms a key part of DevOp techniques, driving infrastructure transparency and measuring the impact of changes, both prior to the change and after it has taken place.

There are many tools that are available for performance monitoring, from comprehensive Application Performance Monitoring (APM) tools through to focused monitoring for particular applications. We'll be covering these throughout the book; however, some of the best tools for basic server monitoring are already available with the operating system. Like most command-line tools, the performance monitoring tools that are shipped with the OS can be used standalone or can be chained with other commands to create complex tools.

Getting ready

The following tools are a part of the standard install of most Linux distributions and should be available with it, the exception being for sysstat tools, which generally need to be installed. To install systat tools on Ubuntu, issue the following command:

sudo apt-get install sysstat

This will make several performance-monitoring tools available, in particular sar and mpstat.

How to do it…

Let's gather basic OS statistics:

To gather basic information on your server, run the following command:
```
vmstat 1 99999
```
This should produce output similar to the following screenshot:
The command shows you the system statistics every second, for the number of times specified (99999 in this instance).
Note
By default, the columns in vmstat stand for the following:
Procs – r: Total number of processes waiting to run
Procs – b: Total number of busy processes
Memory – swpd: Used virtual memory
Memory – free: Free virtual memory
Memory – buff: Memory used as buffers
Memory – cache: Memory used as cache.
Swap – si: Memory swapped from disk (for every second)
Swap – so: Memory swapped to disk (for every second)
IO – bi: Blocks in (in other words) the blocks received from device (for every second)
IO – bo: Blocks out (in other words) the blocks sent to the device (for every second)
System – in: Interrupts per second
System – cs: Context switches
CPU – us, sy, id, wa, st: CPU user time, system time, idle time, and wait time
The vmstat command can also be useful to show memory usage information, particularly active and inactive memory. To show vmstat information for memory usage, you can issue the following command:
```
vmstat –a 1 999
```
This will give you an output similar to the following screenshot:
You can also reformat the output to be displayed in Mega Bytes using the following command:
```
vmstat –a –S M 1 999
```
The vmstat is not limited to gathering only CPU and RAM information; it can also be used to gather information for disks and other block devices. You can use the following command to gather basic disk statistics:
```
vmstat -d 1 99999
```
This should produce an output that looks something like the following screenshot:
Sometimes, the output of vmstat can be slightly cluttered. You can widen the output using the -w options. This can be used on any vmstat command, such as the following:
```
vmstat -d -w
```
Although vmstat is capable of displaying disk statistics, there is a tool that is better suited to this task in the shape of iostat. The iostat is able to display relatively detailed statistics of IO on a server in real time and can be used to reveal performance bottlenecks caused by disk devices.
The following command will display basic statistics and just like vmstat, it will repeat the information every n seconds for n times, where n is a user-specified input:
```
iostat 1 99999
```
This will give you an output similar to the following screenshot:
By default, iostat will show you the CPU information and disk information for all devices. You can drill into the information that iostat produces by using some simple options. For instance, you can show only information for device sda, and only disk statistics by using the following options:
```
iostat -d -p sda 1 9999
```

DevOps Automation Cookbook

By : Michael Duffy

DevOps Automation Cookbook

By: Michael Duffy

Overview of this book

Related Content you might be interested in

Current Title:

DevOps Automation Cookbook

Gathering basic OS statistics

Getting ready

How to do it…

Note