Welcome to Docker on Amazon Web Services! I'm very excited to have written this book and to share how to leverage the wonderful technologies that the Docker and Amazon Web Services (AWS) ecosystems provide to build truly world-class solutions for deploying and operating your applications in production.
Docker has become the modern standard for building, packaging, publishing, and operating applications, leveraging the power of containers to increase the speed of application delivery, increase security, and reduce costs. This book will show you how to supercharge your process of building Docker applications, using the best practices of continuous delivery to provide a fully automated, consistent, reliable, and portable workflow for testing, building, and publishing your Docker applications. In my view, this is a fundamental prerequisite before you even consider deploying your application to the cloud, and the first few chapters will focus on establishing a local Docker environment and creating a local continuous delivery workflow for a sample application that we will be using throughout the book.
AWS is the world's leading public cloud provider, and provides a rich set of solutions for managing and operating your Docker applications. This book will cover all of the major services that AWS provides to support Docker and containers, including the Elastic Container Service (ECS), Fargate, Elastic Beanstalk, and Elastic Kubernetes Service (EKS), and also will discuss how you can leverage the Docker for AWS solution provided by Docker Inc to deploy Docker Swarm clusters.
Running a complete application environment in AWS comprises much more than your container platform, and this book will also describe best practices for managing access to your AWS account and leveraging other AWS services to support the requirements of your applications. For example, you will learn how to set up AWS application load balancers to publish highly available, load-balanced endpoints for your application, create AWS Relational Database Service (RDS) instances to provide a managed application database, integrate your applications with the AWS Secrets Manager to provide a secure secrets management solution, and create a complete continuous delivery pipeline using the AWS CodePipeline, CodeBuild, and CloudFormation services that will automatically test, build, and publish Docker images for any new changes to your application, and then automatically deploy it into development and production environments.
You will build all of this supporting infrastructure using the AWS CloudFormation service, which provides powerful infrastructure-as-code templates that allow you define all of the AWS services and resources I have mentioned in a single manifest that you can deploy to AWS with a single click of a button.
I'm sure by now you are just as excited as I am to learn about all of these wonderful technologies, and I'm sure by the end of this book, you will have developed the expert knowledge and skills required to be able to deploy and manage your Docker applications, using the latest cutting-edge techniques and best practices.
Docker on Amazon Web Services is for anybody who wants to build, deploy, and operate applications using the power of containers, Docker, and AWS.
Readers ideally should have a basic understanding of Docker and containers, and have worked with AWS or another cloud provider, although no previous experience with containers or AWS is required, as this book takes a step-by-step approach and explains key concepts as you progress. An understanding of how to use the Linux command line, Git, and basic Python scripting knowledge will be useful, but is not required.
See the To get the most out of this book section for a complete list of the recommended prerequisite skills.
Chapter 1, Container and Docker Fundamentals, will provide a brief introduction to Docker and containers, and provide an overview of the various services and options available in AWS to run your Docker applications. You will set up your local environment, installing Docker, Docker Compose, and various other tools that are required to complete the examples in each chapter. Finally, you will download the sample application and learn how to test, build, and run the application locally, so that you have a good understanding of how the application works and specific tasks you need to perform to get the application up and running.
Chapter 2, Building Applications Using Docker, will describe how to build a fully automated Docker-based workflow for testing, building, packaging, and publishing your applications as production-ready Docker release images, using Docker, Docker Compose, and other tools. This will establish the foundation of a portable continuous delivery workflow that you can consistently execute across multiple machines without having to install application-specific dependencies in each local environment.
Chapter 3, Getting Started with AWS, will describe how to create a free AWS account and start using a variety of free-tier services that allow you to get familiar with the wide array of AWS services on offer. You will learn how to establish best practice administrative and user access patterns to your account, configuring multi-factor authentication (MFA) for enhanced security and installing the AWS command-line interface, which can be used for a wide variety of operational and automation use cases. You will also be introduced to CloudFormation, which is a management tool and service provided free by AWS that you will use throughout this book that allows you to deploy complex environments with a single click of a button, using a powerful and expressive infrastructure as code template format.
Chapter 4, Introduction to ECS, will get you up and running with the Elastic Container Service (ECS), which is the flagship service for running your Docker applications in AWS. You will learn about the architecture of ECS, create your first ECS cluster, define your container configurations using ECS task definitions, and then deploy a Docker application as an ECS service. Finally, you will be briefly introduced to the ECS command-line interface (CLI), which allows you to interact with local Docker Compose files and automatically deploy Docker Compose resources to AWS using ECS.
Chapter 5, Publishing Docker Images Using ECR, will teach you how to establish a private Docker registry using the Elastic Container Registry (ECR), authenticate to your registry using IAM credentials, and then publish Docker images to private repositories within your registry. You will also learn how to share your Docker images with other accounts and AWS services, and how to configure life cycle policies to automatically clean up orphaned images, ensuring you only pay for active and current images.
Chapter 6, Building Custom ECS Container Instances, will show you how to use a popular open source tool called Packer to build and publish custom Amazon Machine Images (AMIs) for the EC2 instances (ECS container instances) that run your container workloads in ECS clusters. You will install a set of helper scripts that enable your instances to integrate with CloudFormation and download custom provisioning actions at instance creation time, allowing you to dynamically configure the ECS cluster your instances will join, configure the CloudWatch logs groups your instances should publish logging information to, and finally, signal back to CloudFormation that provisioning has succeeded or failed.
Chapter 7, Creating ECS Clusters, will teach you how to build ECS clusters based upon EC2 auto-scaling groups that leverage the features of the custom AMI you created in the previous chapter. You will define your EC2 auto-scaling group, ECS cluster, and other supporting resources using CloudFormation, and configure CloudFormation Init metadata to perform custom runtime configuration and provisioning of the ECS container instances that make up your ECS cluster.
Chapter 8, Deploying Applications Using ECS, will expand the environment created in the previous chapter, adding supporting resources such as Relational Database Service (RDS) instances and AWS Application Load Balancers (ALBs) to your CloudFormation template. You will then define an ECS task definition and ECS service for the sample application, and learn how ECS can perform automated rolling deployments and updates for your applications. To orchestrate required deployment tasks such as running database migrations, you will extend CloudFormation and write your own Lambda function to create an ECS task runner custom resource, providing the powerful capability to run any provisioning action that can be executed as an ECS task.
Chapter 9, Managing Secrets, will introduce the AWS Secrets Manager, which is a fully managed service that stores secret data in an encrypted format that can be easily and securely accessed by authorized parties such as your users, AWS resources, and applications. You will interact with Secrets Manager using the AWS CLI, creating secrets for sensitive credentials such as database passwords, and then learn how to use an entrypoint script for your containers that injects secret values as internal environment variables at container startup before handing off to the main application. Finally, you will create a CloudFormation custom resource that exposes secrets to other AWS services that do not support AWS Secrets Manager, such as providing an administrative password for Relational Database Service (RDS) instances.
Chapter 10, Isolating Network Access, describes how to use the awsvpc networking mode in your ECS task definitions to isolate network access and separate ECS control plane communications from your container and application communications. This will allow you to adopt best practice security patterns such as deploying your containers on private networks, and implement solutions for providing internet access, including the AWS VPC NAT Gateway service.
Chapter 11, Managing the ECS Infrastructure Life Cycle, will provide you with an understanding of operational challenges when running ECS clusters, which includes taking your ECS container instances out of service, whether it be to scale in your auto-scaling groups or to replace your ECS container instances with a new Amazon machine image. You will learn how to leverage EC2 auto-scaling life cycle hooks to invoke an AWS Lambda function whenever an ECS container instance is about to be terminated, which allows you to perform graceful shutdown operations such as draining active containers to other instances in the cluster, before signaling EC2 auto-scaling to proceed with instance termination.
Chapter 12, ECS Auto Scaling, will describe how ECS clusters manage resources such as CPU, memory, and network ports, and how this affects the capacity of your clusters. If you want to be able to dynamically auto-scale your clusters, you need to dynamically monitor ECS cluster capacity, and scale out or scale in the cluster at capacity thresholds that ensure you will meet the service level expectations of your organization or use case. You will be implement a solution that calculates ECS cluster capacity whenever an ECS container instance state change event is generated via the AWS CloudWatch Events service, publishes capacity metrics to CloudWatch, and dynamically scales your cluster up or down using CloudWatch alarms. With a dynamic cluster capacity solution in place, you will then be able to configure the AWS application auto-scaling service to dynamically adjust the number of instances of service based upon appropriate metrics, such as CPU utilization or active connections, without needing to concern yourself with the effect on underlying cluster capacity.
Chapter 13, Continuously Delivering ECS Applications, will establish a continuous delivery pipeline using the AWS CodePipeline service that integrates with GitHub to detect changes to your application source code and infrastructure deployment scripts, use the AWS CodeBuild service to run unit tests, build application artifacts and publish a Docker image using the sample application Docker workflow, and continuously deploy your application changes to AWS using the CloudFormation templates you have used so far in this book.
This will automatically deploy into an AWS development environment that you test, and then create a change set and manual approval action for deployment into production, providing you with a rapid and repeatable path to production for all of your applications new features and bug fixes.
Chapter 14, Fargate and ECS Service Discovery, will introduce AWS Fargate, which provides a solution that fully manages both the ECS service control plane and ECS clusters that you traditionally have to manage using the regular ECS service. You will deploy the AWS X-Ray daemon as an ECS service using Fargate, and configure ECS service discovery to dynamically publish your service endpoints using DNS and Route 53. This will allow you to add support for X-Ray tracing to your sample application, which can be used to trace incoming HTTP requests to your application and monitor AWS service calls, database calls, and other types of calls that are made to service each incoming request.
Chapter 15, Elastic Beanstalk, will provide an overview of the popular Platform-as-a-Service (PaaS) offering, which includes support for Docker applications. You will learn how to create an Elastic Beanstalk multi-container Docker application, establish an environment that consists of a managed EC2 instance, an RDS database instance, and an Application Load Balancer (ALB), and then extend the environment using various techniques to support the requirements of your Docker applications, such as volume mounts and running single-shot tasks per application deployment.
Chapter 16, Docker Swarm in AWS, will focus on how to run Docker Swarm clusters in AWS, using the Docker for AWS blueprint provided for Docker Swarm community edition. This blueprint provides you with a CloudFormation template that establishes a pre-configured Docker Swarm cluster in AWS within minutes, and features integrations with key AWS services such as the Elastic Load Balancing (ELB), Elastic File System (EFS) and Elastic Block Store (EBS) services. You will define a stack using Docker Compose, which configures a multi-service environment expressed in the familiar Docker Compose specification format, and learn how to configure key Docker Swarm resources such as services, volumes, and Docker secrets. You will learn how to create shared Docker volumes that are backed by EFS, relocatable Docker volumes backed by EBS that Docker Swarm will automatically reattach to new containers redeployed after a node failure, and publish an external service endpoint for your application using an ELB that is automatically created and managed for you by Docker Swarm.
Chapter 17, Elastic Kubernetes Service, introduces the newest container management platform offering from AWS, which is based on the popular open source Kubernetes platform. You will first set up Kubernetes in your local Docker Desktop environment, which includes native support for Kubernetes with the Docker 18.06 CE release, and learn how to create a complete environment for your Docker applications using a number of Kubernetes resources, including pods, deployments, services, secrets, persistent volumes, and jobs. You will then establish an EKS cluster in AWS, create an EC2 auto-scaling group that connects worker nodes to your cluster, and ensure your local Kubernetes client can authenticate and connect to the EKS control plane, after which you will deploy the Kubernetes dashboard to provide a comprehensive management interface for your cluster. Finally, you will define a default storage class that uses the Elastic Block Store (EBS) service for persistent volumes and then deploy your Docker applications to AWS, leveraging the same Kubernetes definitions you created earlier for your local environment, providing you with a powerful solution to quickly deploy Docker applications locally for development purposes, and then deploy straight to production using EKS.
- A basic, working knowledge of Docker - if you haven't used Docker before, you should learn about the basic concepts of Docker at https://docs.docker.com/engine/docker-overview/ and then step through Parts 1 (https://docs.docker.com/get-started/) and 2 (https://docs.docker.com/get-started/part2) of the Docker Get Started tutorial. For a more comprehensive understanding of Docker, check out the Learn Docker - Fundamentals of Docker 18.x book from Packt Publishing.
- A basic, working knowledge of Git - if you haven't used Git before, you should run through the Beginner and Getting Started tutorials at https://www.atlassian.com/git/tutorials. For a more comprehensive understanding of Git, check out the Git Essentials book from Packt Publishing.
- Familiarity with AWS - if you haven't used AWS before, running through the Launch a Linux Virtual Machine tutorial at https://aws.amazon.com/getting-started/tutorials/launch-a-virtual-machine/ will provide a useful introduction.
- Familiarity with the Linux/Unix command line - if you haven't used the Linux/Unix command line before, I recommend running through a basic tutorial such as https://maker.pro/linux/tutorial/basic-linux-commands-for-beginners, using the Linux Virtual Machine you created when you went through the Launch a Linux Virtual Machine tutorial.
- Basic understanding of Python - the sample application for this book is written in Python, and some of the examples in later chapters include basic Python scripts. If you have not worked with Python before, you may want to read through the first few lessons at https://docs.python.org/3/tutorial/.
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
- Log in or register at www.packtpub.com.
- Select the
SUPPORTtab. - Click on
Code Downloads & Errata. - Enter the name of the book in the
Searchbox and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
- WinRAR/7-Zip for Windows
- Zipeg/iZip/UnRarX for Mac
- 7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Docker-on-Amazon-Web-Services. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/DockeronAmazonWebServices_ColorImages.pdf
Visit the following link to check out videos of the code being run:http://bit.ly/2Noqdpn
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Note that the gist includes a placeholder called PASTE_ACCOUNT_NUMBER within the policy document, so you will need to replace this with your actual AWS account ID."
A block of code is set as follows:
AWSTemplateFormatVersion: "2010-09-09"
Description: Cloud9 Management Station
Parameters:
EC2InstanceType:
Type: String
Description: EC2 instance type
Default: t2.micro
SubnetId:
Type: AWS::EC2::Subnet::Id
Description: Target subnet for instanceAny command-line input or output is written as follows:
> aws configure AWS Access Key ID [None]:
Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "To create the admin role, select Services | IAM from the AWS console, select Roles from the left-hand menu, and click on the Create role button."
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.