Welcome to Native Docker clustering with Swarm! This is a book about containers and distributed systems. We’ll show how to use the native Docker tools to model micro services, spawn tasks, scale up the size of your application, and push your containers to the Docker clustering limit! In a word, we’ll discuss Docker orchestration.
With the recent rise of Swarm Mode and the enablement of Swarm inside the Docker Engine itself, it turned out that the best way to orchestrate Docker is… Docker!
Good, but what do we mean by “orchestrate Docker”? What is orchestration? Better, what is an orchestra?
An orchestra is an ensemble of musicians led by a conductor, who dictates tempo, rhythm and shapes the sound. Strings, woodwinds, percussions, keyboards and other instruments follow the conductor’s direction to perform an astonishing symphony, for example Beethoven’s The Ninth.
Similarly, in a containers orchestration system the musicians are tasks, and the conductor is a leader service (Swarm primitives). Tasks don’t play a symphony, or at least not only: More abstractly they execute some computational job, for example they run a webserver. The conductor, Swarm, is responsible for their provisioning, their availability, their linkage, their scaling. This (and more) is what we mean by “Docker orchestration”.
This book shows how to provision such Docker “orchestras”, how to guarantee the availability of the service, how to connect tasks and how to scale the platform, to play the symphony of your application.
Chapter 1, Welcome to Docker Swarm, introduces Swarm, and explains why you need a clustering solution for your containers. It illustrates the Swarm features, giving a high-level description of its architecture. We define some use cases and describe how Swarm is different from Fleet, Kubernetes and Mesos. The chapter proceeds with the Docker tools installation and finally with two Swarms provisionings: A local Swarm Standalone and a remote Swarm Mode cluster on DigitalOcean.
Chapter 2, Discover the Discovery Services, is a descriptive and mostly abstract chapter. We’ll learn what discovery mechanisms and consensus algorithms are, and why they are crucial for distributed systems. We’ll describe in detail Raft and its implementation Etcd, the consensus mechanism included in Swarm Mode. We will also show the limitations of the discovery mechanism used in Chapter 1, Welcome to Docker Swarm, by extending the local tiny example with Consul, re deploying it.
Chapter 3, Meeting Docker Swarm Mode, is about the new Docker kit that allows to create task clusters of any size. We will introduce Swarmit, the foundation of Docker Swarm Mode, showing how it works in Docker 1.12+, discuss its architecture, its concepts, how it’s different from the “old” Swarm, and how it organizes workloads by abstracting services and tasks.
Chapter 4, Creating a Production-Grade Swarm, shows and discusses the community-driven projects Swarm2k and Swarm3k, our 2,300 and 4,800 nodes Swarm clusters experiments, which ran hundreds of thousands of containers. We demonstrate how such huge clusters were planned, provisioned, and summarize the lessons we learned.
Chapter 5, Administer a Swarm Cluster, is a chapter about infrastructure. We will show how to increase or decrease Swarms sizes, how to promote and demote nodes, and how to update clusters and nodes properties. We’ll introduce Shipyard and Portainer.io as graphical UIs for Swarm.
Chapter 6, Deploy Real Applications on Swarm, is where we will put real applications in motion on Swarm and where we add to the discussion some notes about Compose, Docker Stacks and Docker Application Bundles. We will show the typical deployment workflow, how to filter and schedule containers over the cluster, launch them as services, handle containers as tasks. We’ll start defining a web service with Nginx, then we’ll deploy a mandatory Wordpress with MySQL example. We’ll finally move on with a more realistic app: Apache Spark.
Chapter 7, Scale Up Your Platform, will develop new topics from the previous chapter. Here we’ll introduce Flocker to add storage capacity to Spark on Swarm, and we’ll show how to install and use it automatically at a scale in conjunction with Swarm. We’ll refine our Spark example by running some real big data jobs and setting up a basic monitoring system for this infrastructure.
Chapter 8, Exploring Additional Features to Swarm, discusses some advanced topics important to Swarm, such as Libnetwork and Libkv.
Chapter 9, Securing a Swarm Cluster and Docker Software Supply Chain, will focus on security considerations for Swarm clusters. Among the arguments, certificates, firewalling concepts for platform, and a mention to Notary.
Chapter 10, Swarm and the Cloud, is a chapter illustrating the most popular options for running Swarm on cloud providers. We’ll install Swarm on AWS and Azure before introducing the Docker Datacenter, and we’ll finally move to OpenStack showing how to install and manage Swarms on the top of Magnum, the Container as a Service solution for OpenStack.
Chapter 11, What is Next?, concludes the discussion with an overview on the next Docker orchestration trends, such as software defined infrastructures, Infrakit, unikernels, Containers as a Service. The adventure continues!
We assume the reader to have some experience using Docker from the command line: throughout the book we’ll continuously pull images, run containers, define services, expose ports and create networks.
Also, the ideal reader possesses a basic understanding of networking protocols and is familiar with public and private cloud concepts like virtual machines and tenant networks.
To follow the examples in the text, you will need Docker and its tools. Chapter 1, Welcome to Docker Swarm, covers their installation.
Also, to get the most from the examples, you will need access to a a public (for example AWS, Azure or DigitalOcean) or private (for example OpenStack) cloud to instantiate Virtual Machines.
This book is for Docker users - developers and system administrators - who want to exploit the current Swarm and Swarmkit features for scaling massive applications with containers.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "When executing docker swarm init, just copy-paste the lines printed as output"
A block of code is set as follows:
digitalocean: image: “docker-1.12-rc4” region: nyc3 ssh_key_fingerprint: “your SSH ID” ssh_user: root
Any command-line input or output is written as follows:
# Set $GOPATH here go get https://github.com/chanwit/belt
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "The UI has the expected options, included a list of templates for launching containers, such as MySQL or a Private Registry, but at the moment of writing it doesn’t support Swarm services yet"
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at [email protected] with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
"Illustrations of Swarm Architecture, Minimal Swarm on Production, and Docker Eco System in Chapter 1, Welcome to Docker Swarm © 2015-2016 Docker, Inc. Used by permission."
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.