In this chapter, we're going to extend what we saw in Chapter 6, Deploy Real Applications on Swarm. Our goal is to deploy a realistic production-grade Spark cluster on top of Swarm, add storage capacity, launch some Spark jobs and setup monitoring for the underlying infrastructure.
In order to do that, this chapter is mostly infrastructure-oriented. In fact, we'll see how to coalesce Libnetwork, Flocker, and Prometheus with Swarm.
For network, we'll use the basic Docker Network overlay system, based on Libnetwork. There are a few great networking plugins out there, such as Weave and others, but either they are not compatible with the new Docker Swarm Mode yet, or they are made obsolete by Swarm-integrated routing mesh mechanisms.
For storage, the situation is more prosperous, because there is much more choice (refer to https://docs.docker.com/engine/extend/plugins/). We'll go with Flocker. Flocker is the grandfather of Docker storage, and can be configured...