Creating clusters in the cloud (GCP, AWS, Azure, and Digital Ocean)

Creating clusters locally is fun. It’s also important during development and when trying to troubleshoot problems locally. But, in the end, Kubernetes is designed for cloud-native applications (applications that run in the cloud). Kubernetes doesn’t want to be aware of individual cloud environments because that doesn’t scale. Instead, Kubernetes has the concept of a cloud-provider interface. Every cloud provider can implement this interface and then host Kubernetes.

The cloud-provider interface

The cloud-provider interface is a collection of Go data types and interfaces. It is defined in a file called cloud.go, available at: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go.

Here is the main interface:

type Interface interface {
     Initialize(clientBuilder ControllerClientBuilder, stop <-chan struct{})
    LoadBalancer() (LoadBalancer, bool)
    Instances() (Instances, bool)
    InstancesV2() (InstancesV2, bool)
    Zones() (Zones, bool)
    Clusters() (Clusters, bool)
    Routes() (Routes, bool)
    ProviderName() string
    HasClusterID() bool
}

This is very clear. Kubernetes operates in terms of instances, zones, clusters, and routes, and also requires access to a load balancer and provider name. The main interface is primarily a gateway. Most methods of the Interface interface above return yet other interfaces.

For example, the Clusters() method returns the Cluster interface, which is very simple:

type Clusters interface {
    ListClusters(ctx context.Context) ([]string, error)
    Master(ctx context.Context, clusterName string) (string, error)
}

The ListClusters() method returns cluster names. The Master() method returns the IP address or DNS name of the control plane of the cluster.

The other interfaces are not much more complicated. The entire file is 313 lines long (at the time of writing) including lots of comments. The take-home point is that it is not too complicated to implement a Kubernetes provider if your cloud utilizes those basic concepts.

Creating Kubernetes clusters in the cloud

Before we look at the cloud providers and their support for managed and non-managed Kubernetes, let’s consider how you should create and maintain clusters. If you commit to a single cloud provider, and you are happy with using their tooling, then you are set. All cloud providers let you create and configure Kubernetes clusters using either a Web UI, a CLI, or an API. However, if you prefer a more general approach and want to utilize GitOps to manage your clusters, you should look into Infrastructure as Code solutions such as Terraform and Pulumi.

If you prefer to roll out non-managed Kubernetes clusters in the cloud, then kOps is a strong candidate. See: https://kops.sigs.k8s.io.

Later, in Chapter 17, Running Kubernetes in Production, we will discuss in detail the topic of multi-cluster provisioning and management. There are many technologies, open source projects, and commercial products in this space.

For now, let’s look at the various cloud providers.

GCP

The Google Cloud Platform (GCP) supports Kubernetes out of the box. The so-called Google Kubernetes Engine (GKE) is a container management solution built on Kubernetes. You don’t need to install Kubernetes on GCP, and you can use the Google Cloud API to create Kubernetes clusters and provision them. The fact that Kubernetes is a built-in part of the GCP means it will always be well integrated and well tested, and you don’t have to worry about changes in the underlying platform breaking the cloud-provider interface.

If you prefer to manage Kubernetes yourself, then you can just deploy it directly on GCP instances (or use kOps alpha support for GCP), but I would generally advise against it as GKE does a lot of work for you and it’s integrated deeply with GCP compute, networking, and core services.

All in all, if you plan to base your system on Kubernetes and you don’t have any existing code on other cloud platforms, then GCP is a solid choice. It leads the pack in terms of maturity, polish, and depth of integration to GCP services, and is usually the first to update to newer versions of Kubernetes.

I spent a lot of time with Kubernetes on GKE, managing tens of clusters, upgrading them, and deploying workloads. GKE is production-grade Kubernetes for sure.

GKE Autopilot

GKE also has the Autopilot project, which takes care of managing worker nodes and node pools for you, so you focus on deploying and configuring workloads.

See: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview.

AWS

AWS has its own container management service called ECS, which is not based on Kubernetes. It also has a managed Kubernetes service called EKS. You can run Kubernetes yourself on AWS EC2 instances. Let’s talk about how to roll your own Kubernetes first and then we’ll discuss EKS.

Kubernetes on EC2

AWS was a supported cloud provider from the get-go. There is a lot of documentation on how to set it up. While you could provision some EC2 instances yourself and use kubeadm to create a cluster, I recommend using the kOps (Kubernetes Operations) project mentioned earlier. kOps initially supported only AWS and is generally considered the most battle-tested and feature-rich tool for self-provisioning Kubernetes clusters on AWS (without using EKS).

It supports the following features:

• Automated Kubernetes cluster CRUD for the cloud (AWS)
• HA Kubernetes clusters
• Uses a state-sync model for dry-run and automatic idempotency
• Custom support for kubectl addons
• kOps can generate Terraform configuration
• Based on a simple meta-model defined in a directory tree
• Easy command-line syntax
• Community support

To create a cluster, you need to do some IAM and DNS configuration, set up an S3 bucket to store the cluster configuration, and then run a single command:

kops create cluster \
    --name=${NAME} \
    --cloud=aws \
    --zones=us-west-2a \
    --discovery-store=s3://prefix-example-com-oidc-store/${NAME}/discovery

The complete instructions are here: https://kops.sigs.k8s.io/getting_started/aws/.

At the end of 2017, AWS joined the CNCF and made two big announcements regarding Kubernetes: its own Kubernetes-based container orchestration solution (EKS) and a container-on-demand solution (Fargate).

Amazon EKS

Amazon Elastic Kubernetes Service (EKS) is a fully managed and highly available Kubernetes solution. It has three control plane nodes running in three AZs. EKS also takes care of upgrades and patching. The great thing about EKS is that it runs a stock Kubernetes. This means you can use all the standard plugins and tools developed by the community. It also opens the door to convenient cluster federation with other cloud providers and/or your own on-premise Kubernetes clusters.

EKS provides deep integration with AWS infrastructure like IAM authentication, which is integrated with Kubernetes Role-Based Access Control (RBAC).

You can also use PrivateLink if you want to access your Kubernetes masters directly from your own Amazon VPC. With PrivateLink, your Kubernetes control plane and the Amazon EKS service endpoint appear as an elastic network interface with private IP addresses in your Amazon VPC.

Another important piece of the puzzle is a special CNI plugin that lets your Kubernetes components talk to each other using AWS networking.

EKS keeps getting better and Amazon demonstrated that it is committed to keeping it up to date and improving it. If you are an AWS shop and getting into Kubernetes, I recommend starting with EKS as opposed to building your own cluster.

The eksctl tool is a great CLI for creating and managing EKS clusters and node groups for testing and development. I successfully created, deleted, and added nodes to several Kubernetes clusters on AWS using eksctl. Check out https://eksctl.io/.

Fargate

Fargate lets you run containers directly without worrying about provisioning hardware. It eliminates a huge part of the operational complexity at the cost of losing some control. When using Fargate, you package your application into a container, specify CPU and memory requirements, define networking and IAM policies, and you’re off to the races. Fargate can run on top of ECS and EKS. It is a very interesting member of the serverless camp although it’s not specific to Kubernetes like GKE’s Autopilot.

Azure

Azure used to have its own container management service based on Mesos-based DC/OS or Docker Swarm to manage your containers. But you can also use Kubernetes, of course. You could also provision the cluster yourself (for example, using Azure’s desired state configuration) and then create the Kubernetes cluster using kubeadm. kOps has alpha support for Azure, and the Kubespray project is a good option too.

However, in the second half of 2017 Azure jumped on the Kubernetes bandwagon too and introduced AKS (Azure Kubernetes Service). It is similar to Amazon EKS, although it’s a little further ahead in its implementation.

AKS provides a Web UI, CLI, and REST API to manage your Kubernetes clusters. Once, an AKS cluster is configured, you can use kubectl and any other Kubernetes tooling directly.

Here are some of the benefits of using AKS:

• Automated Kubernetes version upgrades and patching
• Easy cluster scaling
• Self-healing hosted control plane (masters)
• Cost savings – pay only for running agent pool nodes

AKS also offers integration with Azure Container Instances (ACI), which is similar to AWS Fargate and GKE AutoPilot. This means that not only the control plane of your Kubernetes cluster is managed, but also the worker nodes.

Digital Ocean

Digital Ocean is not a cloud provider on the order of the big three (GCP, AWS, Azure), but it does provide a managed Kubernetes solution and it has data centers across the world (US, Canada, Europe, Asia). It is also much cheaper compared to the alternatives, and cost is a major deciding factor when choosing a cloud provider. With Digital Ocean, the control plane doesn’t cost anything. Besides lower prices Digital Ocean’s claim to fame is simplicity.

DOKS (Digital Ocean Kubernetes Service) gives you a managed Kubernetes control plane (which can be highly available) and integration with Digital Ocean’s droplets (for nodes and node pools), load balancers, and block storage volumes. This covers all the basic needs. Your clusters are of course CNCF conformant.

Digital Ocean will take care of system upgrades, security patches, and the installed packages on the control plane as well as the worker nodes.

Other cloud providers

GCP, AWS, and Azure are leading the pack, but there are quite a few other companies that offer managed Kubernetes services. In general, I recommend using these providers if you already have significant business connections or integrations.

Once upon a time in China

If you operate in China with its special constraints and limitations, you should probably use a Chinese cloud platform. There are three big ones: Alibaba, Tencent, and Huawei.

The Chinese Alibaba cloud is an up-and-comer on the cloud platform scene. It mimics AWS pretty closely, although its English documentation leaves a lot to be desired. The Alibaba cloud supports Kubernetes in several ways via its ACK (Alibaba Container service for Kubernetes) and allows you to:

• Run your own dedicated Kubernetes cluster (you must create 3 master nodes and upgrade and maintain them)
• Use the managed Kubernetes cluster (you’re just responsible for the worker nodes)
• Use the serverless Kubernetes cluster via ECI (Elastic Container Instances), which is similar to Fargate and ACI

ACK is a CNCF-certified Kubernetes distribution. If you need to deploy cloud-native applications in China, then ACK looks like a solid option.

See https://www.alibabacloud.com/product/kubernetes.

Tencent is another large Chinese company with its own cloud platform and Kubernetes support. TKE (Tencent Kubernetes Engine) seems less mature than ACK. See https://intl.cloud.tencent.com/products/tke.

Finally, the Huawei cloud platform offers CCE (Cloud Container Engine), which is built on Kubernetes. It supports VMs, bare metal, and GPU accelerated instances. See https://www.huaweicloud.com/intl/en-us/product/cce.html.

IBM Kubernetes service

IBM is investing heavily in Kubernetes. It acquired Red Hat at the end of 2018. Red Hat was of course a major player in the Kubernetes world, building its OpenShift Kubernetes-based platform and contributing RBAC to Kubernetes. IBM has its own cloud platform and it offers a managed Kubernetes cluster. You can try it out for free with $200 credit and there is also a free tier.

IBM is also involved in the development of Istio and Knative, so you can expect IKS to have deep integration with those technologies.

IKS offers integration with a lot of IBM services.

See https://www.ibm.com/cloud/kubernetes-service.

Oracle Container Service

Oracle also has a cloud platform and of course, it offers a managed Kubernetes service too, with high availability, bare-metal instances, and multi-AZ support.

OKE supports ARM and GPU instances and also offers a few control plane options.

See https://www.oracle.com/cloud/cloud-native/container-engine-kubernetes/.

In this section, we covered the cloud-provider interface and looked at the recommended ways to create Kubernetes clusters on various cloud providers. The scene is still young and the tools evolve quickly. I believe convergence will happen soon. Kubeadm has matured and is the underlying foundation of many other tools to bootstrap and create Kubernetes clusters on and off the cloud. Let’s consider now what it takes to create bare-metal clusters where you have to provision the hardware and low-level networking and storage too.