Deploying workloads using Kustomize

In this section, we will show you how to generate resources from files and compose and customize collections of resources in Kubernetes. You will learn about the declarative management of Kubernetes objects using Kustomize.

Getting ready

Make sure you have a Kubernetes cluster ready and kubectl configured to manage the cluster resources.

The source files created in this section can be found on my GitHub repository located at https://github.com/k8sdevopscookbook/src/tree/master/chapter2/kustomize. It is recommended that you follow the instructions to create and edit them and only use the files in the repository to compare with your files if you run into an issue.

How to do it…

This section is further divided into the following subsections to ease the process:

• Validating the Kubernetes cluster version
• Generating Kubernetes resources from files
• Creating a base for a development and production Deployment

Validating the Kubernetes cluster version

For Kustomize to function, Kubernetes cluster version 1.14.0 or later is required, since Kustomize support is only included with kubectl v.1.14.0 and later.

1. List the nodes to confirm your Kubernetes cluster version and make sure that it is 1.14.0 or later:

$ kubectl get nodes
 NAME STATUS ROLES AGE VERSION
 ip-172-20-112-25.ec2.internal Ready master 7h19m v1.15.0
 ip-172-20-126-108.ec2.internal Ready node 7h18m v1.15.0
 ip-172-20-51-209.ec2.internal Ready node 7h18m v1.15.0
 ip-172-20-92-89.ec2.internal Ready node 7h19m v1.15.0

In the preceding example, the version shows v1.15.0.

Generating Kubernetes resources from files

Let's learn how to customize the nginx rollout we did in the previous recipe using Kustomize this time:

1. Create a directory named nginx:

$ mkdir nginx

2. Copy the deployment-nginx.yaml file you created in the Deploying workload using YAML files recipe under the nginx directory. This file still uses image: nginx:1.7.9 as the container image:

$ cp deployment-nginx.yaml ./nginx/

3. Create a kustomization.yaml file by specifying a new image version:

$ cat <<EOF >./nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment-nginx.yaml
images:
  - name: nginx
    newName: nginx
    newTag: 1.16.0
commonAnnotations:
  kubernetes.io/change-cause: "Initial deployment with 1.16.0"
EOF

4. Check that the new version is injected into your Deployment by running the following command. In the output, you will see image: nginx:1.16.0 instead of the original image version nginx:1.7.9 that we have previously used in the deployment-nginx.yaml file:

$ kubectl kustomize ./nginx/

5. Apply the Deployment with customizations using the -k parameter:

$ kubectl apply -k nginx

6. Create a new kustomization.yaml file by specifying a newer image version:

$ cat <<EOF > nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment-nginx.yaml
images:
  - name: nginx
    newName: nginx
    newTag: 1.17.0
commonAnnotations:
 kubernetes.io/change-cause: "image updated to 1.17.0"
EOF

7. Apply the customized Deployment using the -k parameter:

$ kubectl apply -k nginx

8. Now, display the rollout history for the Deployment:

$ kubectl rollout history deployment nginx-deployment
deployment.extensions/nginx-deployment
REVISION CHANGE-CAUSE
1        Initial deployment with 1.16.0
2        image updated to 1.17.0

Now you have learned how to edit, scale up, and also roll out a new version of the application using Kustomize.

Creating a base for a development and production Deployment

Let's perform the following steps to create a base for a local Docker image registry Deployment that we will use later in this chapter:

1. Create a directory named registry and another one underneath called base:

$ mkdir registry && mkdir registry/base

2. Under registry/base, download the Deployment file named deployment-registry.yaml from the example repository:

$ cd registry/base/
$ wget https://raw.githubusercontent.com/k8sdevopscookbook/src/master/chapter2/kustomize/registry/base/deployment-registry.yaml

3. Review the file to understand its structure. You will see that it is a Deployment manifest consisting of two containers named registry and registryui. You will see that the registry container has a volumeMount named registry-storage and this volume is provided by a persistent volume claim named registry-pvc:

$ cat deployment-registry.yaml
apiVersion: extensions/v1beta1
kind: Deployment
# actual file is longer, shortened to highlight important structure of the file only
      - image: registry:2
#....#
        - name: registry-storage
          mountPath: /var/lib/registry
#....#
      - name: registryui
        image: hyper/docker-registry-web:latest
#....#
      - name: registry-storage
        persistentVolumeClaim:
          claimName: registry-pvc

4. Under the same registry/base, download the service manifest file named service-registry.yaml from the example repository:

$ wget https://raw.githubusercontent.com/k8sdevopscookbook/src/master/chapter2/kustomize/registry/base/service-registry.yaml

5. Review the file to understand its structure. You will see that it is a service manifest that exposes the service on each Node's IP at a static port; in this recipe, port 5000 for the registry service and port 80 for the registry-ui:

$ cat <<EOF > registry/base/service-registry.yaml
kind: Service
# actual file is longer, shortened to highlight important structure of the file only
  type: NodePort
  ports:
    - name: registry
      port: 5000
      protocol: TCP 
      nodePort: 30120
    - name: registry-ui
      port: 80
      protocol: TCP
      nodePort: 30220 
#....#

6. Create a PersistentVolumeClaim manifest file named pvc-registry.yaml with the following content:

$ cat <<EOF > registry/base/pvc-registry.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
  labels:
    app: kube-registry-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10G
EOF

7. And finally, create the kustomization.yaml file. The following command will create the Kustomize resource content with the three separate manifest files we created previously:

$ cat <<EOF >./registry/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment-registry.yaml
  - service-registry.yaml
  - pvc-registry.yaml
EOF

8. Now, create two overlays to be used for development and production Deployments. The first one is for development:

$ mkdir registry/overlays && mkdir registry/overlays/dev 
$ cat <<EOF >./registry/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
namePrefix: dev-
commonAnnotations:
  note: Hello, I am development!
EOF

9. And the second manifest will create the overlay for production:

$ mkdir registry/overlays/prod
$ cat <<EOF >./registry/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
namePrefix: prod-
commonAnnotations:
  note: Hello, I am production!
EOF

10. Check that dev and prod prefixes are injected into your Deployment. When you point to the prod folder, the annotation note will display "Hello, I am production!":

$ kubectl kustomize ./registry/overlays/prod/
# result shortened to highlight the annotation
metadata:
  annotations:
    note: Hello, I am production!
  labels:
    app: kube-registry-pv-claim
  name: prod-registry-pvc
#...#

11. When you point to the dev folder, the annotation note will display "Hello, I am development!":

$ kubectl kustomize ./dev/
... # removed
metadata:
  annotations:
    note: Hello, I am development!
  labels:
    app: kube-registry-pv-claim
  name: dev-registry-pvc
... # removed

12. Now, deploy the dev version of your application:

$ kubectl apply -k ./registry/overlays/dev

Similarly, you can inject labels, patch image versions, change the number of replicas, and deploy resources into a different namespace.

How it works...

This recipe showed you how to manage and implement basic version control of your configuration files using Git.

In the Creating a base for a development and production Deployment recipe, the resources we created between step 2 and step 6 under the base directory represents an upstream repository of the application/workload, and the customizations we created between step 8 and step 10 under the overlay directory are the changes you control and store in your repository.

Later, if you need to see the difference of a variant, you can use the diff parameter as follows:

$ kubectl diff -k registry/overlays/prod/

By separating the changes from the base, we were able to customize template-free YAML files for multiple purposes, leaving the original YAML files unchanged, making the version controlling of source and changes possible.

See also

• Kustomize concepts overview slides: https://speakerdeck.com/spesnova/introduction-to-kustomize
• Declarative application management in Kubernetes background whitepaper – a highly recommended read: https://goo.gl/T66ZcD
• Common terms in Kustomize: https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md
• Additional Kustomize examples: https://github.com/kubernetes-sigs/kustomize/tree/master/examples