Automate me!

So, we've established how great it is to be able to discover the services available on our network without cumbersome static addressing, and how to utilize that to ensure the secure delivery of messages among our services. Wouldn't it be great to share this dynamic information with other components in the network? Well, why would we want to do that? To understand how this functionality can improve the lives of so many people, let's jump into the way-back machine and once again recall what we used to do, and often still do, when making adjustments to the network:

1. Yeah! We have our application deployed into the network and I can communicate with it. However, I can't reach the service my application needs to hit. Better call the network team.
2. The network team told me that the problem must be on my end because the destination service is alive and responding to their monitoring tools.
3. After a few days of troubleshooting, we learn that there is a firewall in the path that is protecting that service from nefarious creatures.
4. So, next, we submit a ticket to the firewall team and request a rule be configured to allow traffic from, you guessed it, my IP address.
5. Of course, the firewall team is not only getting hammered with requests, but they know that one wrong move on the list of firewall rules and the number of tickets will be the least of their problems, so extreme care must be taken.
6. Eventually, the team is able to safely add and test the new rule, and we're off and running.

Now after all of that, let's hope that we didn't forget to include any important information in that ticket (such as an additional port). Not to mention that when the address changes (not if), we'll need to enter the process again. This process continues today, even with cloud automation, for a number of different reasons. In my personal experience, these changes are the ones that tend to create the most confusion and delay in any project, and in a dynamic world, the changes aren't slowing down.

Hopefully, you can see how the knowledge that Consul is aware of plays such a critical role in this scenario. As services are added, updated, and removed, the Consul server cluster is aware of the changes, thanks to the distributed agents. Now the main challenge is how to get that information from Consul out to my network equipment, and there are a few ways to solve that.

One of the most straightforward methods for network devices to learn about service changes is for the equipment to learn about those changes directly from Consul via its API. This functionality is already in use with the Consul integration with the F5 Big-IP load balancer using the F5 Application Services interface. Whenever Consul discovers network updates, the load balancer learns of these updates and can adjust the load balancing pools automatically. Although this integration does require the network components to develop functionality specifically for the Consul API, there is no Consul agent required on the load balancer itself.

Figure 1.6 – Load balancer querying Consul for service updates

An alternative to this level of integration is to have Consul post updates to the service availability to a messaging service, such as ActiveMQ or RabbitMQ. Any messaging service, or other application, that can receive an HTTP message can receive service updates from Consul, without having to query it directly.

Figure 1.7 – Consul dynamically updating a message bus

A more common method of integration is utilizing Consul's template feature. Just as it sounds, you can create a configuration file template to match whatever device that you need to manage. Within that template, you specify areas where you need input from Consul, such as the IP address of a particular endpoint! If you remember, Consul makes intelligent decisions based on the data it receives. All of that data the Consul servers receive from the clients are pieces in the puzzle of an intelligent and dynamic network. A very simplified analogy would be setting up a payee for paying our bills. I presume this is something many of us do, certainly more common than writing checks. On our bank website, we establish a payee – some company or person that we pay money to every month. If we look at our monthly electricity bills, for example, the amount we pay varies based on the level of electricity my kids use that month. Why can't they ever learn to turn off the lights? So, we have our payee set up, and we know we are paying the electric company. That is our template. All we need is the data point of the amount due. Once the electric company (the client) informs us (the server) of the amount due, we can fill in the amount and let the electronic payment go. Hopefully, this will happen before the electricity is shut off!

Figure 1.8 – Terrific templating

That template functionality, however, doesn't only apply to configuration files. HashiCorp does have several other products besides Consul, one of which is Terraform. If you aren't familiar with Terraform, it provides a standardized method to define system infrastructure as code for multiple cloud providers, but it also has providers for multiple applications. This includes firewalls, load balancers, monitoring systems, and so on. Now imagine coupling the dynamic aspect of Consul service discovery, with Terraform's infrastructure-as-code platform!

Important Note

The Consul synchronization functionality with Terraform, called Consul-Terraform-Sync, is in public beta at the time of writing and supports a limited set of partner modules for automation. With HashiCorp's development velocity, by the time you're reading this, the functionality is likely fully supported.