In this chapter, we'll be focusing on service discovery. For this, we'll be deploying two new instances to simulate a scenario where Prometheus generates targets dynamically using a popular service discovery software. This approach will allow us to not only expose the required configurations, but also validate how everything works together.
The setup we'll be using resembles the following diagram:
The usual deployment pattern for Consul is to have an agent running in client mode on every node in the infrastructure, which will then contact Consul instances running in server mode. Furthermore, client instances act as API proxies, so it is common practice for Prometheus Consul service discovery to be configured using the localhost. However, to make their different responsibilities clear...