Spotify is one of the companies of reference when building microservices-oriented applications. They are extremely creative and talented when it boils down to coming up with new ideas.
One of my favourite ideas among them is what they call the Simian Army. I like to call it active monitoring.
In this book, I have talked a lot times about how humans fail at performing different tasks. No matter how much effort you put in to creating your software, there are going to be bugs that will compromise the stability of the system.
This is a big problem, but it becomes a huge deal when, with the modern cloud providers, your infrastructure is automated with a script.
Think about it, what happens if in a pool of thousand servers, three of them have the time zone out of sync with the rest of the servers? Well, depending on the nature of your system, it could be fine or it could be a big deal. Can you imagine your bank giving you a statement with disordered...