On the first look, the script we created works correctly. Doesn't it?. I've seen similar scripts in other places, and there is only one thing I have to say. Do not run this pipeline in production!!! It is too dangerous. It can easily crash your entire cluster or make your service disappear. Can you guess why?
Let us imagine the following situation. Prometheus detects that certain threshold is reached (for example, memory utilization, response time, and so on) and send a notification to Alertmanager. It sends a build request to Jenkins which, in turn, scales the service by increasing the number of replicas by one. So far, so good.
What happens if scaling does not resolve the problem? What if the threshold reached in Prometheus persists? After a while, the process will be repeated, and the service will be scaled up one more time. That might...