We managed to build a system that scales (and de-scales) our worker nodes automatically. Even though we might need to extend alerts to other types of metrics, the system we created is already good as it is. Kind of... The problem is that new nodes are empty. They do not host any services until we deploy new ones if we updated some of those that are already running inside our cluster. That, in itself, is not a problem if we deploy new releases to production often.
Let's say that, on average, we deploy a new release every hour. That would mean that our newly added nodes will be empty only for a short while. Our deployment pipelines will re-balance the cluster. But, what happens if we do not deploy any new release until the next day? Having empty nodes for a while is not a big problem since our services have memory reservations based...
