Understanding observability on Kubernetes
No production system is complete without a way to monitor it. In software, we define observability as the ability to, at any point in time, understand how our system is performing (and, in the best case, why). Observability grants significant benefits in security, performance, and operational capacity. By knowing how your system is responding at the VM, container, and application level, you can tune it to perform efficiently, react quickly to events, and more easily troubleshoot bugs.
For instance, let's take a scenario where your application is running extremely slowly. In order to find the bottleneck, you may look at the application code itself, the resource specifications of the Pod, the number of Pods in the deployment, the memory and CPU usage at the Pod level or Node level, and externalities such as a MySQL database running outside your cluster.
By adding observability tooling, you would be able to diagnose many of these variables...