There's no point in running an environment and having products out in the wild if you don't understand how they are behaving; you can't just assume everything is fine. In fact, you need to work from the standpoint that nothing is fine and that it can all break at any time.
When you look at your environment in this manner, you will begin to realize that you need data in order to figure out what the issues are and how you might be able to resolve them. In theory, you will need to observe everything. The downside to all of this is that you will end up with a surplus of data—and, as a result, you will probably be swamped. In practice, however, observing everything (that is, every possible aspect of all of the tools in play) is impractical. You will need to decide what to watch in order to receive the best possible amount of information to be alerted to, find, and understand where an issue has developed.
We will start with logging.