One of the most important benefits of tracing is the degree of visibility it gives into the operations of complex distributed systems. Ben Sigelman, one of the creators of the tracing infrastructure at Google, said that when the team deployed tracing there in 2005, "it was like someone finally turned the lights on: everything from ordinary programming errors to broken caches to bad network hardware to unknown dependencies came into plain view" [1]. This visibility is important not only when investigating a specific issue or incident, but on a regular basis as the foundation of understanding how the whole application functions, how it is architected, and so on.
Imagine you are a new team member who is not familiar with the system you are hired to work on. How would you learn about the system architecture, deployment modes, or bottlenecks? Documentation is rarely up to date enough to be a reliable source of information. Most engineers would rather build new features...