Book Image

Mastering Distributed Tracing

By : Yuri Shkuro
Book Image

Mastering Distributed Tracing

By: Yuri Shkuro

Overview of this book

Mastering Distributed Tracing will equip you to operate and enhance your own tracing infrastructure. Through practical exercises and code examples, you will learn how end-to-end tracing can be used as a powerful application performance management and comprehension tool. The rise of Internet-scale companies, like Google and Amazon, ushered in a new era of distributed systems operating on thousands of nodes across multiple data centers. Microservices increased that complexity, often exponentially. It is harder to debug these systems, track down failures, detect bottlenecks, or even simply understand what is going on. Distributed tracing focuses on solving these problems for complex distributed systems. Today, tracing standards have developed and we have much faster systems, making instrumentation less intrusive and data more valuable. Yuri Shkuro, the creator of Jaeger, a popular open-source distributed tracing system, delivers end-to-end coverage of the field in Mastering Distributed Tracing. Review the history and theoretical foundations of tracing; solve the data gathering problem through code instrumentation, with open standards like OpenTracing, W3C Trace Context, and OpenCensus; and discuss the benefits and applications of a distributed tracing infrastructure for understanding, and profiling, complex systems.
Table of Contents (21 chapters)
Mastering Distributed Tracing
Contributors
Preface
Other Books You May Enjoy
Leave a review - let other readers know what you think
15
Afterword
Index

Monitoring and troubleshooting


As with any other distributed system, the tracing backend itself must be observable. Jaeger exposes numerous metrics for all of its components, from the client libraries to the backend components. Typical metrics include stats for the number of spans created, received, processed, sampled and non-sampled, and so on. As an example, the following are some of the metrics emitted by the Jaeger client in the frontend service of the HotROD application (from Chapter 2, Take Tracing for a HotRod Ride):

hotrod_frontend_jaeger_started_spans{sampled="n"} 0
hotrod_frontend_jaeger_started_spans{sampled="y"} 24
hotrod_frontend_jaeger_finished_spans 24
hotrod_frontend_jaeger_traces{sampled="n",state="joined"} 0
hotrod_frontend_jaeger_traces{sampled="n",state="started"} 0
hotrod_frontend_jaeger_traces{sampled="y",state="joined"} 0
hotrod_frontend_jaeger_traces{sampled="y",state="started"} 1

As we can see, it reports the number of started and finished spans, partitioned by the...