Book Image

Mastering Distributed Tracing

By : Yuri Shkuro
Book Image

Mastering Distributed Tracing

By: Yuri Shkuro

Overview of this book

Mastering Distributed Tracing will equip you to operate and enhance your own tracing infrastructure. Through practical exercises and code examples, you will learn how end-to-end tracing can be used as a powerful application performance management and comprehension tool. The rise of Internet-scale companies, like Google and Amazon, ushered in a new era of distributed systems operating on thousands of nodes across multiple data centers. Microservices increased that complexity, often exponentially. It is harder to debug these systems, track down failures, detect bottlenecks, or even simply understand what is going on. Distributed tracing focuses on solving these problems for complex distributed systems. Today, tracing standards have developed and we have much faster systems, making instrumentation less intrusive and data more valuable. Yuri Shkuro, the creator of Jaeger, a popular open-source distributed tracing system, delivers end-to-end coverage of the field in Mastering Distributed Tracing. Review the history and theoretical foundations of tracing; solve the data gathering problem through code instrumentation, with open standards like OpenTracing, W3C Trace Context, and OpenCensus; and discuss the benefits and applications of a distributed tracing infrastructure for understanding, and profiling, complex systems.

Related Content you might be interested in

Current Title:

Small book image
Mastering Distributed Tracing
Yuri Shkuro
Feb, 2019 | 444 pages
Small book image
Modern Distributed Tracing in .NET
Liudmila Molkova
Jun, 2023 | 336 pages
Small book image
Cloud-Native Observability with OpenTelemetry
Alex Boten
May, 2022 | 386 pages
Small book image
Hands-On Enterprise Java Microservices with Eclipse MicroProfile
Scott Stark | Antoine SabotDurand | Pavol Loffay | Cesar Saavedra | Jeff Mesnil | Heiko W Rupp
Aug, 2019 | 256 pages
Table of Contents (21 chapters)
Mastering Distributed Tracing
Mastering Distributed Tracing
Contributors
Contributors
Preface
Preface
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Leave a review - let other readers know what you think
Free Chapter
1
Why Distributed Tracing?
Why Distributed Tracing?
Microservices and cloud-native applications
What is observability?
The observability challenge of microservices
Traditional monitoring tools
Distributed tracing
My experience with tracing
Why this book?
Summary
References
2
Take Tracing for a HotROD Ride
Take Tracing for a HotROD Ride
Prerequisites
Start Jaeger
Meet the HotROD
The architecture
The data flow
Contextualized logs
Span tags versus logs
Identifying sources of latency
Resource usage attribution
Summary
References
3
Distributed Tracing Fundamentals
Distributed Tracing Fundamentals
The idea
Request correlation
Anatomy of distributed tracing
Sampling
Preserving causality
Trace models
Clock skew adjustment
Trace analysis
Summary
References
4
Instrumentation Basics with OpenTracing
Instrumentation Basics with OpenTracing
Prerequisites
OpenTracing
Exercise 1 – the Hello application
Exercise 2 – the first trace
Exercise 3 – tracing functions and passing context
Exercise 4 – tracing RPC requests
Exercise 5 – using baggage
Exercise 6 – auto-instrumentation
Exercise 7 – extra credit
Summary
References
5
Instrumentation of Asynchronous Applications
Instrumentation of Asynchronous Applications
Prerequisites
The Tracing Talk chat application
Instrumenting with OpenTracing
Instrumenting asynchronous code
Summary
References
6
Tracing Standards and Ecosystem
Tracing Standards and Ecosystem
Styles of instrumentation
Anatomy of tracing deployment and interoperability
Five shades of tracing
Know your audience
The ecosystem
Summary
References
7
Tracing with Service Meshes
Tracing with Service Meshes
Service meshes
Observability via a service mesh
Prerequisites
The Hello application
Distributed tracing with Istio
Using Istio to generate a service graph
Distributed context and routing
Summary
References
8
All About Sampling
All About Sampling
Head-based consistent sampling
Tail-based consistent sampling
Partial sampling
Summary
References
9
Turning the Lights On
Turning the Lights On
Tracing as a knowledge base
Service graphs
Performance analysis
Long-term profiling
Summary
References
10
Distributed Context Propagation
Distributed Context Propagation
Brown Tracing Plane
Pivot tracing
Chaos engineering
Traffic labeling
Summary
References
11
Integration with Metrics and Logs
Integration with Metrics and Logs
Three pillars of observability
Prerequisites
The Hello application
Integration with metrics
Integration with logs
Summary
References
12
Gathering Insights with Data Mining
Gathering Insights with Data Mining
Feature extraction
Components of a data mining pipeline
Feature extraction exercise
Prerequisites
The Span Count job
Observing trends
Historical analysis
Ad hoc analysis
Summary
References
13
Implementing Tracing in Large Organizations
Implementing Tracing in Large Organizations
Why is it hard to deploy tracing instrumentation?
Reduce the barrier to adoption
Where to start
Building the culture
Tracing Quality Metrics
Troubleshooting guide
Don't be on the critical path
Summary
References
14
Under the Hood of a Distributed Tracing System
Under the Hood of a Distributed Tracing System
Why host your own?
Bet on emerging standards
Architecture and deployment modes
Monitoring and troubleshooting
Resiliency
Summary
References
15
Afterword
Afterword
References
Index
Index

Prerequisites

As you can see in Figure 12.2, there are quite a few components that we need to run in order to bring up the exercise architecture. Fortunately, most of the components can be run in Docker containers. The only two components that we will need to run directly are the microservices simulator (in Go) and the Apache Flink feature extraction job (Java).

This section provides instructions on setting up the environment and running the exercise.

Project source code

The code of the Flink job can be found in the Chapter12/ directory of the book's source code repository on GitHub. Please refer to Chapter 4, Instrumentation Basics with OpenTracing, for instructions on how to download it, then switch to the Chapter12 directory, from where all example code can be run.

The source code of the application is organized in the following structure:

Mastering-Distributed-Tracing/
  Chapter12/
    Makefile
    docker-compose.yml
    elasticsearch.yml
    es-create-mapping.json
    hotrod-original.json...
Previous Section
End of Section 5
Next Section