Book Image

Data Observability for Data Engineering

By : Michele Pinto, Sammy El Khammal
Book Image

Data Observability for Data Engineering

By: Michele Pinto, Sammy El Khammal

Overview of this book

In the age of information, strategic management of data is critical to organizational success. The constant challenge lies in maintaining data accuracy and preventing data pipelines from breaking. Data Observability for Data Engineering is your definitive guide to implementing data observability successfully in your organization. This book unveils the power of data observability, a fusion of techniques and methods that allow you to monitor and validate the health of your data. You’ll see how it builds on data quality monitoring and understand its significance from the data engineering perspective. Once you're familiar with the techniques and elements of data observability, you'll get hands-on with a practical Python project to reinforce what you've learned. Toward the end of the book, you’ll apply your expertise to explore diverse use cases and experiment with projects to seamlessly implement data observability in your organization. Equipped with the mastery of data observability intricacies, you’ll be able to make your organization future-ready and resilient and never worry about the quality of your data pipelines again.
Table of Contents (17 chapters)
1
Part 1: Introduction to Data Observability
4
Part 2: Implementing Data Observability
8
Part 3: How to adopt Data Observability in your organization
12
Part 4: Appendix

Turning SLOs into rules

In this section, we will see how objectives can be turned into actionable rules by creating contextual checkpoints from the pipeline or externally. At the start of any rule is the expectation, which can be defined as "What does the consumer expect from the dataset?"

An expectation formalizes the objective into a rule and the corresponding metric to be tracked. The expectation is then a good way to document the objectives and the metrics needed to respect them. The two components of the expectation have their importance: the rule tells the observer how the data should behave, and the metric is used to detect whether the behavior is deviant or not.

Let’s look at the different types of rules that we can set.

Different types of rules

The backbone of a rule is the indicator. Based on this, a rule can be set and will start checking how the metric is behaving. These rules are often guided by the principles of data quality discussed in Chapter...