Book Image

Data Observability for Data Engineering

By : Michele Pinto, Sammy El Khammal
Book Image

Data Observability for Data Engineering

By: Michele Pinto, Sammy El Khammal

Overview of this book

In the age of information, strategic management of data is critical to organizational success. The constant challenge lies in maintaining data accuracy and preventing data pipelines from breaking. Data Observability for Data Engineering is your definitive guide to implementing data observability successfully in your organization. This book unveils the power of data observability, a fusion of techniques and methods that allow you to monitor and validate the health of your data. You’ll see how it builds on data quality monitoring and understand its significance from the data engineering perspective. Once you're familiar with the techniques and elements of data observability, you'll get hands-on with a practical Python project to reinforce what you've learned. Toward the end of the book, you’ll apply your expertise to explore diverse use cases and experiment with projects to seamlessly implement data observability in your organization. Equipped with the mastery of data observability intricacies, you’ll be able to make your organization future-ready and resilient and never worry about the quality of your data pipelines again.
Table of Contents (17 chapters)
1
Part 1: Introduction to Data Observability
4
Part 2: Implementing Data Observability
8
Part 3: How to adopt Data Observability in your organization
12
Part 4: Appendix

Checklist to implement data observability

In this section, we will delve into a comprehensive list of considerations to keep in mind when you embark on the journey of implementing a data observability solution. These questions will not only guide you through your initial project into data observability but also prove invaluable as you progress to more advanced implementations. By carefully addressing these considerations, you will be able to establish a robust foundation for your data observability initiative, one that not only aligns with your organization’s objectives but also harnesses its full potential for maximum benefit.

The questions we need to answer are the following:

  • Which pipeline should I select to start with the implementation?
  • How many applications should I include in the scope?
  • What criteria are important to select the observability tool?
  • How do we define the set of metrics we want to track?
  • How will alerts and notifications be configured...