Book Image

Data Ingestion with Python Cookbook

By : Gláucia Esppenchutz
Book Image

Data Ingestion with Python Cookbook

By: Gláucia Esppenchutz

Overview of this book

Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges. You’ll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you’ll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation. By the end of the book, you’ll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.
Table of Contents (17 chapters)
Part 1: Fundamentals of Data Ingestion
Part 2: Structuring the Ingestion Pipeline

Part 1: Fundamentals of Data Ingestion

In this part, you will be introduced to the fundamentals of data ingestion and data engineering, passing through the basic definition of an ingestion pipeline, the common types of data sources, and the technologies involved.

This part has the following chapters:

  • Chapter 1, Introduction to Data Ingestion
  • Chapter 2, Principals of Data Access – Accessing Your Data
  • Chapter 3, Data Discovery – Understanding Our Data Before Ingesting It
  • Chapter 4, Reading CSV and JSON Files and Solving Problems
  • Chapter 5, Ingesting Data from Structured and Unstructured Databases
  • Chapter 6, Using PySpark with Defined and Non-Defined Schemas
  • Chapter 7, Ingesting Analytical Data