Book Image

Data Engineering with Apache Spark, Delta Lake, and Lakehouse

By : Manoj Kukreja
5 (2)
Book Image

Data Engineering with Apache Spark, Delta Lake, and Lakehouse

5 (2)
By: Manoj Kukreja

Overview of this book

In the world of ever-changing data and schemas, it is important to build data pipelines that can auto-adjust to changes. This book will help you build scalable data platforms that managers, data scientists, and data analysts can rely on. Starting with an introduction to data engineering, along with its key concepts and architectures, this book will show you how to use Microsoft Azure Cloud services effectively for data engineering. You'll cover data lake design patterns and the different stages through which the data needs to flow in a typical data lake. Once you've explored the main features of Delta Lake to build data lakes with fast performance and governance in mind, you'll advance to implementing the lambda architecture using Delta Lake. Packed with practical examples and code snippets, this book takes you through real-world examples based on production scenarios faced by the author in his 10 years of experience working with big data. Finally, you'll cover data lake deployment strategies that play an important role in provisioning the cloud resources and deploying the data pipelines in a repeatable and continuous way. By the end of this data engineering book, you'll know how to effectively deal with ever-changing data and create scalable data pipelines to streamline data science, ML, and artificial intelligence (AI) tasks.
Table of Contents (17 chapters)
1
Section 1: Modern Data Engineering and Tools
5
Section 2: Data Pipelines and Stages of Data Engineering
11
Section 3: Data Engineering Challenges and Effective Deployment Strategies

Performing time travel

A unique feature of Delta Lake is its ability to perform time travel. By using this feature, you can query and restore previous snapshots of your table. Access to previous snapshots is granted by using the versionAsOf option.

Important Note

The time travel functionality in Delta Lake implements data lineage. Data lineage is an extremely critical tool for data audits and compliance purposes. The same feature comes in handy for data engineers who are trying to trace data anomalies.

  1. This is how you can query previous versions of the delta table. In this example, we are querying version 0 of the table – in other words, when it was created:
    %sql
    SELECT * FROM store_orders VERSION AS OF 0 WHERE order_number=5;

    This results in the following output:

    Figure 6.25 – Checking the data in the sales_orders table for the sample row for version 0

    Notice how the previous version of the table shows the sale_ price value as 98.41.

  2. This time we will delete...