Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Engineering Lakehouses with Open Table Formats
  • Table Of Contents Toc
Engineering Lakehouses with Open Table Formats

Engineering Lakehouses with Open Table Formats

By : Dipankar Mazumdar, Vinoth Govindarajan
close
close
Engineering Lakehouses with Open Table Formats

Engineering Lakehouses with Open Table Formats

By: Dipankar Mazumdar, Vinoth Govindarajan

Overview of this book

Engineering Lakehouses with Open Table Formats provides detailed insights into lakehouse concepts, and dives deep into the practical implementation of open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake. You’ll explore the internals of a table format and learn in detail about the transactional capabilities of lakehouses. You’ll also get hands on with each table format with exercises using popular computing engines, such as Apache Spark, Flink, Trino, and Python-based tools. The book addresses advanced topics, including performance optimization techniques and interoperability among different formats, equipping you to build production-ready lakehouses. With step-by-step explanations, you’ll get to grips with the key components of lakehouse architecture and learn how to build, maintain, and optimize them. By the end of this book, you’ll be proficient in evaluating and implementing open table formats, optimizing lakehouse performance, and applying these concepts to real-world scenarios, ensuring you make informed decisions in selecting the right architecture for your organization’s data needs.
Table of Contents (15 chapters)
close
close
13
Other Books You May Enjoy
14
Index

Summary

This chapter showcased the practical application of open table formats—Apache Iceberg, Apache Hudi, and Delta Lake—through real-world use cases that span manufacturing analytics, real-time CDC, and machine learning workflows.

We began with Acme Manufacturing’s journey to unify operational data using an Iceberg-based lakehouse. Their architecture, built with MinIO, Dremio, Nessie, and Superset, demonstrated how to implement a Bronze-Silver-Gold table model with Git-like data branching for WAP workflows. Iceberg’s support for schema evolution and SQL accessibility enabled seamless team adoption and real-time insights.

Next, we explored how GlobalMart leveraged Apache Hudi to implement a real-time CDC pipeline. By integrating Debezium, Kafka, Spark Structured Streaming, and Hudi, the team achieved near-real-time data freshness, time travel, and incremental query support. This architecture streamlined compliance reporting, reduced infrastructure...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Engineering Lakehouses with Open Table Formats
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon