Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Building Big Data Pipelines with Apache Beam

By : Lukavský

3.7 (9)

Building Big Data Pipelines with Apache Beam

3.7 (9)

By: Lukavský

Overview of this book

Apache Beam is an open source unified programming model for implementing and executing data processing pipelines, including Extract, Transform, and Load (ETL), batch, and stream processing. This book will help you to confidently build data processing pipelines with Apache Beam. You’ll start with an overview of Apache Beam and understand how to use it to implement basic pipelines. You’ll also learn how to test and run the pipelines efficiently. As you progress, you’ll explore how to structure your code for reusability and also use various Domain Specific Languages (DSLs). Later chapters will show you how to use schemas and query your data using (streaming) SQL. Finally, you’ll understand advanced Apache Beam concepts, such as implementing your own I/O connectors. By the end of this book, you’ll have gained a deep understanding of the Apache Beam model and be able to apply it to solve problems.

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Share Your Thoughts

Section 1 Apache Beam: Essentials

Section 1 Apache Beam: Essentials

Free Chapter

Chapter 1: Introduction to Data Processing with Apache Beam

Chapter 1: Introduction to Data Processing with Apache Beam

Technical requirements

Why Apache Beam?

Writing your first pipeline

Running our pipeline against streaming data

Exploring the key properties of unbounded data

Measuring event time progress inside data streams

Assigning data to windows

Unifying batch and streaming data processing

Summary

Chapter 2: Implementing, Testing, and Deploying Basic Pipelines

Chapter 2: Implementing, Testing, and Deploying Basic Pipelines

Technical requirements

Setting up the environment for this book

Task 1 – Calculating the K most frequent words in a stream of lines of text

Task 2 – Calculating the maximal length of a word in a stream

Specifying the PCollection Coder object and the TypeDescriptor object

Understanding default triggers, on time, and closing behavior

Introducing the primitive PTransform object – Combine

Task 3 – Calculating the average length of words in a stream

Task 4 – Calculating the average length of words in a stream with fixed lookback

Ensuring pipeline upgradability

Task 5 – Calculating performance statistics for a sport activity tracking application

Introducing the primitive PTransform object – GroupByKey

Introducing the primitive PTransform object – Partition

Summary

Chapter 3: Implementing Pipelines Using Stateful Processing

Chapter 3: Implementing Pipelines Using Stateful Processing

Technical requirements

Task 6 – Using an external service for data augmentation

Introducing the primitive PTransform object – stateless ParDo

Task 7 – Batching queries to an external RPC service

Task 8 – Batching queries to an external RPC service with defined batch sizes

Introducing the primitive PTransform object – stateful ParDo

Using side outputs

Defining droppable data in Beam

Task 9 – Separating droppable data from the rest of the data processing

Task 10 – Separating droppable data from the rest of the data processing, part 2

Using side inputs

Summary

Section 2 Apache Beam: Toward Improving Usability

Section 2 Apache Beam: Toward Improving Usability

Chapter 4: Structuring Code for Reusability

Chapter 4: Structuring Code for Reusability

Technical requirements

Explaining PTransform expansion

Task 11 – Enhancing SportTracker by runner motivation using side inputs

Introducing composite transform – CoGroupByKey

Task 12 – enhancing SportTracker by runner motivation using CoGroupByKey

Introducing the Join library DSL

Stream-to-stream joins explained

Task 13 – Writing a reusable PTransform – StreamingInnerJoin

Table-stream duality

Summary

Chapter 5: Using SQL for Pipeline Implementation

Chapter 5: Using SQL for Pipeline Implementation

Technical requirements

Understanding schemas

Implementing our first streaming pipeline using SQL

Task 14 – Implementing SQLMaxWordLength

Task 15 – Implementing SchemaSportTracker

Task 16 – Implementing SQLSportTrackerMotivation

Further development of Apache Beam SQL

Summary

Chapter 6: Using Your Preferred Language with Portability

Chapter 6: Using Your Preferred Language with Portability

Technical requirements

Introducing the portability layer

Implementing our first pipelines in the Python SDK

Task 17 – Implementing MaxWordLength in the Python SDK

Python SDK type hints and coders

Task 18 – Implementing SportTracker in the Python SDK

Task 19 – Implementing RPCParDo in the Python SDK

Task 20 – Implementing SportTrackerMotivation in the Python SDK

Using the DataFrame API

Interactive programming using InteractiveRunner

Introducing and using cross-language pipelines

Summary

Section 3 Apache Beam: Advanced Concepts

Section 3 Apache Beam: Advanced Concepts

Chapter 7: Extending Apache Beam's I/O Connectors

Chapter 7: Extending Apache Beam's I/O Connectors

Technical requirements

Defining splittable DoFn as a unification for bounded and unbounded sources

Task 21 – Implementing our own splittable DoFn – a streaming file source

Task 22 – A non-I/O application of splittable DoFn – PiSampler

The legacy Source API and the Read transform

Writing a custom data sink

Summary

Chapter 8: Understanding How Runners Execute Pipelines

Chapter 8: Understanding How Runners Execute Pipelines

Describing the anatomy of an Apache Beam runner

Explaining the differences between classic and portable runners

Understanding how a runner handles state

Exploring the Apache Beam capability matrix

Understanding windowing semantics in depth

Debugging pipelines and using Apache Beam metrics for observability

Summary

Why subscribe?

Other Books You May Enjoy

Other Books You May Enjoy

Packt is searching for authors like you

Share Your Thoughts

Explaining PTransform expansion

A PTransform is a short name for parallel transform – an Apache Beam primitive for transforming PInput into POutput. PInput is a labeling interface that marks objects as suitable as input to PTransform, while POutput marks objects as suitable as outputs. We already know these objects quite well – a typical one that's used for both input and output is PCollection. But there are others as well – most notably PCollectionTuple and PCollectionList. There are also two special objects – PBegin and PDone. As we already know, an Apache Beam program – a pipeline – is a DAG whose edges represent PCollections and whose nodes represent PTransforms. PTransforms in the DAG that take PBegin as input are roots, while PTransforms that produce PDone are the leaves of the DAG.

This can be seen in the following diagram:

Figure 4.1 – DAG of PTransforms and PCollections

A PTransform is a recursive...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Building Big Data Pipelines with Apache Beam

Search

Your notes and bookmarks