Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Building Big Data Pipelines with Apache Beam
  • Table Of Contents Toc
Building Big Data Pipelines with Apache Beam

Building Big Data Pipelines with Apache Beam

By : Jan Lukavský
3.7 (9)
Buy this Book
close
close
Building Big Data Pipelines with Apache Beam

Building Big Data Pipelines with Apache Beam

3.7 (9)
By: Jan Lukavský
Buy this Book

Overview of this book

Apache Beam is an open source unified programming model for implementing and executing data processing pipelines, including Extract, Transform, and Load (ETL), batch, and stream processing. This book will help you to confidently build data processing pipelines with Apache Beam. You’ll start with an overview of Apache Beam and understand how to use it to implement basic pipelines. You’ll also learn how to test and run the pipelines efficiently. As you progress, you’ll explore how to structure your code for reusability and also use various Domain Specific Languages (DSLs). Later chapters will show you how to use schemas and query your data using (streaming) SQL. Finally, you’ll understand advanced Apache Beam concepts, such as implementing your own I/O connectors. By the end of this book, you’ll have gained a deep understanding of the Apache Beam model and be able to apply it to solve problems.
Table of Contents (13 chapters)
close
close
close
Preface
Icon
Preface
Icon
Who this book is for
Icon
What this book covers
Icon
To get the most out of this book
Icon
Download the example code files
Icon
Download the color images
Icon
Conventions used
Icon
Get in touch
Icon
Share Your Thoughts
1
Section 1 Apache Beam: Essentials
Icon
Section 1 Apache Beam: Essentials
Lock Free Chapter
2
Chapter 1: Introduction to Data Processing with Apache Beam
Icon
Chapter 1: Introduction to Data Processing with Apache Beam
Icon
Technical requirements
Icon
Why Apache Beam?
Icon
Writing your first pipeline
Icon
Running our pipeline against streaming data
Icon
Exploring the key properties of unbounded data
Icon
Measuring event time progress inside data streams
Icon
Assigning data to windows
Icon
Unifying batch and streaming data processing
Icon
Summary
3
Chapter 2: Implementing, Testing, and Deploying Basic Pipelines
Icon
Chapter 2: Implementing, Testing, and Deploying Basic Pipelines
Icon
Technical requirements
Icon
Setting up the environment for this book
Icon
Task 1 – Calculating the K most frequent words in a stream of lines of text
Icon
Task 2 – Calculating the maximal length of a word in a stream
Icon
Specifying the PCollection Coder object and the TypeDescriptor object
Icon
Understanding default triggers, on time, and closing behavior
Icon
Introducing the primitive PTransform object – Combine
Icon
Task 3 – Calculating the average length of words in a stream
Icon
Task 4 – Calculating the average length of words in a stream with fixed lookback
Icon
Ensuring pipeline upgradability
Icon
Task 5 – Calculating performance statistics for a sport activity tracking application
Icon
Introducing the primitive PTransform object – GroupByKey
Icon
Introducing the primitive PTransform object – Partition
Icon
Summary
4
Chapter 3: Implementing Pipelines Using Stateful Processing
Icon
Chapter 3: Implementing Pipelines Using Stateful Processing
Icon
Technical requirements
Icon
Task 6 – Using an external service for data augmentation
Icon
Introducing the primitive PTransform object – stateless ParDo
Icon
Task 7 – Batching queries to an external RPC service
Icon
Task 8 – Batching queries to an external RPC service with defined batch sizes
Icon
Introducing the primitive PTransform object – stateful ParDo
Icon
Using side outputs
Icon
Defining droppable data in Beam
Icon
Task 9 – Separating droppable data from the rest of the data processing
Icon
Task 10 – Separating droppable data from the rest of the data processing, part 2
Icon
Using side inputs
Icon
Summary
5
Section 2 Apache Beam: Toward Improving Usability
Icon
Section 2 Apache Beam: Toward Improving Usability
6
Chapter 4: Structuring Code for Reusability
Icon
Chapter 4: Structuring Code for Reusability
Icon
Technical requirements
Icon
Explaining PTransform expansion
Icon
Task 11 – Enhancing SportTracker by runner motivation using side inputs
Icon
Introducing composite transform – CoGroupByKey
Icon
Task 12 – enhancing SportTracker by runner motivation using CoGroupByKey
Icon
Introducing the Join library DSL
Icon
Stream-to-stream joins explained
Icon
Task 13 – Writing a reusable PTransform – StreamingInnerJoin
Icon
Table-stream duality
Icon
Summary
7
Chapter 5: Using SQL for Pipeline Implementation
chevron up
Icon
Chapter 5: Using SQL for Pipeline Implementation
Icon
Technical requirements
Icon
Understanding schemas
Icon
Implementing our first streaming pipeline using SQL
Icon
Task 14 – Implementing SQLMaxWordLength
Icon
Task 15 – Implementing SchemaSportTracker
Icon
Task 16 – Implementing SQLSportTrackerMotivation
Icon
Further development of Apache Beam SQL
Icon
Summary
8
Chapter 6: Using Your Preferred Language with Portability
Icon
Chapter 6: Using Your Preferred Language with Portability
Icon
Technical requirements
Icon
Introducing the portability layer
Icon
Implementing our first pipelines in the Python SDK
Icon
Task 17 – Implementing MaxWordLength in the Python SDK
Icon
Python SDK type hints and coders
Icon
Task 18 – Implementing SportTracker in the Python SDK
Icon
Task 19 – Implementing RPCParDo in the Python SDK
Icon
Task 20 – Implementing SportTrackerMotivation in the Python SDK
Icon
Using the DataFrame API
Icon
Interactive programming using InteractiveRunner
Icon
Introducing and using cross-language pipelines
Icon
Summary
9
Section 3 Apache Beam: Advanced Concepts
Icon
Section 3 Apache Beam: Advanced Concepts
10
Chapter 7: Extending Apache Beam's I/O Connectors
Icon
Chapter 7: Extending Apache Beam's I/O Connectors
Icon
Technical requirements
Icon
Defining splittable DoFn as a unification for bounded and unbounded sources
Icon
Task 21 – Implementing our own splittable DoFn – a streaming file source
Icon
Task 22 – A non-I/O application of splittable DoFn – PiSampler
Icon
The legacy Source API and the Read transform
Icon
Writing a custom data sink
Icon
Summary
11
Chapter 8: Understanding How Runners Execute Pipelines
Icon
Chapter 8: Understanding How Runners Execute Pipelines
Icon
Describing the anatomy of an Apache Beam runner
Icon
Explaining the differences between classic and portable runners
Icon
Understanding how a runner handles state
Icon
Exploring the Apache Beam capability matrix
Icon
Understanding windowing semantics in depth
Icon
Debugging pipelines and using Apache Beam metrics for observability
Icon
Summary
Icon
Why subscribe?
12
Other Books You May Enjoy
Icon
Other Books You May Enjoy
Icon
Packt is searching for authors like you
Icon
Share Your Thoughts

Further development of Apache Beam SQL

In this section, we will sum up the possible further development of Apache Beam SQL and what parts are currently expected to be missing or somewhat incomplete.

At the end of the previous chapter, we described the retract and upsert streams and defined time-varying relations on top of these streams. Although Apache Beam does contain generic retractions as part of its model, they are not implemented at the moment. The same is true for SQL. Among other things, it implies that Apache Beam SQL currently does not support full stream-to-stream joins, only windowed joins.

A windowed join, by itself, does not guarantee that retractions will not be needed, but when using a default trigger without allowed lateness – or a trigger that fires past the end of the window, plus allowed lateness only –no retractions are needed. The reason for this is that all the data is projected onto the timestamp at the end of the window, and the window ends...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Building Big Data Pipelines with Apache Beam
End of Section 7
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options
font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon