Book Image

Building Big Data Pipelines with Apache Beam

By : Jan Lukavský
Book Image

Building Big Data Pipelines with Apache Beam

By: Jan Lukavský

Overview of this book

Apache Beam is an open source unified programming model for implementing and executing data processing pipelines, including Extract, Transform, and Load (ETL), batch, and stream processing. This book will help you to confidently build data processing pipelines with Apache Beam. You’ll start with an overview of Apache Beam and understand how to use it to implement basic pipelines. You’ll also learn how to test and run the pipelines efficiently. As you progress, you’ll explore how to structure your code for reusability and also use various Domain Specific Languages (DSLs). Later chapters will show you how to use schemas and query your data using (streaming) SQL. Finally, you’ll understand advanced Apache Beam concepts, such as implementing your own I/O connectors. By the end of this book, you’ll have gained a deep understanding of the Apache Beam model and be able to apply it to solve problems.
Table of Contents (13 chapters)
1
Section 1 Apache Beam: Essentials
5
Section 2 Apache Beam: Toward Improving Usability
9
Section 3 Apache Beam: Advanced Concepts

The legacy Source API and the Read transform

Before the creation of the splittable DoFn object, Beam used the Source API and its associated Read transform. Although this transform is currently deprecated and should not be used for implementing new sources, it is still supported. On some runners and under specific conditions, using the deprecated Read transform might still be preferred. We have already seen examples of this – for example, the use_deprecated_read flag passed when using the --experiments flag for Python's ReadFromKafka transform.

The Read transform accepts a single parameter: either an object of the BoundedSource type or the UnboundedSource type. Whether the source is bounded or unbounded then determines if the resulting PCollection object is bounded or unbounded.

We apply the Read transform as follows:

Pipeline p = ...;
p.apply(Read.from(new MyUnboundedSource());

We will not go into the details of BoundedSource or UnboundedSource, mostly because...