Chapter 1, Introduction to Apex, tells us how processing of data-in-motion is realized by Apache Apex. It also gives us a few Apex stream processing use cases and applications, and talks about their value propositions.

Chapter 2, Getting Started with Application Development, shows us how the Apex development process works from project creation to application deployment; the result is a simple working application.

Chapter 3, The Apex Library, talks about the Malhar library, which contains functional building blocks for writing real-world Apex applications.

Chapter 4, Scalability, Low Latency, and Performance, teaches us how Apex can scale and parallize processing, how to achieve dynamic scaling and better resource allocation in general, and why low latency and high throughput are both achievable without trading one off against the other. Operator partitioning and related techniques are central to this endeavor and are shown in practice in a sample application.

Chapter 5, Fault Tolerance and Reliability, explores the implementation of fault-tolerance and reliability in Apex including exactly-once semantics via distributed checkpointing and effective state management.

Chapter 6, Example Project – Real-Time Aggregation and Visualization, puts together all the building blocks to show a streaming analytics project and how to integrate it with a UI and existing infrastructure.

Chapter 7, Example Project – Real-Time Ride Service Data Processing, relies on a historical dataset to simulate a real-time ride service data stream. We are using event time and out-of-order processing, in particular, to build a simple analytics application that can serve as a template for more complicated event stream data pipelines.

Chapter 8, Example Project – ETL Using SQL, shows how to build a classic ETL application using Apex and Apache Calcite.

Chapter 9, Introduction to Apache Beam, introduces the Beam stream processing framework and an approach that allows a stream application engine such as Apex to be swapped in if needed.

Chapter 10, The Future of Stream Processing, looks at the road ahead for Apex and stream processing in general. We are going to examine the role of machine learning, as well as the role of SQL and why it is important for streaming.

Apex applications can be built and run locally on the user’s development machine via a properly written JUnit test. To do this, the user need only ensure that recent versions of the following software packages are present:

To run Apex applications on a cluster, one needs a cluster with Hadoop installed and a client to launch them. This client needs to be installed on the edge node (sometimes referred to as the gateway node or the client node); there is no need to install anything on the entire cluster.

There are several options to install the client, and some of them are listed on the Apex download page: http://apex.apache.org/downloads.html.

Without an existing Hadoop cluster, an easy way to get started for experimentation is a sandbox VM that already has a single node cluster configured (sandbox VMs are available from Hadoop vendors, as docker images and so on).

This book is a practical and example-oriented guide on how to use Apache Apex to successfully implement solutions for real-world big data problems. The book assumes knowledge of application development with Java and familiarity with distributed systems. It does not require prior knowledge of Apex or stream processing, although knowledge of other big data processing frameworks is helpful. 

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text (such as database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles) are shown as follows where the code word is "10011": "This result tells the driver to drive toward the zip code 10011."

A block of code is set as follows:

PubSubWebSocketAppDataResult wsResult = dag.addOperator("QueryResult", new PubSubWebSocketAppDataResult());
  wsResult.setTopic("nyctaxi.result");
  try {
    wsResult.setUri(new URI("ws://localhost:8890/pubsub"));
  } catch (URISyntaxException ex) {
    throw Throwables.propagate(ex);
  }
dag.addStream("server_to_query_output", dataServer.queryResult, wsResult.input);

Any command-line input or output is written as follows:

bash> apex 
apex> launch target/malhar-examples-nyc-taxi-3.8.0-SNAPSHOT.apa

New terms and important words are shown in bold.

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an email to [email protected], and mention the book title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.