Book Image

Practical Real-time Data Processing and Analytics

Book Image

Practical Real-time Data Processing and Analytics

Overview of this book

With the rise of Big Data, there is an increasing need to process large amounts of data continuously, with a shorter turnaround time. Real-time data processing involves continuous input, processing and output of data, with the condition that the time required for processing is as short as possible. This book covers the majority of the existing and evolving open source technology stack for real-time processing and analytics. You will get to know about all the real-time solution aspects, from the source to the presentation to persistence. Through this practical book, you’ll be equipped with a clear understanding of how to solve challenges on your own. We’ll cover topics such as how to set up components, basic executions, integrations, advanced use cases, alerts, and monitoring. You’ll be exposed to the popular tools used in real-time processing today such as Apache Spark, Apache Flink, and Storm. Finally, you will put your knowledge to practical use by implementing all of the techniques in the form of a practical, real-world use case. By the end of this book, you will have a solid understanding of all the aspects of real-time data processing and analytics, and will know how to deploy the solutions in production environments in the best possible manner.

Related Content you might be interested in

Current Title:

Small book image
Practical Real-time Data Processing and Analytics
Sep, 2017 | 360 pages
Small book image
Mastering Apache Storm
Ankit Jain
Aug, 2017 | 284 pages
Small book image
Building Data Streaming Applications with Apache Kafka
Manish Kumar | Chanchal Singh
Aug, 2017 | 278 pages
Small book image
Apache Kafka 1.0 Cookbook
Raúl Estrada
Dec, 2017 | 250 pages
Table of Contents (20 chapters)
Title Page
Title Page
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Introducing Real-Time Analytics
Introducing Real-Time Analytics
What is big data?
Big data infrastructure
Real–time analytics – the myth and the reality
Near real–time solution – an architecture that works
Lambda architecture – analytics possibilities
IOT – thoughts and possibilities
Cloud – considerations for NRT and IOT
Summary
2
Real Time Applications – The Basic Ingredients
Real Time Applications – The Basic Ingredients
The NRT system and its building blocks
NRT – high-level system view
NRT – technology view
Summary
3
Understanding and Tailing Data Streams
Understanding and Tailing Data Streams
Understanding data streams
Setting up infrastructure for data ingestion
Taping data from source to the processor - expectations and caveats
Comparing and choosing what works best for your use case
Do it yourself
Summary
4
Setting up the Infrastructure for Storm
Setting up the Infrastructure for Storm
Overview of Storm
Storm architecture and its components
Setting up and configuring Storm
Real-time processing job on Storm
Summary
5
Configuring Apache Spark and Flink
Configuring Apache Spark and Flink
Setting up and a quick execution of Spark
Setting up and a quick execution of Flink
Setting up and a quick execution of Apache Beam
Balancing in Apache Beam
Summary
6
Integrating Storm with a Data Source
Integrating Storm with a Data Source
RabbitMQ – messaging that works
RabbitMQ exchanges
RabbitMQ – integration with Storm
PubNub data stream publisher
String together Storm-RMQ-PubNub sensor data topology
Summary
7
From Storm to Sink
From Storm to Sink
Setting up and configuring Cassandra
Storm and Cassandra topology
Storm and IMDB integration for dimensional data
Integrating the presentation layer with Storm
Do It Yourself
Summary
8
Storm Trident
Storm Trident
State retention and the need for Trident
Basic Storm Trident topology
Trident internals
Trident operations
DRPC
Do It Yourself
Summary
9
Working with Spark
Working with Spark
Spark overview
Distinct advantages of Spark
Spark – use cases
Spark architecture - working inside the engine
Spark pragmatic concepts
Spark 2.x – advent of data frames and datasets
Summary
10
Working with Spark Operations
Working with Spark Operations
Spark – packaging and API
RDD pragmatic exploration
Shared variables – broadcast variables and accumulators
Summary
11
Spark Streaming
Spark Streaming
Spark Streaming concepts
Spark Streaming - introduction and architecture
Packaging structure of Spark Streaming
Connecting Kafka to Spark Streaming
Summary
12
Working with Apache Flink
Working with Apache Flink
Flink architecture and execution engine
Flink basic components and processes
Integration of source stream to Flink
Flink processing and computation
Flink persistence
FlinkCEP
Pattern API
Gelly
DIY
Summary
13
Case Study
Case Study
Introduction
Data modeling
Tools and frameworks
Setting up the infrastructure
Implementing the case study
Running the case study
Summary

DRPC

Distributed Remote Produce Call (DRPC). It is computing very intense functionality on the fly using storm. It gives input as function name and corresponding arguments and the output is results for each of those function calls.

Code example DRPC client:

DRPCClient client = new DRPCClient("drpc.server.location", 3772);
System.out.println(client.execute("words", "cat dog the man");

DRPC Storm code:

topology.newDRPCStream("words")
.each(new Fields("args"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.stateQuery(wordCounts, new Fields("word"), new MapGet(), new Fields("count"))
.aggregate(new Fields("count"), new Sum(), new Fields("sum"));

The program is explained in the following diagram:

Previous Section
End of Section 6
Next Section