Book Image

Real-Time Big Data Analytics

By : Sumit Gupta, Shilpi Saxena
Book Image

Real-Time Big Data Analytics

By: Sumit Gupta, Shilpi Saxena

Overview of this book

Enterprise has been striving hard to deal with the challenges of data arriving in real time or near real time. Although there are technologies such as Storm and Spark (and many more) that solve the challenges of real-time data, using the appropriate technology/framework for the right business use case is the key to success. This book provides you with the skills required to quickly design, implement and deploy your real-time analytics using real-world examples of big data use cases. From the beginning of the book, we will cover the basics of varied real-time data processing frameworks and technologies. We will discuss and explain the differences between batch and real-time processing in detail, and will also explore the techniques and programming concepts using Apache Storm. Moving on, we’ll familiarize you with “Amazon Kinesis” for real-time data processing on cloud. We will further develop your understanding of real-time analytics through a comprehensive review of Apache Spark along with the high-level architecture and the building blocks of a Spark program. You will learn how to transform your data, get an output from transformations, and persist your results using Spark RDDs, using an interface called Spark SQL to work with Spark. At the end of this book, we will introduce Spark Streaming, the streaming library of Spark, and will walk you through the emerging Lambda Architecture (LA), which provides a hybrid platform for big data processing by combining real-time and precomputed batch data to provide a near real-time view of incoming data.
Table of Contents (17 chapters)
Real-Time Big Data Analytics
About the Authors
About the Reviewer

Other sources for input to Storm

In our earlier example, we have seen integrated with Storm, one of them being Kafka (just discussed in the previous section). In Storm samples, the word count topology (which we covered in detail in an earlier chapter) doesn't use any data source for input. Instead, some sentences are hardcoded in the spout itself and this seed data is emitted to the topology. This may be fine for test cases and samples, but for real-world implementations this is neither ideal nor expected. Storm has to feed a stream of live events into the topology in almost all of the real-world implementations. We can have a variety of input sources that can be integrated with Storm. Let's have a closer look at code snippets on what all we can plug in with Storm to feed the data.

A file as an input source

We can use a Storm spout to be effectively reading from a file; though that's not a real use case for a streaming app, but we can have Storm very well read in from the file. All we need...