Book Image

Storm Real-time Processing Cookbook

By : Quinton Anderson
Book Image

Storm Real-time Processing Cookbook

By: Quinton Anderson

Overview of this book

<p>Storm is a free and open source distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!<br />Storm Real Time Processing Cookbook will have basic to advanced recipes on Storm for real-time computation.<br /><br />The book begins with setting up the development environment and then teaches log stream processing. This will be followed by real-time payments workflow, distributed RPC, integrating it with other software such as Hadoop and Apache Camel, and more.</p>

Related Content you might be interested in

Current Title:

Small book image
Storm Real-time Processing Cookbook
Quinton Anderson
Aug, 2013 | 254 pages
No titles found
Table of Contents (16 chapters)
Storm Real-time Processing Cookbook
Storm Real-time Processing Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.packtpub.com
www.packtpub.com
Preface
Preface
Free Chapter
1
Setting Up Your Development Environment
Setting Up Your Development Environment
Introduction
Setting up your development environment
Distributed version control
Creating a "Hello World" topology
Creating a Storm cluster – provisioning the machines
Creating a Storm cluster – provisioning Storm
Deriving basic click statistics
Unit testing a bolt
Implementing an integration test
Deploying to the cluster
2
Log Stream Processing
Log Stream Processing
Introduction
Creating a log agent
Creating the log spout
Rule-based analysis of the log stream
Indexing and persisting the log data
Counting and persisting log statistics
Creating an integration test for the log stream cluster
Creating a log analytics dashboard
3
Calculating Term Importance with Trident
Calculating Term Importance with Trident
Introduction
Creating a URL stream using a Twitter filter
Deriving a clean stream of terms from the documents
Calculating the relative importance of each term
4
Distributed Remote Procedure Calls
Distributed Remote Procedure Calls
Introduction
Using DRPC to complete the required processing
Integration testing of a Trident topology
Implementing a rolling window topology
Simulating time in integration testing
5
Polyglot Topology
Polyglot Topology
Introduction
Implementing the multilang protocol in Qt
Implementing the SplitSentence bolt in Qt
Implementing the count bolt in Ruby
Defining the word count topology in Clojure
6
Integrating Storm and Hadoop
Integrating Storm and Hadoop
Introduction
Implementing TF-IDF in Hadoop
Persisting documents from Storm
Integrating the batch and real-time views
7
Real-time Machine Learning
Real-time Machine Learning
Introduction
Implementing a transactional topology
Creating a Random Forest classification model using R
Operational classification of transactional streams using Random Forest
Creating an association rules model in R
Creating a recommendation engine
Real-time online machine learning
8
Continuous Delivery
Continuous Delivery
Introduction
Setting up a CI server
Setting up system environments
Defining a delivery pipeline
Implementing automated acceptance testing
9
Storm on AWS
Storm on AWS
Introduction
Deploying Storm on AWS using Pallet
Setting up a Virtual Private Cloud
Deploying Storm into Virtual Private Cloud using Vagrant
Index
Index

Counting and persisting log statistics

There are many statistics that can be gathered for log streams; for the purposes of this recipe and to illustrate the concept, only a single-time series will be dealt with (log volume per minute); however, this should fully illustrate the design and approach for implementing other analyses.

How to do it…

  1. Download and install the storm-cassandra contrib project into your Maven repository:

    git clone https://github.com/quintona/storm-cassandra
cd storm-cassandra
mvn clean install

  2. Create a new BaseRichBolt class called VolumeCountingBolt in the storm.cookbook.log package. The bolt must declare three output fields:

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields(FIELD_ROW_KEY, FIELD_COLUMN, FIELD_INCREMENT));
   }

  3. Then implement a static utility method to derive the minute representation of the log's time:

    public static Long getMinuteForTime(Date time) {
      Calendar c = Calendar.getInstance();
      c.setTime...
Previous Section
End of Section 7
Next Section