Book Image

Storm Real-time Processing Cookbook

By : Quinton Anderson
Book Image

Storm Real-time Processing Cookbook

By: Quinton Anderson

Overview of this book

<p>Storm is a free and open source distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!<br />Storm Real Time Processing Cookbook will have basic to advanced recipes on Storm for real-time computation.<br /><br />The book begins with setting up the development environment and then teaches log stream processing. This will be followed by real-time payments workflow, distributed RPC, integrating it with other software such as Hadoop and Apache Camel, and more.</p>

Related Content you might be interested in

Current Title:

Small book image
Storm Real-time Processing Cookbook
Quinton Anderson
Aug, 2013 | 254 pages
No titles found
Table of Contents (16 chapters)
Storm Real-time Processing Cookbook
Storm Real-time Processing Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.packtpub.com
www.packtpub.com
Preface
Preface
Free Chapter
1
Setting Up Your Development Environment
Setting Up Your Development Environment
Introduction
Setting up your development environment
Distributed version control
Creating a "Hello World" topology
Creating a Storm cluster – provisioning the machines
Creating a Storm cluster – provisioning Storm
Deriving basic click statistics
Unit testing a bolt
Implementing an integration test
Deploying to the cluster
2
Log Stream Processing
Log Stream Processing
Introduction
Creating a log agent
Creating the log spout
Rule-based analysis of the log stream
Indexing and persisting the log data
Counting and persisting log statistics
Creating an integration test for the log stream cluster
Creating a log analytics dashboard
3
Calculating Term Importance with Trident
Calculating Term Importance with Trident
Introduction
Creating a URL stream using a Twitter filter
Deriving a clean stream of terms from the documents
Calculating the relative importance of each term
4
Distributed Remote Procedure Calls
Distributed Remote Procedure Calls
Introduction
Using DRPC to complete the required processing
Integration testing of a Trident topology
Implementing a rolling window topology
Simulating time in integration testing
5
Polyglot Topology
Polyglot Topology
Introduction
Implementing the multilang protocol in Qt
Implementing the SplitSentence bolt in Qt
Implementing the count bolt in Ruby
Defining the word count topology in Clojure
6
Integrating Storm and Hadoop
Integrating Storm and Hadoop
Introduction
Implementing TF-IDF in Hadoop
Persisting documents from Storm
Integrating the batch and real-time views
7
Real-time Machine Learning
Real-time Machine Learning
Introduction
Implementing a transactional topology
Creating a Random Forest classification model using R
Operational classification of transactional streams using Random Forest
Creating an association rules model in R
Creating a recommendation engine
Real-time online machine learning
8
Continuous Delivery
Continuous Delivery
Introduction
Setting up a CI server
Setting up system environments
Defining a delivery pipeline
Implementing automated acceptance testing
9
Storm on AWS
Storm on AWS
Introduction
Deploying Storm on AWS using Pallet
Setting up a Virtual Private Cloud
Deploying Storm into Virtual Private Cloud using Vagrant
Index
Index

About the Reviewers

Maarten Ectors is an executive who is an expert in cloud computing, big data, and disruptive innovations. Maarten's strengths are his combination of deep technical and business skills as well as strategic insights.

Currently, Maarten is responsible for the cloud strategy at Canonical—the company behind Ubuntu—where he is changing the future of cloud, big data, and other disruptive innovations. Previously, Maarten had his own company and was defining and executing the cloud strategy of a global mobile company. Maarten worked for Nokia Siemens Networks in several roles. He was heading cloud and disruptive innovation, founded Startups@NSN, was responsible for implementing offshoring in Europe, and so on. Earlier, he worked as the Director of professional services for Telcordia (now Ericsson) and as a Senior Project / Product Manager for a dotcom. Maarten started his career at Accenture, where he was active in Java developments, portals, mobile applications, content management, ecommerce, security, project management, and so on.

I would like to thank my family for always being there for me. Especially my wonderful wife, Esther, and my great kids.

Alexey Kachayev began his development career in a small team creating an open source CMS for social networks. For over 2 years, he had been working as a Software Engineer at CloudMade, developing geo-relative technology for enterprise clients in Python and Scala. Currently, Alexey is the CTO at Attendify and is focused on development of a distributed applications platform in Erlang. He is an active speaker at conferences and an open source contributor (working on projects in Python, Clojure, and Haskell).

His area of professional interests include distributed systems and algorithms, types theory, and functional language compilers.

I would like to thank Nathan Marz and the Storm project contributors team for developing such a great technology and spreading great ideas.

Paco Nathan is the Chief Scientist at Mesosphere in San Francisco. He is a recognized expert in Hadoop, R, Data Science, and Cloud Computing, and has led innovative data teams building large-scale apps for the past decade. Paco is an evangelist for the Mesos and Cascading open source projects. He is also the author of Enterprise Data Workflows with Cascading, O'Reilly. He has a blog about Data Science at http://liber118.com/pxn/.

Previous Section
Next Chapter