Book Image

Mastering Apache Storm

By : Ankit Jain
Book Image

Mastering Apache Storm

By: Ankit Jain

Overview of this book

Apache Storm is a real-time Big Data processing framework that processes large amounts of data reliably, guaranteeing that every message will be processed. Storm allows you to scale your data as it grows, making it an excellent platform to solve your big data problems. This extensive guide will help you understand right from the basics to the advanced topics of Storm. The book begins with a detailed introduction to real-time processing and where Storm fits in to solve these problems. You’ll get an understanding of deploying Storm on clusters by writing a basic Storm Hello World example. Next we’ll introduce you to Trident and you’ll get a clear understanding of how you can develop and deploy a trident topology. We cover topics such as monitoring, Storm Parallelism, scheduler and log processing, in a very easy to understand manner. You will also learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, Kafka, and Hadoop to realize the full potential of Storm. With real-world examples and clear explanations, this book will ensure you will have a thorough mastery of Apache Storm. You will be able to use this knowledge to develop efficient, distributed real-time applications to cater to your business needs.

Related Content you might be interested in

Current Title:

Small book image
Mastering Apache Storm
Ankit Jain
Aug, 2017 | 284 pages
Small book image
Practical Real-time Data Processing and Analytics
Saurabh Gupta | Shilpi Saxena
Sep, 2017 | 360 pages
Small book image
Building Data Streaming Applications with Apache Kafka
Manish Kumar | Chanchal Singh
Aug, 2017 | 278 pages
Small book image
Apache Kafka 1.0 Cookbook
Raúl Estrada
Dec, 2017 | 250 pages
Table of Contents (19 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Real-Time Processing and Storm Introduction
Real-Time Processing and Storm Introduction
Apache Storm
Features of Storm
Storm components
The Storm data model
Programming languages
Summary
2
Storm Deployment, Topology Development, and Topology Options
Storm Deployment, Topology Development, and Topology Options
Storm prerequisites
Setting up the Storm cluster
Developing the hello world example
The different options of the Storm topology
Walkthrough of the Storm UI
Dynamic log level settings
Summary
3
Storm Parallelism and Data Partitioning
Storm Parallelism and Data Partitioning
Parallelism of a topology
Rebalance the parallelism of a topology
Different types of stream grouping in the Storm cluster
Guaranteed message processing
Tick tuple
Summary
4
Trident Introduction
Trident Introduction
Trident introduction
Understanding Trident's data model
Writing Trident functions, filters, and projections
Trident repartitioning operations
Trident aggregator
Utilizing the groupBy operation
When to use Trident
Summary
5
Trident Topology and Uses
Trident Topology and Uses
Trident groupBy operation
Non-transactional topology
Trident hello world topology
Trident state
Distributed RPC
When to use Trident
Summary
6
Storm Scheduler
Storm Scheduler
Introduction to Storm scheduler
Default scheduler
Isolation scheduler
Resource-aware scheduler
Custom scheduler
Summary
7
Monitoring of Storm Cluster
Monitoring of Storm Cluster
Cluster statistics using the Nimbus thrift client
Monitoring the Storm cluster using JMX
Monitoring the Storm cluster using Ganglia
Summary
8
Integration of Storm and Kafka
Integration of Storm and Kafka
Introduction to Kafka
Kafka architecture
Installation of Kafka brokers
Share ZooKeeper between Storm and Kafka
Kafka producers and publishing data into Kafka
Kafka Storm integration
Deploy the Kafka topology on Storm cluster
Summary
9
Storm and Hadoop Integration
Storm and Hadoop Integration
Introduction to Hadoop
Installation of Hadoop
Write Storm topology to persist data into HDFS
Integration of Storm with Hadoop
Setting up Storm-YARN
Storm-Starter topologies on Storm-YARN
Summary
10
Storm Integration with Redis, Elasticsearch, and HBase
Storm Integration with Redis, Elasticsearch, and HBase
Integrating Storm with HBase
Integrating Storm with Redis
Integrating Storm with Elasticsearch
Integrating Storm with Esper
Summary
11
Apache Log Processing with Storm
Apache Log Processing with Storm
Apache log processing elements
Producing Apache log in Kafka using Logstash
Splitting the Apache log line
Identifying country, operating system type, and browser type from the log file
Calculate the search keyword
Persisting the process data
Kafka spout and define topology
Deploy topology
MySQL queries
Summary
12
Twitter Tweet Collection and Machine Learning
Twitter Tweet Collection and Machine Learning
Exploring machine learning
Twitter sentiment analysis
Kafka spout, sentiments bolt, and HDFS bolt
Summary

Kafka spout and define topology

This section will explain how we can read the Apache log from a Kafka topic. This section also defines the LogProcessingTopology that will chain together all the bolts created in the preceding sections. Let's perform the following steps to consume the data from Kafka and define the topology:

  1. Add the following dependency and repository for Kafka in the pom.xml file:
       <dependency> 
             <groupId>org.apache.storm</groupId> 
             <artifactId>storm-kafka</artifactId> 
             <version>1.0.2</version> 
             <exclusions> 
                   <exclusion> 
                         <groupId>org.apache.kafka</groupId> 
                         <artifactId>kafka-clients</artifactId> 
                   </exclusion> 
             </exclusions> 
       </dependency> 
 
       <dependency> 
             <groupId>org.apache.kafka<...
Previous Section
End of Section 8
Next Section