Book Image

Learning Storm

By : Ankit Jain, Anand Nalya
Book Image

Learning Storm

By: Ankit Jain, Anand Nalya

Overview of this book

<p>Starting with the very basics of Storm, you will learn how to set up Storm on a single machine and move on to deploying Storm on your cluster. You will understand how Kafka can be integrated with Storm using the Kafka spout.</p> <p>You will then proceed to explore the Trident abstraction tool with Storm to perform stateful stream processing, guaranteeing single message processing in every topology. You will move ahead to learn how to integrate Hadoop with Storm. Next, you will learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, and Kafka to realize the full potential of Storm.</p> <p>Finally, you will perform in-depth case studies on Apache log processing and machine learning with a focus on Storm, and through these case studies, you will discover Storm's realm of possibilities.</p>
Table of Contents (16 chapters)
Learning Storm
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Preface
Index

A transactional topology


As mentioned in the definition of the non-transactional topology, Trident processes tuples in a batch, but this doesn't define what's in each batch. In the case of a transactional topology, a transactional spout guarantees what's in each batch. A transactional spout has the following characteristics:

  • Each batch is assigned a unique transactional ID (txid). In the case of failure, the entire batch is replayed. Hence, replays of the failed batch will contain the same set of tuples as the first time the batch was emitted. The txid transactional ID of the failed batch remains the same as the first time.

  • Tuples of one batch are not mixed with tuples of another batch. Hence, overlaps of tuples between batches are not allowed.

Let's consider the previous sample Trident topology example and see how we can write a transactional topology. Suppose the sample Trident topology computes the country field's count and stores the counts in a key/value store (Memory Map, Cassandra, Memcached...