Real-time Graph Analysis | Storm Blueprints: Patterns for Distributed Real-time Computation

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Storm Blueprints: Patterns for Distributed Real-time Computation

By : Brian O'Neill

4.1 (8)

Storm Blueprints: Patterns for Distributed Real-time Computation

4.1 (8)

By: Brian O'Neill

Overview of this book

A blueprints book with 10 different projects built in 10 different chapters which demonstrate the various use cases of storm for both beginner and intermediate users, grounded in real-world example applications. Although the book focuses primarily on Java development with Storm, the patterns are more broadly applicable and the tips, techniques, and approaches described in the book apply to architects, developers, and operations. Additionally, the book should provoke and inspire applications of distributed computing to other industries and domains. Hadoop enthusiasts will also find this book a good introduction to Storm, providing a potential migration path from batch processing to the world of real-time analytics.

Storm Blueprints: Patterns for Distributed Real-time Computation

Storm Blueprints: Patterns for Distributed Real-time Computation

Credits

Credits

About the Authors

About the Authors

About the Reviewers

About the Reviewers

www.PacktPub.com

www.PacktPub.com

Preface

Preface

Free Chapter

Distributed Word Count

Distributed Word Count

Introducing elements of a Storm topology – streams, spouts, and bolts

Introducing the word count topology data flow

Implementing the word count topology

Introducing parallelism in Storm

Understanding stream groupings

Guaranteed processing

Summary

Configuring Storm Clusters

Configuring Storm Clusters

Introducing the anatomy of a Storm cluster

Introducing the Storm technology stack

Installing Storm on Linux

Submitting topologies to a Storm cluster

Automating the cluster configuration

A rapid introduction to Puppet

Summary

Trident Topologies and Sensor Data

Trident Topologies and Sensor Data

Examining our use case

Introducing Trident topologies

Introducing Trident spouts

Introducing Trident operations – filters and functions

Introducing Trident aggregators – Combiners and Reducers

Introducing the Trident state

Executing the topology

Summary

Real-time Trend Analysis

Real-time Trend Analysis

Use case

Architecture

Installing the required software

Introducing the sample application

Introducing the log analysis topology

The final topology

Running the log analysis topology

Summary

Real-time Graph Analysis

Real-time Graph Analysis

Use case

Architecture

A brief introduction to graph databases

Software installation

Setting up Titan to use the Cassandra storage backend

Graph data model

Connecting to the Twitter stream

Twitter graph topology

Implementing GraphState

Implementing GraphFactory

Implementing GraphTupleProcessor

Putting it all together – the TwitterGraphTopology class

Querying the graph with Gremlin

Summary

Artificial Intelligence

Artificial Intelligence

Designing for our use case

Establishing the architecture

Implementing the architecture

Summary

Integrating Druid for Financial Analytics

Integrating Druid for Financial Analytics

Use case

Integrating a non-transactional system

The topology

Implementing the architecture

Executing the implementation

Examining the analytics

Summary

Natural Language Processing

Natural Language Processing

Motivating a Lambda architecture

Examining our use case

Realizing a Lambda architecture

Designing the topology for our use case

Implementing the design

Examining the analytics

Batch processing / historical analysis

Hadoop

Summary

Deploying Storm on Hadoop for Advertising Analysis

Deploying Storm on Hadoop for Advertising Analysis

Examining the use case

Establishing the architecture

Configuring the infrastructure

Deploying the analytics

Performing the analytics

Deploying the topology

Executing the topology

Summary

Storm in the Cloud

Storm in the Cloud

Introducing Amazon Elastic Compute Cloud (EC2)

Introducing Apache Whirr

Configuring a Storm cluster with Whirr

Introducing Whirr Storm

Introducing Vagrant

Creating Storm-provisioning scripts

Summary

Index

Index

Implementing GraphTupleProcessor

In order to populate the graph database with relationships gleaned from Twitter status messages, we need to implement the GraphTupleProcessor interface. The following code illustrates parsing the Twitter status message's JSON object and creating "user" and "hashtag" vertices with "mentions" relationships.

TweetGraphTupleProcessor.java
public class TweetGraphTupleProcessor implements GraphTupleProcessor {
    @Override
    public void process(Graph g, TridentTuple tuple, TridentCollector collector) {
        Long timestamp = tuple.getLong(0);
        JSONObject json = (JSONObject)tuple.get(1);

        Vertex user = findOrCreateUser(g, (String)json.get("user"), (String)json.get("name"));

        JSONArray hashtags = (JSONArray)json.get("hashtags");
        for(int i = 0; i < hashtags.size(); i++){
            Vertex v = findOrCreateVertex(g, "hashtag", ((String)hashtags.get(i)).toLowerCase());
            createEdgeAtTime(g, user, v, "mentions", timestamp...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Storm Blueprints: Patterns for Distributed Real-time Computation

Search

Your notes and bookmarks