Book Image

Storm Real-time Processing Cookbook

By : Quinton Anderson
Book Image

Storm Real-time Processing Cookbook

By: Quinton Anderson

Overview of this book

<p>Storm is a free and open source distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!<br />Storm Real Time Processing Cookbook will have basic to advanced recipes on Storm for real-time computation.<br /><br />The book begins with setting up the development environment and then teaches log stream processing. This will be followed by real-time payments workflow, distributed RPC, integrating it with other software such as Hadoop and Apache Camel, and more.</p>
Table of Contents (16 chapters)
Storm Real-time Processing Cookbook
Credits
About the Author
About the Reviewers
www.packtpub.com
Preface
Index

Creating a log agent


Modern enterprise architectures consist of a huge number of solutions, each comprising many nodes. Some MapReduce clusters contain hundreds of nodes. Each node contains an array of applications and services, both at the operating system and Application layers. These services and applications generate varying volumes of log data. There is an increasing recognition of the importance of log data within the enterprise community for the following reasons:

  • It is a key source of information for any IT operations team to maintain available systems

  • It is key to finding and resolving problems, both in the production and systems testing phases

  • It is increasingly becoming a source of business value, where valuable business data is trapped in this semi-structured data, including:

    • Risk and compliance data

    • Business operational data

    • Web analytics

    • Security

    • Financial forecasts

In order to leverage valuable log data, it must be sourced from these nodes and delivered securely and easily to a centralized...