Book Image

Storm Real-time Processing Cookbook

By : Quinton Anderson
Book Image

Storm Real-time Processing Cookbook

By: Quinton Anderson

Overview of this book

<p>Storm is a free and open source distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!<br />Storm Real Time Processing Cookbook will have basic to advanced recipes on Storm for real-time computation.<br /><br />The book begins with setting up the development environment and then teaches log stream processing. This will be followed by real-time payments workflow, distributed RPC, integrating it with other software such as Hadoop and Apache Camel, and more.</p>
Table of Contents (16 chapters)
Storm Real-time Processing Cookbook
Credits
About the Author
About the Reviewers
www.packtpub.com
Preface
Index

Defining a delivery pipeline


The delivery pipeline not only ensures what is released is stable and controlled, but also enables this to be done rapidly. A pipeline can potentially enable every source commit to be a release candidate. The pipeline needs to be built out in distinct steps, each removing different kinds of risks from the delivery process. The following is an example of a build pipeline:

You will need to define a pipeline that is appropriate to your technology and organizational process.

The first step is the traditional CI step. This ensures that the code base is built and is unit tested. It is important to keep the build and unit test process to a very short time, thus allowing developers to receive rapid feedback on their changes. It is also necessary to perform a more complete test of the code base in a deployed or semi-deployed form. This can be divided into integration tests and acceptance tests; whatever approach you take for this, the acceptance tests will generally be...