Stream processing is the act of continuously computing results as new data becomes available. A very simple example of this is computing the average of some numbers in a continuous fashion. To begin with, we start with the following information:
- Number of items = 0
- Current average = 0
As a new number comes in, we perform the following steps:
- Compute a new total = Number of items x Current average + New number
- Increment the number of items by one
- Set the current average = New total / Number of items
As you can see, the continuous average computation algorithm is quite different from the batch-oriented algorithm. It is important to bear in mind the following facts when using this algorithm:
- The average value gets updated as new numbers become available
- The previously computed average value is reused to compute a new average
The following recipe using...