Summary

This chapter focused much more on what can be done on Hadoop 2, and in particular YARN, than the details of Hadoop internals. This is almost certainly a good thing, as it demonstrates that Hadoop is realizing its goal of becoming a much more flexible and generic data processing platform that is no longer tied to batch processing. In particular, we highlighted how Samza shows that the processing frameworks that can be implemented on YARN can innovate and enable functionality vastly different from that available in Hadoop 1.

In particular, we saw how Samza goes to the opposite end of the latency spectrum from batch processing and enables per-message processing of individual messages as they arrive.

We also saw how Samza provides a callback mechanism that MapReduce developers will be familiar with, but uses it for a very different processing model. We also discussed the ways in which Samza utilizes YARN as its main execution framework and how it implements the model described in Chapter...