This chapter and the preceding chapter on Catalyst Optimizer have been quite challenging. However, it makes sense to cover so many Apache Spark internals, as in the subsequent chapters we will always refer back to the functionalities provided by Catalyst and Tungsten.
We've learned that many features the JVM provides are far from optimal for massive parallel data processing. This starts with the Garbage Collectors, includes inefficient data structures and ends with the introduction of columnar storage and the removal of the volcano iterator model by fusing individual operators together using whole stage code generation.
In the next chapter we'll have a look at a more practical function of Apache Spark. We'll take a look at how to process data in real-time using Apache Spark Streaming.