In this chapter, we discuss the challenges associated with writing efficient and scalable analytics running on Spark. We will start by introducing the reader to the general concepts of distributed parallelization and scalability and how they relate to Spark. We will recap over Spark's distributed architecture giving the reader an understanding of its underlying principles and how this supports the parallel processing paradigm. We will learn about the characteristics of scalable analytics and the elements of Spark that underpin these (for example, RDD, combineByKey, and GraphX).
Next, we will learn about why sometimes even basic algorithms, despite working at small scale, will often fail in big data. We'll see how to avoid issues when writing Spark jobs that run over massive datasets, including an example using mean/variance. The reader will learn about the structure of algorithms and how to write custom data science analytics that scale over petabytes of data...