Summary

This chapter covers all the basics of Apache Spark, which all machine learning professionals are expected to understand in order to utilize Apache Spark for practical machine learning projects. We focus our discussion on Apache Spark computing, and relate it to some of the most important machine learning components, in order to connect Apache Spark and machine learning together to fully prepare our readers for machine learning projects.

First, we provided a Spark overview, and also discussed Spark's advantages as well as Spark's computing model for machine learning.

Second, we reviewed machine learning algorithms, Spark's MLlib libraries, and other machine learning libraries.

In the third section, Spark's core innovations of RDD and DataFrame has been discussed, as well as Spark's DataFrame API for R.

Fourth, we reviewed some ML frameworks, and specifically discussed a RM4Es framework for machine learning as an example, and then further discussed Spark computing frameworks for machine learning.

Fifth, we discussed machine learning as workflows, went through one workflow example, and then reviewed Spark's pipelines and its API.

Finally, we studied the notebook approach for machine learning, and reviewed R's famous notebook Markdown, then we discussed a Spark Notebook provided by Databricks, so we can use Spark Notebook to unite all the above Spark elements for machine learning practice easily.

With all the above Spark basics covered, the readers should be ready to start utilizing Apache Spark for some machine learning projects from here on. Therefore, we will work on data preparation on Spark in the next chapter, then jump into our first real life machine learning projects in Chapter 3, A Holistic View on Spark.