Book Image

Machine Learning with Spark - Second Edition

By : Rajdeep Dua, Manpreet Singh Ghotra
Book Image

Machine Learning with Spark - Second Edition

By: Rajdeep Dua, Manpreet Singh Ghotra

Overview of this book

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you'll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML. Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML. By the end of this book, you will acquire the skills to leverage Spark's features to create your own scalable machine learning applications and power a modern data-driven business.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning with Spark - Second Edition
Rajdeep Dua | Manpreet Singh Ghotra
Apr, 2017 | 532 pages
Small book image
Kali Linux Intrusion and Exploitation Cookbook
Dhruv Shah | Ishan Girdhar
Apr, 2017 | 512 pages
Small book image
Apache Spark 2.x Machine Learning Cookbook
Shuen Mei | Siamak Amirghodsi | Meenakshi Rajendran | Broderick Hall
Sep, 2017 | 666 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Table of Contents (13 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Getting Up and Running with Spark
Getting Up and Running with Spark
Installing and setting up Spark locally
Spark clusters
The Spark programming model
SchemaRDD
Spark data frame
The first step to a Spark program in Scala
The first step to a Spark program in Java
The first step to a Spark program in Python
The first step to a Spark program in R
Getting Spark running on Amazon EC2
Configuring and running Spark on Amazon Elastic Map Reduce
UI in Spark
Supported machine learning algorithms by Spark
Benefits of using Spark ML as compared to existing libraries
Spark Cluster on Google Compute Engine - DataProc
Summary
2
Math for Machine Learning
Math for Machine Learning
Linear algebra
Gradient descent
Prior, likelihood, and posterior
Calculus
Plotting
Summary
3
Designing a Machine Learning System
Designing a Machine Learning System
What is Machine Learning?
Introducing MovieStream
Business use cases for a machine learning system
Types of machine learning models
The components of a data-driven machine learning system
An architecture for a machine learning system
Spark MLlib
Performance improvements in Spark ML over Spark MLlib
Comparing algorithms supported by MLlib
MLlib supported methods and developer APIs
MLlib vision
MLlib versions compared
Summary
4
Obtaining, Processing, and Preparing Data with Spark
Obtaining, Processing, and Preparing Data with Spark
Accessing publicly available datasets
Exploring and visualizing your data
Processing and transforming your data
Extracting useful features from your data
Summary
5
Building a Recommendation Engine with Spark
Building a Recommendation Engine with Spark
Types of recommendation models
Extracting the right features from your data
Training the recommendation model
Using the recommendation model
Evaluating the performance of recommendation models
FP-Growth algorithm
Summary
6
Building a Classification Model with Spark
Building a Classification Model with Spark
Types of classification models
Extracting the right features from your data
Training classification models
Using classification models
Improving model performance and tuning parameters
Additional features
Summary
7
Building a Regression Model with Spark
Building a Regression Model with Spark
Types of regression models
Evaluating the performance of regression models
Extracting the right features from your data
Training and using regression models
Improving model performance and tuning parameters
Summary
8
Building a Clustering Model with Spark
Building a Clustering Model with Spark
Types of clustering models
Extracting the right features from your data
K-means - training a clustering model
K-means - evaluating the performance of clustering models
Effect of iterations on WSSSE
Bisecting KMeans
Bisecting K-means - training a clustering model
Gaussian Mixture Model
Summary
9
Dimensionality Reduction with Spark
Dimensionality Reduction with Spark
Types of dimensionality reduction
Extracting the right features from your data
Training a dimensionality reduction model
Using a dimensionality reduction model
Evaluating dimensionality reduction models
Summary
10
Advanced Text Processing with Spark
Advanced Text Processing with Spark
What's so special about text data?
Extracting the right features from your data
Using a tf-idf model
Evaluating the impact of text processing
Text classification with Spark 2.0
Word2Vec models
Word2Vec with Spark ML on the 20 Newsgroups dataset
Summary
11
Real-Time Machine Learning with Spark Streaming
Real-Time Machine Learning with Spark Streaming
Online learning
Stream processing
Online learning with Spark Streaming
Online model evaluation
Structured Streaming
Summary
12
Pipeline APIs for Spark ML
Pipeline APIs for Spark ML
Introduction to pipelines
How pipelines work
Machine learning pipeline with an example
Summary

Spark clusters

A Spark cluster is made up of two types of processes: a driver program and multiple executors. In the local mode, all these processes are run within the same JVM. In a cluster, these processes are usually run on separate nodes.

For example, a typical cluster that runs in Spark's standalone mode (that is, using Spark's built-in cluster management modules) will have the following:

  • A master node that runs the Spark standalone master process as well as the driver program
  • A number of worker nodes, each running an executor process

While we will be using Spark's local standalone mode throughout this book to illustrate concepts and examples, the same Spark code that we write can be run on a Spark cluster. In the preceding example, if we run the code on a Spark standalone cluster, we could simply pass in the URL for the master node, as follows:

  $ MASTER=spark://IP:PORT --class org.apache.spark.examples.SparkPi 
    ./examples/jars/spark-examples_2.11-2.0.0.jar 100

Here, IP is the IP address and PORT is the port of the Spark master. This tells Spark to run the program on the cluster where the Spark master process is running.

A full treatment of Spark's cluster management and deployment is beyond the scope of this book. However, we will briefly teach you how to set up and use an Amazon EC2 cluster later in this chapter.

For an overview of the Spark cluster-application deployment, take a look at the following links:

Previous Section
End of Section 3
Next Section