Book Image

Machine Learning with Spark - Second Edition

By : Rajdeep Dua, Manpreet Singh Ghotra
Book Image

Machine Learning with Spark - Second Edition

By: Rajdeep Dua, Manpreet Singh Ghotra

Overview of this book

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you'll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML. Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML. By the end of this book, you will acquire the skills to leverage Spark's features to create your own scalable machine learning applications and power a modern data-driven business.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning with Spark - Second Edition
Rajdeep Dua | Manpreet Singh Ghotra
Apr, 2017 | 532 pages
Small book image
Kali Linux Intrusion and Exploitation Cookbook
Dhruv Shah | Ishan Girdhar
Apr, 2017 | 512 pages
Small book image
Apache Spark 2.x Machine Learning Cookbook
Shuen Mei | Siamak Amirghodsi | Meenakshi Rajendran | Broderick Hall
Sep, 2017 | 666 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Table of Contents (13 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Getting Up and Running with Spark
Getting Up and Running with Spark
Installing and setting up Spark locally
Spark clusters
The Spark programming model
SchemaRDD
Spark data frame
The first step to a Spark program in Scala
The first step to a Spark program in Java
The first step to a Spark program in Python
The first step to a Spark program in R
Getting Spark running on Amazon EC2
Configuring and running Spark on Amazon Elastic Map Reduce
UI in Spark
Supported machine learning algorithms by Spark
Benefits of using Spark ML as compared to existing libraries
Spark Cluster on Google Compute Engine - DataProc
Summary
2
Math for Machine Learning
Math for Machine Learning
Linear algebra
Gradient descent
Prior, likelihood, and posterior
Calculus
Plotting
Summary
3
Designing a Machine Learning System
Designing a Machine Learning System
What is Machine Learning?
Introducing MovieStream
Business use cases for a machine learning system
Types of machine learning models
The components of a data-driven machine learning system
An architecture for a machine learning system
Spark MLlib
Performance improvements in Spark ML over Spark MLlib
Comparing algorithms supported by MLlib
MLlib supported methods and developer APIs
MLlib vision
MLlib versions compared
Summary
4
Obtaining, Processing, and Preparing Data with Spark
Obtaining, Processing, and Preparing Data with Spark
Accessing publicly available datasets
Exploring and visualizing your data
Processing and transforming your data
Extracting useful features from your data
Summary
5
Building a Recommendation Engine with Spark
Building a Recommendation Engine with Spark
Types of recommendation models
Extracting the right features from your data
Training the recommendation model
Using the recommendation model
Evaluating the performance of recommendation models
FP-Growth algorithm
Summary
6
Building a Classification Model with Spark
Building a Classification Model with Spark
Types of classification models
Extracting the right features from your data
Training classification models
Using classification models
Improving model performance and tuning parameters
Additional features
Summary
7
Building a Regression Model with Spark
Building a Regression Model with Spark
Types of regression models
Evaluating the performance of regression models
Extracting the right features from your data
Training and using regression models
Improving model performance and tuning parameters
Summary
8
Building a Clustering Model with Spark
Building a Clustering Model with Spark
Types of clustering models
Extracting the right features from your data
K-means - training a clustering model
K-means - evaluating the performance of clustering models
Effect of iterations on WSSSE
Bisecting KMeans
Bisecting K-means - training a clustering model
Gaussian Mixture Model
Summary
9
Dimensionality Reduction with Spark
Dimensionality Reduction with Spark
Types of dimensionality reduction
Extracting the right features from your data
Training a dimensionality reduction model
Using a dimensionality reduction model
Evaluating dimensionality reduction models
Summary
10
Advanced Text Processing with Spark
Advanced Text Processing with Spark
What's so special about text data?
Extracting the right features from your data
Using a tf-idf model
Evaluating the impact of text processing
Text classification with Spark 2.0
Word2Vec models
Word2Vec with Spark ML on the 20 Newsgroups dataset
Summary
11
Real-Time Machine Learning with Spark Streaming
Real-Time Machine Learning with Spark Streaming
Online learning
Stream processing
Online learning with Spark Streaming
Online model evaluation
Structured Streaming
Summary
12
Pipeline APIs for Spark ML
Pipeline APIs for Spark ML
Introduction to pipelines
How pipelines work
Machine learning pipeline with an example
Summary

Benefits of using Spark ML as compared to existing libraries

AMQ Lab at Berkley Evaluated Spark, and RDDs were evaluated through a series of experiments on Amazon EC2 as well as benchmarks of user applications.

  • Algorithms used: Logistical Regression and k-means
  • Use case: First iteration, multiple iterations.

All the tests used m1.xlarge EC2 nodes with 4 cores and 15 GB of RAM. HDFS was for storage with 256 MB blocks. Refer to the following graph:

The preceding graph shows the comparison between the performance of Hadoop and Spark for the first and subsequent iteration for Logistical Regression:

The preceding graph shows the comparison between the performance of Hadoop and Spark for the first and subsequent iteration for K Means clustering algorithm.

The overall results show the following:

  • Spark outperforms Hadoop by up to 20 times in iterative machine learning and graph applications. The speedup comes from avoiding I/O and deserialization costs by storing data in memory as Java objects.
  • The applications written perform and scale well. Spark can speed up an analytics report that was running on Hadoop by 40 times.
  • When nodes fail, Spark can recover quickly by rebuilding only the lost RDD partitions.
  • Spark was be used to query a 1-TB dataset interactively with latencies of 5-7 seconds.
For more information, go to http://people.csail.mit.edu/matei/papers/2012/nsdi_spark.pdf.

Spark versus Hadoop for a SORT Benchmark--In 2014, the Databricks team participated in a SORT benchmark test (http://sortbenchmark.org/). This was done on a 100-TB dataset. Hadoop was running in a dedicated data center and a Spark cluster of over 200 nodes was run on EC2. Spark was run on HDFS distributed storage.

Spark was 3 times faster than Hadoop and used 10 times fewer machines. Refer to the following graph:

Previous Section
End of Section 15
Next Section