Book Image

Machine Learning with Spark - Second Edition

By : Rajdeep Dua, Manpreet Singh Ghotra
Book Image

Machine Learning with Spark - Second Edition

By: Rajdeep Dua, Manpreet Singh Ghotra

Overview of this book

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you'll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML. Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML. By the end of this book, you will acquire the skills to leverage Spark's features to create your own scalable machine learning applications and power a modern data-driven business.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning with Spark - Second Edition
Rajdeep Dua | Manpreet Singh Ghotra
Apr, 2017 | 532 pages
Small book image
Kali Linux Intrusion and Exploitation Cookbook
Dhruv Shah | Ishan Girdhar
Apr, 2017 | 512 pages
Small book image
Apache Spark 2.x Machine Learning Cookbook
Shuen Mei | Siamak Amirghodsi | Meenakshi Rajendran | Broderick Hall
Sep, 2017 | 666 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Table of Contents (13 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Getting Up and Running with Spark
Getting Up and Running with Spark
Installing and setting up Spark locally
Spark clusters
The Spark programming model
SchemaRDD
Spark data frame
The first step to a Spark program in Scala
The first step to a Spark program in Java
The first step to a Spark program in Python
The first step to a Spark program in R
Getting Spark running on Amazon EC2
Configuring and running Spark on Amazon Elastic Map Reduce
UI in Spark
Supported machine learning algorithms by Spark
Benefits of using Spark ML as compared to existing libraries
Spark Cluster on Google Compute Engine - DataProc
Summary
2
Math for Machine Learning
Math for Machine Learning
Linear algebra
Gradient descent
Prior, likelihood, and posterior
Calculus
Plotting
Summary
3
Designing a Machine Learning System
Designing a Machine Learning System
What is Machine Learning?
Introducing MovieStream
Business use cases for a machine learning system
Types of machine learning models
The components of a data-driven machine learning system
An architecture for a machine learning system
Spark MLlib
Performance improvements in Spark ML over Spark MLlib
Comparing algorithms supported by MLlib
MLlib supported methods and developer APIs
MLlib vision
MLlib versions compared
Summary
4
Obtaining, Processing, and Preparing Data with Spark
Obtaining, Processing, and Preparing Data with Spark
Accessing publicly available datasets
Exploring and visualizing your data
Processing and transforming your data
Extracting useful features from your data
Summary
5
Building a Recommendation Engine with Spark
Building a Recommendation Engine with Spark
Types of recommendation models
Extracting the right features from your data
Training the recommendation model
Using the recommendation model
Evaluating the performance of recommendation models
FP-Growth algorithm
Summary
6
Building a Classification Model with Spark
Building a Classification Model with Spark
Types of classification models
Extracting the right features from your data
Training classification models
Using classification models
Improving model performance and tuning parameters
Additional features
Summary
7
Building a Regression Model with Spark
Building a Regression Model with Spark
Types of regression models
Evaluating the performance of regression models
Extracting the right features from your data
Training and using regression models
Improving model performance and tuning parameters
Summary
8
Building a Clustering Model with Spark
Building a Clustering Model with Spark
Types of clustering models
Extracting the right features from your data
K-means - training a clustering model
K-means - evaluating the performance of clustering models
Effect of iterations on WSSSE
Bisecting KMeans
Bisecting K-means - training a clustering model
Gaussian Mixture Model
Summary
9
Dimensionality Reduction with Spark
Dimensionality Reduction with Spark
Types of dimensionality reduction
Extracting the right features from your data
Training a dimensionality reduction model
Using a dimensionality reduction model
Evaluating dimensionality reduction models
Summary
10
Advanced Text Processing with Spark
Advanced Text Processing with Spark
What's so special about text data?
Extracting the right features from your data
Using a tf-idf model
Evaluating the impact of text processing
Text classification with Spark 2.0
Word2Vec models
Word2Vec with Spark ML on the 20 Newsgroups dataset
Summary
11
Real-Time Machine Learning with Spark Streaming
Real-Time Machine Learning with Spark Streaming
Online learning
Stream processing
Online learning with Spark Streaming
Online model evaluation
Structured Streaming
Summary
12
Pipeline APIs for Spark ML
Pipeline APIs for Spark ML
Introduction to pipelines
How pipelines work
Machine learning pipeline with an example
Summary

Bisecting KMeans

It is a variation of generic KMeans.

Reference: http://www.siam.org/meetings/sdm01/pdf/sdm01_05.pdf

The steps of the algorithm are:

  1. Initialize by randomly selecting a point, say then compute the centroid w of M and compute:


The centroid is the center of the cluster. A centroid is a vector containing one number for each variable, where each number is the mean of a variable for the observations in that cluster.
  1. Divide M =[x1, x2, ... xn] into two, sub-clusters ML and MR, according to the following rule:

  1. Compute the centroids of ML and MR, wL and wR, as in step 2.
  1. If wL = cL and wR = cR, stop.
Otherwise, let cL= wL cR = wR , go to step 2.
Previous Section
End of Section 7
Next Section