Book Image

Machine Learning with Spark - Second Edition

By : Rajdeep Dua, Manpreet Singh Ghotra

Book Image

Machine Learning with Spark - Second Edition

By: Rajdeep Dua, Manpreet Singh Ghotra

Overview of this book

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you'll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML. Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML. By the end of this book, you will acquire the skills to leverage Spark's features to create your own scalable machine learning applications and power a modern data-driven business.

Preface

What this book covers

What you need for this book

Who this book is for

Reader feedback

Customer support

Free Chapter

Getting Up and Running with Spark

Getting Up and Running with Spark

Installing and setting up Spark locally

The Spark programming model

Spark data frame

The first step to a Spark program in Scala

The first step to a Spark program in Java

The first step to a Spark program in Python

The first step to a Spark program in R

Getting Spark running on Amazon EC2

Configuring and running Spark on Amazon Elastic Map Reduce

Supported machine learning algorithms by Spark

Benefits of using Spark ML as compared to existing libraries

Spark Cluster on Google Compute Engine - DataProc

Math for Machine Learning

Math for Machine Learning

Gradient descent

Prior, likelihood, and posterior

Designing a Machine Learning System

Designing a Machine Learning System

What is Machine Learning?

Introducing MovieStream

Business use cases for a machine learning system

Types of machine learning models

The components of a data-driven machine learning system

An architecture for a machine learning system

Performance improvements in Spark ML over Spark MLlib

Comparing algorithms supported by MLlib

MLlib supported methods and developer APIs

MLlib versions compared

Obtaining, Processing, and Preparing Data with Spark

Obtaining, Processing, and Preparing Data with Spark

Accessing publicly available datasets

Exploring and visualizing your data

Processing and transforming your data

Extracting useful features from your data

Building a Recommendation Engine with Spark

Building a Recommendation Engine with Spark

Types of recommendation models

Extracting the right features from your data

Training the recommendation model

Using the recommendation model

Evaluating the performance of recommendation models

FP-Growth algorithm

Building a Classification Model with Spark

Building a Classification Model with Spark

Types of classification models

Extracting the right features from your data

Training classification models

Using classification models

Improving model performance and tuning parameters

Additional features

Building a Regression Model with Spark

Building a Regression Model with Spark

Types of regression models

Evaluating the performance of regression models

Extracting the right features from your data

Training and using regression models

Improving model performance and tuning parameters

Building a Clustering Model with Spark

Building a Clustering Model with Spark

Types of clustering models

Extracting the right features from your data

K-means - training a clustering model

K-means - evaluating the performance of clustering models

Effect of iterations on WSSSE

Bisecting KMeans

Bisecting K-means - training a clustering model

Gaussian Mixture Model

Dimensionality Reduction with Spark

Dimensionality Reduction with Spark

Types of dimensionality reduction

Extracting the right features from your data

Training a dimensionality reduction model

Using a dimensionality reduction model

Evaluating dimensionality reduction models

Advanced Text Processing with Spark

Advanced Text Processing with Spark

What's so special about text data?

Extracting the right features from your data

Using a tf-idf model

Evaluating the impact of text processing

Text classification with Spark 2.0

Word2Vec models

Word2Vec with Spark ML on the 20 Newsgroups dataset

Real-Time Machine Learning with Spark Streaming

Real-Time Machine Learning with Spark Streaming

Online learning

Stream processing

Online learning with Spark Streaming

Online model evaluation

Structured Streaming

Pipeline APIs for Spark ML

Pipeline APIs for Spark ML

Introduction to pipelines

How pipelines work

Machine learning pipeline with an example

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Gradient descent

An SGD implementation of gradient descent uses a simple distributed sampling of the data examples. Loss is a part of the optimization problem, and therefore, is a true sub-gradient.

This requires access to the full dataset, which is not optimal.

The parameter miniBatchFraction specifies the fraction of the full data to use. The average of the gradients over this subset

is a stochastic gradient. S is a sampled subset of size |S|= miniBatchFraction.

In the following code, we show how to use stochastic gardient descent on a mini batch to calculate the weights and the loss. The output of this program is a vector of weights and loss.

object SparkSGD { 
 def main(args: Array[String]): Unit = { 
    val m = 4 
    val n = 200000 
    val sc = new SparkContext("local[2]", "") 
    val points = sc.parallelize(0 until m, 
      2).mapPartitionsWithIndex { (idx, iter) => 
      val random...