Book Image

Machine Learning with Spark - Second Edition

By : Rajdeep Dua, Manpreet Singh Ghotra

Book Image

Machine Learning with Spark - Second Edition

By: Rajdeep Dua, Manpreet Singh Ghotra

Overview of this book

This book will teach you about popular machine learning algorithms and their implementation. You will learn how various machine learning concepts are implemented in the context of Spark ML. You will start by installing Spark in a single and multinode cluster. Next you'll see how to execute Scala and Python based programs for Spark ML. Then we will take a few datasets and go deeper into clustering, classification, and regression. Toward the end, we will also cover text processing using Spark ML. Once you have learned the concepts, they can be applied to implement algorithms in either green-field implementations or to migrate existing systems to this new platform. You can migrate from Mahout or Scikit to use Spark ML. By the end of this book, you will acquire the skills to leverage Spark's features to create your own scalable machine learning applications and power a modern data-driven business.

Preface

What this book covers

What you need for this book

Who this book is for

Reader feedback

Customer support

Free Chapter

Getting Up and Running with Spark

Getting Up and Running with Spark

Installing and setting up Spark locally

The Spark programming model

Spark data frame

The first step to a Spark program in Scala

The first step to a Spark program in Java

The first step to a Spark program in Python

The first step to a Spark program in R

Getting Spark running on Amazon EC2

Configuring and running Spark on Amazon Elastic Map Reduce

Supported machine learning algorithms by Spark

Benefits of using Spark ML as compared to existing libraries

Spark Cluster on Google Compute Engine - DataProc

Math for Machine Learning

Math for Machine Learning

Gradient descent

Prior, likelihood, and posterior

Designing a Machine Learning System

Designing a Machine Learning System

What is Machine Learning?

Introducing MovieStream

Business use cases for a machine learning system

Types of machine learning models

The components of a data-driven machine learning system

An architecture for a machine learning system

Performance improvements in Spark ML over Spark MLlib

Comparing algorithms supported by MLlib

MLlib supported methods and developer APIs

MLlib versions compared

Obtaining, Processing, and Preparing Data with Spark

Obtaining, Processing, and Preparing Data with Spark

Accessing publicly available datasets

Exploring and visualizing your data

Processing and transforming your data

Extracting useful features from your data

Building a Recommendation Engine with Spark

Building a Recommendation Engine with Spark

Types of recommendation models

Extracting the right features from your data

Training the recommendation model

Using the recommendation model

Evaluating the performance of recommendation models

FP-Growth algorithm

Building a Classification Model with Spark

Building a Classification Model with Spark

Types of classification models

Extracting the right features from your data

Training classification models

Using classification models

Improving model performance and tuning parameters

Additional features

Building a Regression Model with Spark

Building a Regression Model with Spark

Types of regression models

Evaluating the performance of regression models

Extracting the right features from your data

Training and using regression models

Improving model performance and tuning parameters

Building a Clustering Model with Spark

Building a Clustering Model with Spark

Types of clustering models

Extracting the right features from your data

K-means - training a clustering model

K-means - evaluating the performance of clustering models

Effect of iterations on WSSSE

Bisecting KMeans

Bisecting K-means - training a clustering model

Gaussian Mixture Model

Dimensionality Reduction with Spark

Dimensionality Reduction with Spark

Types of dimensionality reduction

Extracting the right features from your data

Training a dimensionality reduction model

Using a dimensionality reduction model

Evaluating dimensionality reduction models

Advanced Text Processing with Spark

Advanced Text Processing with Spark

What's so special about text data?

Extracting the right features from your data

Using a tf-idf model

Evaluating the impact of text processing

Text classification with Spark 2.0

Word2Vec models

Word2Vec with Spark ML on the 20 Newsgroups dataset

Real-Time Machine Learning with Spark Streaming

Real-Time Machine Learning with Spark Streaming

Online learning

Stream processing

Online learning with Spark Streaming

Online model evaluation

Structured Streaming

Pipeline APIs for Spark ML

Pipeline APIs for Spark ML

Introduction to pipelines

How pipelines work

Machine learning pipeline with an example

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Extracting the right features from your data

In this section, we will use explicit rating data, without additional user, item metadata, or other information related to the user-item interactions. Hence, the features that we need as inputs are simply the user IDs, movie IDs, and the ratings assigned to each user and movie pair.

Extracting features from the MovieLens 100k dataset

In this example, we will use the same MovieLens dataset that we used in the previous chapter. Use the directory in which you placed the MovieLens 100k dataset as the input path in the following code.

First, let's inspect the raw ratings dataset:

object FeatureExtraction { 

def getFeatures(): Dataset[FeatureExtraction.Rating] = { 
  val spark = SparkSession.builder.master("local[2]...