Book Image

Mastering Machine Learning with Spark 2.x

By : Michal Malohlava, Alex Tellez, Max Pumperla

Book Image

Mastering Machine Learning with Spark 2.x

By: Michal Malohlava, Alex Tellez, Max Pumperla

Overview of this book

The purpose of machine learning is to build systems that learn from data. Being able to understand trends and patterns in complex data is critical to success; it is one of the key strategies to unlock growth in the challenging contemporary marketplace today. With the meteoric rise of machine learning, developers are now keen on finding out how can they make their Spark applications smarter. This book gives you access to transform data into actionable knowledge. The book commences by defining machine learning primitives by the MLlib and H2O libraries. You will learn how to use Binary classification to detect the Higgs Boson particle in the huge amount of data produced by CERN particle collider and classify daily health activities using ensemble Methods for Multi-Class Classification. Next, you will solve a typical regression problem involving flight delay predictions and write sophisticated Spark pipelines. You will analyze Twitter data with help of the doc2vec algorithm and K-means clustering. Finally, you will build different pattern mining models using MLlib, perform complex manipulation of DataFrames using Spark and Spark SQL, and deploy your app in a Spark streaming environment.

Preface

What this book covers

What you need for this book

Who this book is for

Reader feedback

Customer support

Free Chapter

Introduction to Large-Scale Machine Learning and Spark

Introduction to Large-Scale Machine Learning and Spark

The sexiest role of the 21st century – data scientist?

Introducing H2O.ai

What's the difference between H2O and Spark's MLlib?

Data science - an iterative process

Detecting Dark Matter - The Higgs-Boson Particle

Detecting Dark Matter - The Higgs-Boson Particle

Type I versus type II error

Spark start and data load

Ensemble Methods for Multi-Class Classification

Ensemble Methods for Multi-Class Classification

Predicting Movie Reviews Using NLP and Spark Streaming

Predicting Movie Reviews Using NLP and Spark Streaming

NLP - a brief primer

Feature extraction

Featurization - feature hashing

Let's do some (model) training!

Word2vec for Prediction and Clustering

Word2vec for Prediction and Clustering

Motivation of word vectors

Word2vec explained

Doc2vec explained

Applying word2vec and exploring our data with vectors

Creating document vectors

Supervised learning task

Extracting Patterns from Clickstream Data

Extracting Patterns from Clickstream Data

Frequent pattern mining

Pattern mining with Spark MLlib

Deploying a pattern mining application

Graph Analytics with GraphX

Graph Analytics with GraphX

Basic graph theory

GraphX distributed graph processing engine

Graph algorithms and applications

Lending Club Loan Prediction

Lending Club Loan Prediction

Preparation of the environment

Exploration – data analysis

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "We also appended the magic column row_id, which uniquely identifies each row in the dataset." A block of code is set as follows:

import org.apache.spark.ml.feature.StopWordsRemover 
val stopWords= StopWordsRemover.loadDefaultStopWords("english") ++ Array("ax", "arent", "re")

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

val MIN_TOKEN_LENGTH = 3
val toTokens= (minTokenLen: Int, stopWords: Array[String],

Any command-line input or output is written as follows:

tar -xvf spark-2.1.1-bin-hadoop2.6.tgz 
export SPARK_HOME="$(pwd)/spark-2.1.1-bin-hadoop2.6

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Download the DECLINED LOAN DATA as shown in the following screenshot"

Warnings or important notes appear like this.

Tips and tricks appear like this.