Book Image

Mastering Apache Spark 2.x - Second Edition

Book Image

Mastering Apache Spark 2.x - Second Edition

Overview of this book

Apache Spark is an in-memory, cluster-based Big Data processing system that provides a wide range of functionalities such as graph processing, machine learning, stream processing, and more. This book will take your knowledge of Apache Spark to the next level by teaching you how to expand Spark’s functionality and build your data flows and machine/deep learning programs on top of the platform. The book starts with a quick overview of the Apache Spark ecosystem, and introduces you to the new features and capabilities in Apache Spark 2.x. You will then work with the different modules in Apache Spark such as interactive querying with Spark SQL, using DataFrames and DataSets effectively, streaming analytics with Spark Streaming, and performing machine learning and deep learning on Spark using MLlib and external tools such as H20 and Deeplearning4j. The book also contains chapters on efficient graph processing, memory management and using Apache Spark on the cloud. By the end of this book, you will have all the necessary information to master Apache Spark, and use it efficiently for Big Data processing and analytics.

Related Content you might be interested in

Current Title:

Small book image
Mastering Apache Spark 2.x - Second Edition
Jul, 2017 | 354 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Small book image
Apache Spark 2.x Cookbook
Rishi Yadav
May, 2017 | 294 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Table of Contents (21 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
A First Taste and What’s New in Apache Spark V2
A First Taste and What’s New in Apache Spark V2
Spark machine learning
Spark Streaming
Spark SQL
Spark graph processing
Extended ecosystem
What's new in Apache Spark V2?
Cluster design
Cluster management
Cloud-based deployments
Performance
Cloud
Summary
2
Apache Spark SQL
Apache Spark SQL
The SparkSession--your gateway to structured data processing
Importing and saving data
Understanding the DataSource API
DataFrames
Using SQL
Using Datasets
User-defined functions
RDDs versus DataFrames versus Datasets
Summary
3
The Catalyst Optimizer
The Catalyst Optimizer
Understanding the workings of the Catalyst Optimizer
Managing temporary views with the catalog API
The SQL abstract syntax tree
How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan
Code generation
Summary
4
Project Tungsten
Project Tungsten
Memory management beyond the Java Virtual Machine Garbage Collector
Cache-friendly layout of data in memory
Code generation
Summary
5
Apache Spark Streaming
Apache Spark Streaming
Overview
Errors and recovery
Streaming sources
Summary
6
Structured Streaming
Structured Streaming
The concept of continuous applications
Windowing
Increased performance with good old friends
How transparent fault tolerance and exactly-once delivery guarantee is achieved
Example - connection to a MQTT message broker
Summary
7
Apache Spark MLlib
Apache Spark MLlib
Architecture
Classification with Naive Bayes
Clustering with K-Means
Artificial neural networks
Summary
8
Apache SparkML
Apache SparkML
What does the new API look like?
The concept of pipelines
Model evaluation
CrossValidation and hyperparameter tuning
Winning a Kaggle competition with Apache SparkML
Summary
9
Apache SystemML
Apache SystemML
Why do we need just another library?
A cost-based optimizer for machine learning algorithms
Performance measurements
Apache SystemML in action
Summary
10
Deep Learning on Apache Spark with DeepLearning4j and H2O
Deep Learning on Apache Spark with DeepLearning4j and H2O
H2O
Deeplearning4j
Summary
11
Apache Spark GraphX
Apache Spark GraphX
Overview
Graph analytics/processing with GraphX
Summary
12
Apache Spark GraphFrames
Apache Spark GraphFrames
Architecture
Examples
Summary
13
Apache Spark with Jupyter Notebooks on IBM DataScience Experience
Apache Spark with Jupyter Notebooks on IBM DataScience Experience
Why notebooks are the new standard
Learning by example
Summary
14
Apache Spark on Kubernetes
Apache Spark on Kubernetes
Bare metal, virtual machines, and containers
Understanding the core concepts of Docker
Understanding Kubernetes
Using Kubernetes for provisioning containerized Spark applications
Example--Apache Spark on Kubernetes
Summary

Model evaluation

As mentioned before, model evaluation is built-in to ApacheSparkML and you'll find all that you need in the org.apache.spark.ml.evaluation package. Let's continue with our binary classification. This means that we'll have to use org.apache.spark.ml.evaluation.BinaryClassificationEvaluator:

import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
val evaluator = new BinaryClassificationEvaluator()

import org.apache.spark.ml.param.ParamMap
var evaluatorParamMap = ParamMap(evaluator.metricName -> "areaUnderROC")
var aucTraining = evaluator.evaluate(result, evaluatorParamMap)

To code previous initialized a BinaryClassificationEvaluator function and tells it to calculate the areaUnderROC, one of the many possible metrics to assess the prediction performance of a machine learning algorithm.

As we have the actual label and the prediction present in a DataFrame called result, it is simple to calculate this score and is done using the following line of code:

var aucTraining...
Previous Section
End of Section 4
Next Section