Book Image

Apache Spark 2.x Cookbook

By : Rishi Yadav
Book Image

Apache Spark 2.x Cookbook

By: Rishi Yadav

Overview of this book

While Apache Spark 1.x gained a lot of traction and adoption in the early years, Spark 2.x delivers notable improvements in the areas of API, schema awareness, Performance, Structured Streaming, and simplifying building blocks to build better, faster, smarter, and more accessible big data applications. This book uncovers all these features in the form of structured recipes to analyze and mature large and complex sets of data. Starting with installing and configuring Apache Spark with various cluster managers, you will learn to set up development environments. Further on, you will be introduced to working with RDDs, DataFrames and Datasets to operate on schema aware data, and real-time streaming with various sources such as Twitter Stream and Apache Kafka. You will also work through recipes on machine learning, including supervised learning, unsupervised learning & recommendation engines in Spark. Last but not least, the final few chapters delve deeper into the concepts of graph processing using GraphX, securing your implementations, cluster optimization, and troubleshooting.

Related Content you might be interested in

Current Title:

Small book image
Apache Spark 2.x Cookbook
Rishi Yadav
May, 2017 | 294 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Small book image
Scala and Spark for Big Data Analytics
Md Rezaul Karim | Sridhar Alla
Jul, 2017 | 796 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Table of Contents (19 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Apache Spark
Getting Started with Apache Spark
Introduction
Leveraging Databricks Cloud
Deploying Spark using Amazon EMR
Installing Spark from binaries
Building the Spark source code with Maven
Launching Spark on Amazon EC2
Deploying Spark on a cluster in standalone mode
Deploying Spark on a cluster with Mesos
Deploying Spark on a cluster with YARN
Understanding SparkContext and SparkSession
Understanding resilient distributed dataset - RDD
2
Developing Applications with Spark
Developing Applications with Spark
Introduction
Exploring the Spark shell
Developing a Spark applications in Eclipse with Maven
Developing a Spark applications in Eclipse with SBT
Developing a Spark application in IntelliJ IDEA with Maven
Developing a Spark application in IntelliJ IDEA with SBT
Developing applications using the Zeppelin notebook
Setting up Kerberos to do authentication
Enabling Kerberos authentication for Spark
3
Spark SQL
Spark SQL
Understanding the evolution of schema awareness
Understanding the Catalyst optimizer
Inferring schema using case classes
Programmatically specifying the schema
Understanding the Parquet format
Loading and saving data using the JSON format
Loading and saving data from relational databases
Loading and saving data from an arbitrary source
Understanding joins
Analyzing nested structures
4
Working with External Data Sources
Working with External Data Sources
Introduction
Loading data from the local filesystem
Loading data from HDFS
Loading data from Amazon S3
Loading data from Apache Cassandra
5
Spark Streaming
Spark Streaming
Introduction
WordCount using Structured Streaming
Taking a closer look at Structured Streaming
Streaming Twitter data
Streaming using Kafka
Understanding streaming challenges
6
Getting Started with Machine Learning
Getting Started with Machine Learning
Introduction
Creating vectors
Calculating correlation
Understanding feature engineering
Understanding Spark ML
Understanding hyperparameter tuning
7
Supervised Learning with MLlib — Regression
Supervised Learning with MLlib — Regression
Introduction
Using linear regression
Understanding the cost function
Doing linear regression with lasso
Doing ridge regression
8
Supervised Learning with MLlib — Classification
Supervised Learning with MLlib — Classification
Introduction
Doing classification using logistic regression
Doing binary classification using SVM
Doing classification using decision trees
Doing classification using random forest
Doing classification using gradient boosted trees
Doing classification with Naïve Bayes
9
Unsupervised Learning
Unsupervised Learning
Introduction
Clustering using k-means
Dimensionality reduction with principal component analysis
Dimensionality reduction with singular value decomposition
10
Recommendations Using Collaborative Filtering
Recommendations Using Collaborative Filtering
Introduction
Collaborative filtering using explicit feedback
Collaborative filtering using implicit feedback
11
Graph Processing Using GraphX and GraphFrames
Graph Processing Using GraphX and GraphFrames
Introduction
Fundamental operations on graphs
Using PageRank
Finding connected components
Performing neighborhood aggregation
Understanding GraphFrames
12
Optimizations and Performance Tuning
Optimizations and Performance Tuning
Optimizing memory
Leveraging speculation
Optimizing joins
Using compression to improve performance
Using serialization to improve performance
Optimizing the level of parallelism
Understanding project Tungsten

Deploying Spark on a cluster with Mesos

Mesos is slowly emerging as a data center operating system for managing all the compute resources across a data center. Mesos runs on any computer running the Linux operating system. It is built using the same principles as the Linux kernel. Let's see how we can install Mesos.

How to do it…

Mesosphere provides a binary distribution of Mesos. The most recent package of the Mesos distribution can be installed from the Mesosphere repositories by performing the following steps:

  1. Execute Mesos on a Ubuntu OS with the trusty version:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF 
          DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release 
            -cs)
$ sudo vi /etc/apt/sources.list.d/mesosphere.list
deb http://repos.mesosphere.io/Ubuntu trusty main
  1. Update the repositories:
$ sudo apt-get -y update
  1. Install Mesos:
$ sudo apt-get -y install mesos
  1. To connect Spark to Mesos and to integrate Spark with Mesos, make Spark binaries available to Mesos and configure the Spark driver to connect to Mesos.
  2. Use the Spark binaries from the first recipe and upload them to HDFS:
$ hdfs dfs -put spark-2.1.0-bin-hadoop2.7.tgz spark-2.1.0-bin-hadoop2.7.tgz
  1. The master URL of a single master Mesos is mesos://host:5050; the master URL of a ZooKeeper-managed Mesos cluster is mesos://zk://host:2181.
  2. Set the following variables in spark-env.sh:
$ sudo vi spark-env.sh
export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI= hdfs://localhost:9000/user/hduser/spark-2.1.0-bin-
          hadoop2.7.tgz
  1. Run the following commands from the Scala program:
Val conf = new SparkConf().setMaster("mesos://host:5050")
Val sparkContext = new SparkContext(conf)
  1. Run the following command from the Spark shell:
$ spark-shell --master mesos://host:5050

Note

Mesos has two run modes:

  • Fine-grained: In the fine-grained (default) mode, every Spark task runs as a separate Mesos task. 
  • Coarse-grained: This mode will launch only one long-running Spark task on each Mesos machine
  1. To run in the coarse-grained mode, set the spark.mesos.coarse property:
Conf.set("spark.mesos.coarse","true")
Previous Section
End of Section 9
Next Section