Book Image

Apache Spark 2.x Cookbook

By : Rishi Yadav
Book Image

Apache Spark 2.x Cookbook

By: Rishi Yadav

Overview of this book

While Apache Spark 1.x gained a lot of traction and adoption in the early years, Spark 2.x delivers notable improvements in the areas of API, schema awareness, Performance, Structured Streaming, and simplifying building blocks to build better, faster, smarter, and more accessible big data applications. This book uncovers all these features in the form of structured recipes to analyze and mature large and complex sets of data. Starting with installing and configuring Apache Spark with various cluster managers, you will learn to set up development environments. Further on, you will be introduced to working with RDDs, DataFrames and Datasets to operate on schema aware data, and real-time streaming with various sources such as Twitter Stream and Apache Kafka. You will also work through recipes on machine learning, including supervised learning, unsupervised learning & recommendation engines in Spark. Last but not least, the final few chapters delve deeper into the concepts of graph processing using GraphX, securing your implementations, cluster optimization, and troubleshooting.

Related Content you might be interested in

Current Title:

Small book image
Apache Spark 2.x Cookbook
Rishi Yadav
May, 2017 | 294 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Small book image
Scala and Spark for Big Data Analytics
Md Rezaul Karim | Sridhar Alla
Jul, 2017 | 796 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Table of Contents (19 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Apache Spark
Getting Started with Apache Spark
Introduction
Leveraging Databricks Cloud
Deploying Spark using Amazon EMR
Installing Spark from binaries
Building the Spark source code with Maven
Launching Spark on Amazon EC2
Deploying Spark on a cluster in standalone mode
Deploying Spark on a cluster with Mesos
Deploying Spark on a cluster with YARN
Understanding SparkContext and SparkSession
Understanding resilient distributed dataset - RDD
2
Developing Applications with Spark
Developing Applications with Spark
Introduction
Exploring the Spark shell
Developing a Spark applications in Eclipse with Maven
Developing a Spark applications in Eclipse with SBT
Developing a Spark application in IntelliJ IDEA with Maven
Developing a Spark application in IntelliJ IDEA with SBT
Developing applications using the Zeppelin notebook
Setting up Kerberos to do authentication
Enabling Kerberos authentication for Spark
3
Spark SQL
Spark SQL
Understanding the evolution of schema awareness
Understanding the Catalyst optimizer
Inferring schema using case classes
Programmatically specifying the schema
Understanding the Parquet format
Loading and saving data using the JSON format
Loading and saving data from relational databases
Loading and saving data from an arbitrary source
Understanding joins
Analyzing nested structures
4
Working with External Data Sources
Working with External Data Sources
Introduction
Loading data from the local filesystem
Loading data from HDFS
Loading data from Amazon S3
Loading data from Apache Cassandra
5
Spark Streaming
Spark Streaming
Introduction
WordCount using Structured Streaming
Taking a closer look at Structured Streaming
Streaming Twitter data
Streaming using Kafka
Understanding streaming challenges
6
Getting Started with Machine Learning
Getting Started with Machine Learning
Introduction
Creating vectors
Calculating correlation
Understanding feature engineering
Understanding Spark ML
Understanding hyperparameter tuning
7
Supervised Learning with MLlib — Regression
Supervised Learning with MLlib — Regression
Introduction
Using linear regression
Understanding the cost function
Doing linear regression with lasso
Doing ridge regression
8
Supervised Learning with MLlib — Classification
Supervised Learning with MLlib — Classification
Introduction
Doing classification using logistic regression
Doing binary classification using SVM
Doing classification using decision trees
Doing classification using random forest
Doing classification using gradient boosted trees
Doing classification with Naïve Bayes
9
Unsupervised Learning
Unsupervised Learning
Introduction
Clustering using k-means
Dimensionality reduction with principal component analysis
Dimensionality reduction with singular value decomposition
10
Recommendations Using Collaborative Filtering
Recommendations Using Collaborative Filtering
Introduction
Collaborative filtering using explicit feedback
Collaborative filtering using implicit feedback
11
Graph Processing Using GraphX and GraphFrames
Graph Processing Using GraphX and GraphFrames
Introduction
Fundamental operations on graphs
Using PageRank
Finding connected components
Performing neighborhood aggregation
Understanding GraphFrames
12
Optimizations and Performance Tuning
Optimizations and Performance Tuning
Optimizing memory
Leveraging speculation
Optimizing joins
Using compression to improve performance
Using serialization to improve performance
Optimizing the level of parallelism
Understanding project Tungsten

Optimizing the level of parallelism

Optimizing the level of parallelism is very important to fully utilize the cluster capacity. In the case of HDFS, it means that the number of partitions is the same as the number of input splits, which is mostly the same as the number of blocks. The default block size in HDFS is 128 MB, and that works well in case of Spark as well. 

In this recipe, we will cover different ways to optimize the number of partitions.

How to do it...

Specify the number of partitions when loading a file into RDD with the following steps:

  1. Start the Spark shell:
$ spark-shell
  1. Load the RDD with a custom number of partitions as a second parameter:
scala> sc.textFile("hdfs://localhost:9000/user/hduser/words",10)

Another approach is to change the default parallelism by performing the following steps:

  1. Start the Spark shell with the new value of default parallelism:
$ spark-shell --conf spark.default.parallelism=10

Note

Have the number of partitions two to three times the number of cores to...

Previous Section
End of Section 7
Next Section