Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Apache Spark 2.x Cookbook

By : Yadav

3.3 (3)

Apache Spark 2.x Cookbook

3.3 (3)

By: Yadav

Overview of this book

While Apache Spark 1.x gained a lot of traction and adoption in the early years, Spark 2.x delivers notable improvements in the areas of API, schema awareness, Performance, Structured Streaming, and simplifying building blocks to build better, faster, smarter, and more accessible big data applications. This book uncovers all these features in the form of structured recipes to analyze and mature large and complex sets of data. Starting with installing and configuring Apache Spark with various cluster managers, you will learn to set up development environments. Further on, you will be introduced to working with RDDs, DataFrames and Datasets to operate on schema aware data, and real-time streaming with various sources such as Twitter Stream and Apache Kafka. You will also work through recipes on machine learning, including supervised learning, unsupervised learning & recommendation engines in Spark. Last but not least, the final few chapters delve deeper into the concepts of graph processing using GraphX, securing your implementations, cluster optimization, and troubleshooting.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Conventions

Reader feedback

Customer support

Free Chapter

Getting Started with Apache Spark

Getting Started with Apache Spark

Introduction

Leveraging Databricks Cloud

Deploying Spark using Amazon EMR

Installing Spark from binaries

Building the Spark source code with Maven

Launching Spark on Amazon EC2

Deploying Spark on a cluster in standalone mode

Deploying Spark on a cluster with Mesos

Deploying Spark on a cluster with YARN

Understanding SparkContext and SparkSession

Understanding resilient distributed dataset - RDD

Developing Applications with Spark

Developing Applications with Spark

Introduction

Exploring the Spark shell

Developing a Spark applications in Eclipse with Maven

Developing a Spark applications in Eclipse with SBT

Developing a Spark application in IntelliJ IDEA with Maven

Developing a Spark application in IntelliJ IDEA with SBT

Developing applications using the Zeppelin notebook

Setting up Kerberos to do authentication

Enabling Kerberos authentication for Spark

Spark SQL

Spark SQL

Understanding the evolution of schema awareness

Understanding the Catalyst optimizer

Inferring schema using case classes

Programmatically specifying the schema

Understanding the Parquet format

Loading and saving data using the JSON format

Loading and saving data from relational databases

Loading and saving data from an arbitrary source

Understanding joins

Analyzing nested structures

Working with External Data Sources

Working with External Data Sources

Introduction

Loading data from the local filesystem

Loading data from HDFS

Loading data from Amazon S3

Loading data from Apache Cassandra

Spark Streaming

Spark Streaming

Introduction

WordCount using Structured Streaming

Taking a closer look at Structured Streaming

Streaming Twitter data

Streaming using Kafka

Understanding streaming challenges

Getting Started with Machine Learning

Getting Started with Machine Learning

Introduction

Creating vectors

Calculating correlation

Understanding feature engineering

Understanding Spark ML

Understanding hyperparameter tuning

Supervised Learning with MLlib — Regression

Supervised Learning with MLlib — Regression

Introduction

Using linear regression

Understanding the cost function

Doing linear regression with lasso

Doing ridge regression

Supervised Learning with MLlib — Classification

Supervised Learning with MLlib — Classification

Introduction

Doing classification using logistic regression

Doing binary classification using SVM

Doing classification using decision trees

Doing classification using random forest

Doing classification using gradient boosted trees

Doing classification with Naïve Bayes

Unsupervised Learning

Unsupervised Learning

Introduction

Clustering using k-means

Dimensionality reduction with principal component analysis

Dimensionality reduction with singular value decomposition

Recommendations Using Collaborative Filtering

Recommendations Using Collaborative Filtering

Introduction

Collaborative filtering using explicit feedback

Collaborative filtering using implicit feedback

Graph Processing Using GraphX and GraphFrames

Graph Processing Using GraphX and GraphFrames

Introduction

Fundamental operations on graphs

Using PageRank

Finding connected components

Performing neighborhood aggregation

Understanding GraphFrames

Optimizations and Performance Tuning

Optimizations and Performance Tuning

Optimizing memory

Leveraging speculation

Optimizing joins

Using compression to improve performance

Using serialization to improve performance

Optimizing the level of parallelism

Understanding project Tungsten

Building the Spark source code with Maven

Installing Spark using binaries works fine in most cases. For advanced cases, such as the following (but not limited to), compiling from the source code is a better option:

Compiling for a specific Hadoop version
Adding the Hive integration
Adding the YARN integration

Getting ready

The following are the prerequisites for this recipe to work:

Java 1.8 or a later version
Maven 3.x

How to do it...

The following are the steps to build the Spark source code with Maven:

Increase MaxPermSize of the heap:

       $ echo "export _JAVA_OPTIONS="-XX:MaxPermSize=1G""  >> 
         /home/hduser/.bashrc

Open a new terminal window and download the Spark source code from GitHub:

        $ wget https://github.com/apache/spark/archive/branch-2.1.zip

Unpack the archive:

        $ unzip branch-2.1.zip

Rename unzipped folder to spark:

        $ mv spark-branch-2.1 spark

Move to the spark directory:

        $ cd spark

Compile the sources with the YARN-enabled, Hadoop version 2.7, and Hive-enabled flags and skip the tests for faster compilation:

      $ mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -Phive -
        DskipTests clean package

Move the conf folder to the etc folder so that it can be turned into a symbolic link:

        $ sudo mv spark/conf /etc/

Move the spark directory to /opt as it's an add-on software package:

        $ sudo mv spark /opt/infoobjects/spark

Change the ownership of the spark home directory to root:

        $ sudo chown -R root:root /opt/infoobjects/spark

Change the permissions of the spark home directory, namely 0755 = user:rwx group:r-x world:r-x:

        $ sudo chmod -R 755 /opt/infoobjects/spark

Move to the spark home directory:

        $ cd /opt/infoobjects/spark

Create a symbolic link:

        $ sudo ln -s /etc/spark conf

Put the Spark executable in the path by editing .bashrc:

        $ echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> 
          /home/hduser/.bashrc

Create the log directory in /var:

        $ sudo mkdir -p /var/log/spark

Make hduser the owner of Spark's log directory:

        $ sudo chown -R hduser:hduser /var/log/spark

Create Spark's tmp directory:

        $ mkdir /tmp/spark

Configure Spark with the help of the following command lines:

     $ cd /etc/spark
$ echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" 
       >> spark-env.sh
$ echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" 
       >> spark-env.sh
$ echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
$ echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Apache Spark 2.x Cookbook

Search

Your notes and bookmarks