Book Image

Spark Cookbook

By : Rishi Yadav
Book Image

Spark Cookbook

By: Rishi Yadav

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Spark Cookbook
Rishi Yadav
Jul, 2015 | 226 pages
No titles found
Table of Contents (19 chapters)
Spark Cookbook
Spark Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Getting Started with Apache Spark
Getting Started with Apache Spark
Introduction
Installing Spark from binaries
Building the Spark source code with Maven
Launching Spark on Amazon EC2
Deploying on a cluster in standalone mode
Deploying on a cluster with Mesos
Deploying on a cluster with YARN
Using Tachyon as an off-heap storage layer
2
Developing Applications with Spark
Developing Applications with Spark
Introduction
Exploring the Spark shell
Developing Spark applications in Eclipse with Maven
Developing Spark applications in Eclipse with SBT
Developing a Spark application in IntelliJ IDEA with Maven
Developing a Spark application in IntelliJ IDEA with SBT
3
External Data Sources
External Data Sources
Introduction
Loading data from the local filesystem
Loading data from HDFS
Loading data from HDFS using a custom InputFormat
Loading data from Amazon S3
Loading data from Apache Cassandra
Loading data from relational databases
4
Spark SQL
Spark SQL
Introduction
Understanding the Catalyst optimizer
Creating HiveContext
Inferring schema using case classes
Programmatically specifying the schema
Loading and saving data using the Parquet format
Loading and saving data using the JSON format
Loading and saving data from relational databases
Loading and saving data from an arbitrary source
5
Spark Streaming
Spark Streaming
Introduction
Word count using Streaming
Streaming Twitter data
Streaming using Kafka
6
Getting Started with Machine Learning Using MLlib
Getting Started with Machine Learning Using MLlib
Introduction
Creating vectors
Creating a labeled point
Creating matrices
Calculating summary statistics
Calculating correlation
Doing hypothesis testing
Creating machine learning pipelines using ML
7
Supervised Learning with MLlib – Regression
Supervised Learning with MLlib – Regression
Introduction
Using linear regression
Understanding cost function
Doing linear regression with lasso
Doing ridge regression
8
Supervised Learning with MLlib – Classification
Supervised Learning with MLlib – Classification
Introduction
Doing classification using logistic regression
Doing binary classification using SVM
Doing classification using decision trees
Doing classification using Random Forests
Doing classification using Gradient Boosted Trees
Doing classification with Naïve Bayes
9
Unsupervised Learning with MLlib
Unsupervised Learning with MLlib
Introduction
Clustering using k-means
Dimensionality reduction with principal component analysis
Dimensionality reduction with singular value decomposition
10
Recommender Systems
Recommender Systems
Introduction
Collaborative filtering using explicit feedback
Collaborative filtering using implicit feedback
11
Graph Processing Using GraphX
Graph Processing Using GraphX
Introduction
Fundamental operations on graphs
Using PageRank
Finding connected components
Performing neighborhood aggregation
12
Optimizations and Performance Tuning
Optimizations and Performance Tuning
Introduction
Optimizing memory
Using compression to improve performance
Using serialization to improve performance
Optimizing garbage collection
Optimizing the level of parallelism
Understanding the future of optimization – project Tungsten
Index
Index

Installing Spark from binaries

Spark can be either built from the source code or precompiled binaries can be downloaded from http://spark.apache.org. For a standard use case, binaries are good enough, and this recipe will focus on installing Spark using binaries.

Getting ready

All the recipes in this book are developed using Ubuntu Linux but should work fine on any POSIX environment. Spark expects Java to be installed and the JAVA_HOME environment variable to be set.

In Linux/Unix systems, there are certain standards for the location of files and directories, which we are going to follow in this book. The following is a quick cheat sheet:

Directory

Description

/bin

Essential command binaries

/etc

Host-specific system configuration

/opt

Add-on application software packages

/var

Variable data

/tmp

Temporary files

/home

User home directories

How to do it...

At the time of writing this, Spark's current version is 1.4. Please check the latest version from Spark's download page at http://spark.apache.org/downloads.html. Binaries are developed with a most recent and stable version of Hadoop. To use a specific version of Hadoop, the recommended approach is to build from sources, which will be covered in the next recipe.

The following are the installation steps:

  1. Open the terminal and download binaries using the following command:

    $ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.4.tgz

  2. Unpack binaries:

    $ tar -zxf spark-1.4.0-bin-hadoop2.4.tgz

  3. Rename the folder containing binaries by stripping the version information:

    $ sudo mv spark-1.4.0-bin-hadoop2.4 spark

  4. Move the configuration folder to the /etc folder so that it can be made a symbolic link later:

    $ sudo mv spark/conf/* /etc/spark

  5. Create your company-specific installation directory under /opt. As the recipes in this book are tested on infoobjects sandbox, we are going to use infoobjects as directory name. Create the /opt/infoobjects directory:

    $ sudo mkdir -p /opt/infoobjects

  6. Move the spark directory to /opt/infoobjects as it's an add-on software package:

    $ sudo mv spark /opt/infoobjects/

  7. Change the ownership of the spark home directory to root:

    $ sudo chown -R root:root /opt/infoobjects/spark

  8. Change permissions of the spark home directory, 0755 = user:read-write-execute group:read-execute world:read-execute:

    $ sudo chmod -R 755 /opt/infoobjects/spark

  9. Move to the spark home directory:

    $ cd /opt/infoobjects/spark

  10. Create the symbolic link:

    $ sudo ln -s /etc/spark conf

  11. Append to PATH in .bashrc:

    $ echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc

  12. Open a new terminal.

  13. Create the log directory in /var:

    $ sudo mkdir -p /var/log/spark

  14. Make hduser the owner of the Spark log directory.

    $ sudo chown -R hduser:hduser /var/log/spark

  15. Create the Spark tmp directory:

    $ mkdir /tmp/spark

  16. Configure Spark with the help of the following command lines:

    $ cd /etc/spark
$ echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh
$ echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh
$ echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
$ echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh
Previous Section
End of Section 3
Next Section