Book Image

Fast Data Processing with Spark 2 - Third Edition

By : Holden Karau
Book Image

Fast Data Processing with Spark 2 - Third Edition

By: Holden Karau

Overview of this book

When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere. Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge. You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.

Related Content you might be interested in

Current Title:

Small book image
Fast Data Processing with Spark 2 - Third Edition
Holden Karau
Oct, 2016 | 274 pages
No titles found
Table of Contents (18 chapters)
Fast Data Processing with Spark 2 Third Edition
Fast Data Processing with Spark 2 Third Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Installing Spark and Setting Up Your Cluster
Installing Spark and Setting Up Your Cluster
Directory organization and convention
Installing the prebuilt distribution
Building Spark from source
Spark topology
A single machine
Running Spark on EC2
Deploying Spark with Chef (Opscode)
Deploying Spark on Mesos
Spark on YARN
Spark standalone mode
References
Summary
2
Using the Spark Shell
Using the Spark Shell
The Spark shell
Loading a simple text file
Interactively loading data from S3
Summary
3
Building and Running a Spark Application
Building and Running a Spark Application
Building Spark applications
Data wrangling with iPython
Developing Spark with Eclipse
Developing Spark with other IDEs
Building your Spark job with Maven
Building your Spark job with something else
References
Summary
4
Creating a SparkSession Object
Creating a SparkSession Object
SparkSession versus SparkContext
Building a SparkSession object
SparkContext - metadata
Shared Java and Scala APIs
Python
iPython
Reference
Summary
5
Loading and Saving Data in Spark
Loading and Saving Data in Spark
Spark abstractions
Data modalities
Data modalities and Datasets/DataFrames/RDDs
Loading data into an RDD
Saving your data
References
Summary
6
Manipulating Your RDD
Manipulating Your RDD
Manipulating your RDD in Scala and Java
Manipulating your RDD in Python
References
Summary
7
Spark 2.0 Concepts
Spark 2.0 Concepts
Code and Datasets for the rest of the book
The data scientist and Spark features
Spark v2.0 and beyond
Apache Spark - evolution
Apache Spark - the full stack
The art of a big data store - Parquet
References
Summary
8
Spark SQL
Spark SQL
The Spark SQL architecture
Spark SQL how-to in a nutshell
Spark SQL programming
References
Summary
9
Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists
Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists
Datasets - a quick introduction
Dataset APIs - an overview
Dataset interfaces and functions
References
Summary
10
Spark with Big Data
Spark with Big Data
Parquet - an efficient and interoperable big data format
HBase
Reference
Summary
11
Machine Learning with Spark ML Pipelines
Machine Learning with Spark ML Pipelines
Spark's machine learning algorithm table
Spark machine learning APIs - ML pipelines and MLlib
ML pipelines
Spark ML examples
The API organization
Basic statistics
Linear regression
Classification
Clustering
Recommendation
Hyper parameters
The final thing
References
Summary
12
GraphX
GraphX
Graphs and graph processing - an introduction
Spark GraphX
GraphX - computational model
The first example - graph
Building graphs
The GraphX API landscape
Structural APIs
Community, affiliation, and strengths
Algorithms
Partition strategy
Case study - AlphaGo tweets analytics
References
Summary

A single machine

A single machine is the simplest use case for Spark. It is also a great way to sanity check your build. In spark/bin, there is a shell script called run-example, which can be used to launch a Spark job. The run-example script takes the name of a Spark class and some arguments. Earlier, we used the run-example script from the /bin directory to calculate the value of Pi. There is a collection of the sample Spark jobs in examples/src/main/scala/org/apache/spark/examples/.

All of the sample programs take the parameter, master (the cluster manager), which can be the URL of a distributed cluster or local[N], where N is the number of threads.

Going back to our run-example script, it invokes the more general bin/spark-submit script. For now, let's stick with the run-example script.

To run GroupByTest locally, try running the following command:

bin/run-example GroupByTest

This line will produce an output like this given here:

14/11/15 06:28:40 INFO SparkContext: Job finished: count at  GroupByTest.scala:51, took 0.494519333 s
2000

Note

All the examples in this book can be run on a Spark installation on a local machine. So you can read through the rest of the chapter for additional information after you have gotten some hands-on exposure to Spark running on your local machine.

Previous Section
End of Section 6
Next Section