Book Image

Fast Data Processing with Spark 2 - Third Edition

By : Holden Karau
Book Image

Fast Data Processing with Spark 2 - Third Edition

By: Holden Karau

Overview of this book

When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere. Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge. You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.

Related Content you might be interested in

Current Title:

Small book image
Fast Data Processing with Spark 2 - Third Edition
Holden Karau
Oct, 2016 | 274 pages
No titles found
Table of Contents (18 chapters)
Fast Data Processing with Spark 2 Third Edition
Fast Data Processing with Spark 2 Third Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Installing Spark and Setting Up Your Cluster
Installing Spark and Setting Up Your Cluster
Directory organization and convention
Installing the prebuilt distribution
Building Spark from source
Spark topology
A single machine
Running Spark on EC2
Deploying Spark with Chef (Opscode)
Deploying Spark on Mesos
Spark on YARN
Spark standalone mode
References
Summary
2
Using the Spark Shell
Using the Spark Shell
The Spark shell
Loading a simple text file
Interactively loading data from S3
Summary
3
Building and Running a Spark Application
Building and Running a Spark Application
Building Spark applications
Data wrangling with iPython
Developing Spark with Eclipse
Developing Spark with other IDEs
Building your Spark job with Maven
Building your Spark job with something else
References
Summary
4
Creating a SparkSession Object
Creating a SparkSession Object
SparkSession versus SparkContext
Building a SparkSession object
SparkContext - metadata
Shared Java and Scala APIs
Python
iPython
Reference
Summary
5
Loading and Saving Data in Spark
Loading and Saving Data in Spark
Spark abstractions
Data modalities
Data modalities and Datasets/DataFrames/RDDs
Loading data into an RDD
Saving your data
References
Summary
6
Manipulating Your RDD
Manipulating Your RDD
Manipulating your RDD in Scala and Java
Manipulating your RDD in Python
References
Summary
7
Spark 2.0 Concepts
Spark 2.0 Concepts
Code and Datasets for the rest of the book
The data scientist and Spark features
Spark v2.0 and beyond
Apache Spark - evolution
Apache Spark - the full stack
The art of a big data store - Parquet
References
Summary
8
Spark SQL
Spark SQL
The Spark SQL architecture
Spark SQL how-to in a nutshell
Spark SQL programming
References
Summary
9
Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists
Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists
Datasets - a quick introduction
Dataset APIs - an overview
Dataset interfaces and functions
References
Summary
10
Spark with Big Data
Spark with Big Data
Parquet - an efficient and interoperable big data format
HBase
Reference
Summary
11
Machine Learning with Spark ML Pipelines
Machine Learning with Spark ML Pipelines
Spark's machine learning algorithm table
Spark machine learning APIs - ML pipelines and MLlib
ML pipelines
Spark ML examples
The API organization
Basic statistics
Linear regression
Classification
Clustering
Recommendation
Hyper parameters
The final thing
References
Summary
12
GraphX
GraphX
Graphs and graph processing - an introduction
Spark GraphX
GraphX - computational model
The first example - graph
Building graphs
The GraphX API landscape
Structural APIs
Community, affiliation, and strengths
Algorithms
Partition strategy
Case study - AlphaGo tweets analytics
References
Summary

Dataset interfaces and functions

Now let's work out a few interesting examples, starting out with a simple one and then moving on to progressively complex operations.

Tip

The code files are in fdps-v3/code, and the data files are in fdps-v3/data. You can run the code either from a Scala IDE or just from the Spark Shell.

Start Spark Shell from the bin directory where you have installed the spark:

/Volumes/sdxc-01/spark-2.0.0/bin/spark-shell

Inside the shell, the following command will load the source:

:load /Users/ksankar/fdps-v3/code/DS01.scala

Read/write operations

As we saw earlier, SparkSession.read.* gives us a rich set of features to read different types of data with flexible control over the options. Dataset.write.* does the same for writing data:

val spark = SparkSession.builder 
      .master("local") 
      .appName("Chapter 9") 
      .config("spark.logConf","true") 
      .config("spark.logLevel","ERROR") 
      .getOrCreate() 
println("Running Spark...
Previous Section
End of Section 4
Next Section