Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Fast Data Processing with Spark 2 - Third Edition

By : Krishna Sankar , Holden Karau

Fast Data Processing with Spark 2

By: Krishna Sankar , Holden Karau

Overview of this book

When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere. Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge. You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Installing Spark and Setting Up Your Cluster

1. Installing Spark and Setting Up Your Cluster

Directory organization and convention

Installing the prebuilt distribution

Building Spark from source

Spark topology

A single machine

Running Spark on EC2

Deploying Spark with Chef (Opscode)

Deploying Spark on Mesos

Spark on YARN

Spark standalone mode

References

Summary

2. Using the Spark Shell

2. Using the Spark Shell

The Spark shell

Loading a simple text file

Interactively loading data from S3

Summary

3. Building and Running a Spark Application

3. Building and Running a Spark Application

Building Spark applications

Data wrangling with iPython

Developing Spark with Eclipse

Developing Spark with other IDEs

Building your Spark job with Maven

Building your Spark job with something else

References

Summary

4. Creating a SparkSession Object

4. Creating a SparkSession Object

SparkSession versus SparkContext

Building a SparkSession object

SparkContext - metadata

Shared Java and Scala APIs

Python

iPython

Reference

Summary

5. Loading and Saving Data in Spark

5. Loading and Saving Data in Spark

Spark abstractions

Data modalities

Data modalities and Datasets/DataFrames/RDDs

Loading data into an RDD

Saving your data

References

Summary

6. Manipulating Your RDD

6. Manipulating Your RDD

Manipulating your RDD in Scala and Java

Manipulating your RDD in Python

References

Summary

7. Spark 2.0 Concepts

7. Spark 2.0 Concepts

Code and Datasets for the rest of the book

The data scientist and Spark features

Spark v2.0 and beyond

Apache Spark - evolution

Apache Spark - the full stack

The art of a big data store - Parquet

References

Summary

8. Spark SQL

8. Spark SQL

The Spark SQL architecture

Spark SQL how-to in a nutshell

Spark SQL programming

References

Summary

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

Datasets - a quick introduction

Dataset APIs - an overview

Dataset interfaces and functions

References

Summary

10. Spark with Big Data

10. Spark with Big Data

Parquet - an efficient and interoperable big data format

HBase

Reference

Summary

11. Machine Learning with Spark ML Pipelines

11. Machine Learning with Spark ML Pipelines

Spark's machine learning algorithm table

Spark machine learning APIs - ML pipelines and MLlib

ML pipelines

Spark ML examples

The API organization

Basic statistics

Linear regression

Classification

Clustering

Recommendation

Hyper parameters

The final thing

References

Summary

12. GraphX

12. GraphX

Graphs and graph processing - an introduction

Spark GraphX

GraphX - computational model

The first example - graph

Building graphs

The GraphX API landscape

Structural APIs

Community, affiliation, and strengths

Algorithms

Partition strategy

Case study - AlphaGo tweets analytics

References

Summary

A single machine

A single machine is the simplest use case for Spark. It is also a great way to sanity check your build. In spark/bin, there is a shell script called run-example, which can be used to launch a Spark job. The run-example script takes the name of a Spark class and some arguments. Earlier, we used the run-example script from the /bin directory to calculate the value of Pi. There is a collection of the sample Spark jobs in examples/src/main/scala/org/apache/spark/examples/.

All of the sample programs take the parameter, master (the cluster manager), which can be the URL of a distributed cluster or local[N], where N is the number of threads.

Going back to our run-example script, it invokes the more general bin/spark-submit script. For now, let's stick with the run-example script.

To run GroupByTest locally, try running the following command:

bin/run-example GroupByTest

This line will produce an output like this given here:

14/11/15 06:28:40 INFO SparkContext: Job finished: count at  GroupByTest.scala:51, took 0.494519333 s
2000

Note

All the examples in this book can be run on a Spark installation on a local machine. So you can read through the rest of the chapter for additional information after you have gotten some hands-on exposure to Spark running on your local machine.

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Fast Data Processing with Spark 2

Search

Your notes and bookmarks