Book Image

Learning PySpark

By : Tomasz Drabas, Denny Lee
Book Image

Learning PySpark

By: Tomasz Drabas, Denny Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Related Content you might be interested in

Current Title:

Small book image
Learning PySpark
Tomasz Drabas | Denny Lee
Feb, 2017 | 274 pages
Small book image
PySpark Cookbook
Denny Lee | Tomasz Drabas
Jun, 2018 | 330 pages
Small book image
Learning Spark SQL
Aurobindo Sarkar
Sep, 2017 | 452 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Table of Contents (20 chapters)
Learning PySpark
Learning PySpark
Credits
Credits
Foreword
Foreword
About the Authors
About the Authors
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Understanding Spark
Understanding Spark
What is Apache Spark?
Spark Jobs and APIs
Spark 2.0 architecture
Summary
2
Resilient Distributed Datasets
Resilient Distributed Datasets
Internal workings of an RDD
Creating RDDs
Global versus local scope
Transformations
Actions
Summary
3
DataFrames
DataFrames
Python to RDD communications
Catalyst Optimizer refresh
Speeding up PySpark with DataFrames
Creating DataFrames
Simple DataFrame queries
Interoperating with RDDs
Querying with the DataFrame API
Querying with SQL
DataFrame scenario – on-time flight performance
Spark Dataset API
Summary
4
Prepare Data for Modeling
Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Getting familiar with your data
Visualization
Summary
5
Introducing MLlib
Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Creating the final dataset
Predicting infant survival
Summary
6
Introducing the ML Package
Introducing the ML Package
Overview of the package
Predicting the chances of infant survival with ML
Parameter hyper-tuning
Other features of PySpark ML in action
Summary
7
GraphFrames
GraphFrames
Introducing GraphFrames
Installing GraphFrames
Preparing your flights dataset
Building the graph
Executing simple queries
Understanding vertex degrees
Determining the top transfer airports
Understanding motifs
Determining airport ranking using PageRank
Determining the most popular non-stop flights
Using Breadth-First Search
Visualizing flights using D3
Summary
8
TensorFrames
TensorFrames
What is Deep Learning?
What is TensorFlow?
Introducing TensorFrames
TensorFrames – quick start
Summary
9
Polyglot Persistence with Blaze
Polyglot Persistence with Blaze
Installing Blaze
Polyglot persistence
Abstracting data
Data operations
Summary
10
Structured Streaming
Structured Streaming
What is Spark Streaming?
Why do we need Spark Streaming?
What is the Spark Streaming application data flow?
Simple streaming application using DStreams
A quick primer on global aggregations
Introducing Structured Streaming
Summary
11
Packaging Spark Applications
Packaging Spark Applications
The spark-submit command
Deploying the app programmatically
Databricks Jobs
Summary
Index
Index

Deploying the app programmatically

Unlike the Jupyter notebooks, when you use the spark-submit command, you need to prepare the SparkSession yourself and configure it so your application runs properly.

In this section, we will learn how to create and configure the SparkSession as well as how to use modules external to Spark.

Note

If you have not created your free account with either Databricks or Microsoft (or any other provider of Spark) do not worry - we will be still using your local machine as this is easier to get us started. However, if you decide to take your application to the cloud it will literally only require changing the --master parameter when you submit the job.

Configuring your SparkSession

The main difference between using Jupyter and submitting jobs programmatically is the fact that you have to create your Spark context (and Hive, if you plan to use HiveQL), whereas when running Spark with Jupyter the contexts are automatically started for you.

In this section, we will develop...

Previous Section
End of Section 3
Next Section