Book Image

PySpark Cookbook

By : Denny Lee, Tomasz Drabas
Book Image

PySpark Cookbook

By: Denny Lee, Tomasz Drabas

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. The PySpark Cookbook presents effective and time-saving recipes for leveraging the power of Python and putting it to use in the Spark ecosystem. You’ll start by learning the Apache Spark architecture and how to set up a Python environment for Spark. You’ll then get familiar with the modules available in PySpark and start using them effortlessly. In addition to this, you’ll discover how to abstract data with RDDs and DataFrames, and understand the streaming capabilities of PySpark. You’ll then move on to using ML and MLlib in order to solve any problems related to the machine learning capabilities of PySpark and use GraphFrames to solve graph-processing problems. Finally, you will explore how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will be able to use the Python API for Apache Spark to solve any problems associated with building data-intensive applications.

Related Content you might be interested in

Current Title:

Small book image
PySpark Cookbook
Denny Lee | Tomasz Drabas
Jun, 2018 | 330 pages
Small book image
Learning PySpark
Denny Lee | Tomasz Drabas
Feb, 2017 | 274 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Small book image
Mastering Apache Spark 2.x
Romeo Kienzler
Jul, 2017 | 354 pages
Table of Contents (13 chapters)
Title Page
Title Page
Packt Upsell
Packt Upsell
Contributors
Contributors
Preface
Preface
Free Chapter
1
Installing and Configuring Spark
Installing and Configuring Spark
Introduction
Installing Spark requirements
Installing Spark from sources
Installing Spark from binaries
Configuring a local instance of Spark
Configuring a multi-node instance of Spark
Installing Jupyter
Configuring a session in Jupyter
Working with Cloudera Spark images
2
Abstracting Data with RDDs
Abstracting Data with RDDs
Introduction
Creating RDDs
Reading data from files
Overview of RDD transformations
Overview of RDD actions
Pitfalls of using RDDs
3
Abstracting Data with DataFrames
Abstracting Data with DataFrames
Introduction
Creating DataFrames
Accessing underlying RDDs
Performance optimizations
Inferring the schema using reflection
Specifying the schema programmatically
Creating a temporary table
Using SQL to interact with DataFrames
Overview of DataFrame transformations
Overview of DataFrame actions
4
Preparing Data for Modeling
Preparing Data for Modeling
Introduction
Handling duplicates
Handling missing observations
Handling outliers
Exploring descriptive statistics
Computing correlations
Drawing histograms
Visualizing interactions between features
5
Machine Learning with MLlib
Machine Learning with MLlib
Loading the data
Exploring the data
Testing the data
Transforming the data
Standardizing the data
Creating an RDD for training
Predicting hours of work for census respondents
Forecasting the income levels of census respondents
Building a clustering models
Computing performance statistics
6
Machine Learning with the ML Module
Machine Learning with the ML Module
Introducing Transformers
Introducing Estimators
Introducing Pipelines
Selecting the most predictable features
Predicting forest coverage types
Estimating forest elevation
Clustering forest cover types
Tuning hyperparameters
Extracting features from text
Discretizing continuous variables
Standardizing continuous variables
Topic mining
7
Structured Streaming with PySpark
Structured Streaming with PySpark
Introduction
Understanding DStreams
Understanding global aggregations
Continuous aggregation with structured streaming
8
GraphFrames – Graph Theory with PySpark
GraphFrames – Graph Theory with PySpark
Introduction
Installing GraphFrames
Preparing the data
Building the graph
Running queries against the graph
Understanding the graph
Using PageRank to determine airport ranking
Finding the fewest number of connections
Visualizing the graph
Index
Index

Running queries against the graph

Now that you have created your graph, start off by creating and running some simple queries against your GraphFrame.

Getting ready

Ensure that you have created the graph GraphFrame (derived from the vertices and edges DataFrames) from the previous section.

How to do it...

Let's start with some simple count queries to determine the number of airports (nodes or vertices; remember?) and the number of flights (the edges), which can be determined by applying count(). The call to count() is similar to a DataFrame except that you also need to include whether you are countingverticesoredges:

print "Airport count: %d" % graph.vertices.count()
print "Trips count: %d" % graph.edges.count()

The output of these queries should be similar to the following output, denoting the 279 vertices (that is, airports) and more than 1.3 million edges (that is, flights):

Output:
  Airports count: 279 
  Trips count: 1361141

Similar to DataFrames, you can also execute the filter and groupBy...

Previous Section
End of Section 6
Next Section