2. Resilient Distributed Datasets | Learning PySpark

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Learning PySpark

By : Drabas, Lee

3.9 (194)

Learning PySpark

3.9 (194)

By: Drabas, Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Understanding Spark

1. Understanding Spark

What is Apache Spark?

Spark Jobs and APIs

Spark 2.0 architecture

Summary

2. Resilient Distributed Datasets

2. Resilient Distributed Datasets

Internal workings of an RDD

Creating RDDs

Global versus local scope

Transformations

Actions

Summary

3. DataFrames

3. DataFrames

Python to RDD communications

Catalyst Optimizer refresh

Speeding up PySpark with DataFrames

Creating DataFrames

Simple DataFrame queries

Interoperating with RDDs

Querying with the DataFrame API

Querying with SQL

DataFrame scenario – on-time flight performance

Spark Dataset API

Summary

4. Prepare Data for Modeling

4. Prepare Data for Modeling

Checking for duplicates, missing observations, and outliers

Getting familiar with your data

Visualization

Summary

5. Introducing MLlib

5. Introducing MLlib

Overview of the package

Loading and transforming the data

Getting to know your data

Creating the final dataset

Predicting infant survival

Summary

6. Introducing the ML Package

6. Introducing the ML Package

Overview of the package

Predicting the chances of infant survival with ML

Parameter hyper-tuning

Other features of PySpark ML in action

Summary

7. GraphFrames

7. GraphFrames

Introducing GraphFrames

Installing GraphFrames

Preparing your flights dataset

Building the graph

Executing simple queries

Understanding vertex degrees

Determining the top transfer airports

Understanding motifs

Determining airport ranking using PageRank

Determining the most popular non-stop flights

Using Breadth-First Search

Visualizing flights using D3

Summary

8. TensorFrames

8. TensorFrames

What is Deep Learning?

What is TensorFlow?

Introducing TensorFrames

TensorFrames – quick start

Summary

9. Polyglot Persistence with Blaze

9. Polyglot Persistence with Blaze

Installing Blaze

Polyglot persistence

Abstracting data

Data operations

Summary

10. Structured Streaming

10. Structured Streaming

What is Spark Streaming?

Why do we need Spark Streaming?

What is the Spark Streaming application data flow?

Simple streaming application using DStreams

A quick primer on global aggregations

Introducing Structured Streaming

Summary

11. Packaging Spark Applications

11. Packaging Spark Applications

The spark-submit command

Deploying the app programmatically

Databricks Jobs

Summary

Index

Index

Internal workings of an RDD

RDDs operate in parallel. This is the strongest advantage of working in Spark: Each transformation is executed in parallel for enormous increase in speed.

The transformations to the dataset are lazy. This means that any transformation is only executed when an action on a dataset is called. This helps Spark to optimize the execution. For instance, consider the following very common steps that an analyst would normally do to get familiar with a dataset:

Count the occurrence of distinct values in a certain column.
Select those that start with an A.
Print the results to the screen.

As simple as the previously mentioned steps sound, if only items that start with the letter A are of interest, there is no point in counting distinct values for all the other items. Thus, instead of following the execution as outlined in the preceding points, Spark could only count the items that start with A, and then print the results to the screen.

Let's break this example down in code. First...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Learning PySpark

Search

Your notes and bookmarks