Book Image

Hands-On Big Data Analytics with PySpark

By : Rudy Lai, Bartłomiej Potaczek
Book Image

Hands-On Big Data Analytics with PySpark

By: Rudy Lai, Bartłomiej Potaczek

Overview of this book

Apache Spark is an open source parallel-processing framework that has been around for quite some time now. One of the many uses of Apache Spark is for data analytics applications across clustered computers. In this book, you will not only learn how to use Spark and the Python API to create high-performance analytics with big data, but also discover techniques for testing, immunizing, and parallelizing Spark jobs. You will learn how to source data from all popular data hosting platforms, including HDFS, Hive, JSON, and S3, and deal with large datasets with PySpark to gain practical big data experience. This book will help you work on prototypes on local machines and subsequently go on to handle messy data in production and at scale. This book covers installing and setting up PySpark, RDD operations, big data cleaning and wrangling, and aggregating and summarizing data into useful reports. You will also learn how to implement some practical and proven techniques to improve certain aspects of programming and administration in Apache Spark. By the end of the book, you will be able to build big data analytical solutions using the various PySpark offerings and also optimize them effectively.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Big Data Analytics with PySpark
Rudy Lai | Bartłomiej Potaczek
Mar, 2019 | 182 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Small book image
Scala and Spark for Big Data Analytics
Md Rezaul Karim | Sridhar Alla
Jul, 2017 | 796 pages
Small book image
Apache Spark 2.x for Java Developers
Sumit Kumar | Sourav Gulati
Jul, 2017 | 350 pages
Table of Contents (15 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Installing Pyspark and Setting up Your Development Environment
Installing Pyspark and Setting up Your Development Environment
An overview of PySpark
Setting up Spark on Windows and PySpark
Core concepts in Spark and PySpark
Summary
2
Getting Your Big Data into the Spark Environment Using RDDs
Getting Your Big Data into the Spark Environment Using RDDs
Loading data on to Spark RDDs
Parallelization with Spark RDDs
Basics of RDD operation
Summary
3
Big Data Cleaning and Wrangling with Spark Notebooks
Big Data Cleaning and Wrangling with Spark Notebooks
Using Spark Notebooks for quick iteration of ideas
Sampling/filtering RDDs to pick out relevant data points
Splitting datasets and creating some new combinations
Summary
4
Aggregating and Summarizing Data into Useful Reports
Aggregating and Summarizing Data into Useful Reports
Calculating averages with map and reduce
Faster average computations with aggregate
Pivot tabling with key-value paired data points
Summary
5
Powerful Exploratory Data Analysis with MLlib
Powerful Exploratory Data Analysis with MLlib
Computing summary statistics with MLlib
Using Pearson and Spearman correlations to discover correlations
Testing our hypotheses on large datasets
Summary
6
Putting Structure on Your Big Data with SparkSQL
Putting Structure on Your Big Data with SparkSQL
Manipulating DataFrames with Spark SQL schemas
Using Spark DSL to build queries
Summary
7
Transformations and Actions
Transformations and Actions
Using Spark transformations to defer computations to a later time
Avoiding transformations
Using the reduce and reduceByKey methods to calculate the results
Performing actions that trigger computations
Reusing the same rdd for different actions
Summary
8
Immutable Design
Immutable Design
Delving into the Spark RDD's parent/child chain
Using RDD in an immutable way
Using DataFrame operations to transform
Immutability in the highly concurrent environment
Using the Dataset API in an immutable way
Summary
9
Avoiding Shuffle and Reducing Operational Expenses
Avoiding Shuffle and Reducing Operational Expenses
Detecting a shuffle in a process
Testing operations that cause a shuffle in Apache Spark
Changing the design of jobs with wide dependencies
Using keyBy() operations to reduce shuffle
Using a custom partitioner to reduce shuffle
Summary
10
Saving Data in the Correct Format
Saving Data in the Correct Format
Saving data in plain text format
Leveraging JSON as a data format
Tabular formats – CSV
Using Avro with Spark
Columnar formats – Parquet
Summary
11
Working with the Spark Key/Value API
Working with the Spark Key/Value API
Available actions on key/value pairs
Using aggregateByKey instead of groupBy()
Actions on key/value pairs
Available partitioners on key/value data
Implementing a custom partitioner
Summary
12
Testing Apache Spark Jobs
Testing Apache Spark Jobs
Separating logic from Spark engine-unit testing
Integration testing using SparkSession
Mocking data sources using partial functions
Using ScalaCheck for property-based testing
Testing in different versions of Spark
Summary
13
Leveraging the Spark GraphX API
Leveraging the Spark GraphX API
Creating a graph from a data source
Using the Vertex API
Using the Edge API
Calculating the degree of the vertex
Calculating PageRank
Summary
14
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Detecting a shuffle in a process

In this section, we will learn how to detect a shuffle in a process.

In this section, we will cover the following topics:

  • Loading randomly partitioned data
  • Issuing repartition using a meaningful partition key
  • Understanding how shuffle occurs by explaining a query

We will load randomly partitioned data to see how and where the data is loaded. Next, we will issue a partition using a meaningful partition key. We will then repartition data to the proper executors using the deterministic and meaningful key. In the end, we will explain our queries by using the explain() method and understand the shuffle. Here, we have a very simple test.

We will create a DataFrame with some data. For example, we created an InputRecord with some random UID and user_1, and another input with random ID in user_1, and the last record for user_2. Let's imagine that...

Previous Section
End of Section 2
Next Section