Book Image

Hands-On Big Data Analytics with PySpark

By : Rudy Lai, Bartłomiej Potaczek

Book Image

Hands-On Big Data Analytics with PySpark

By: Rudy Lai, Bartłomiej Potaczek

Overview of this book

Apache Spark is an open source parallel-processing framework that has been around for quite some time now. One of the many uses of Apache Spark is for data analytics applications across clustered computers. In this book, you will not only learn how to use Spark and the Python API to create high-performance analytics with big data, but also discover techniques for testing, immunizing, and parallelizing Spark jobs. You will learn how to source data from all popular data hosting platforms, including HDFS, Hive, JSON, and S3, and deal with large datasets with PySpark to gain practical big data experience. This book will help you work on prototypes on local machines and subsequently go on to handle messy data in production and at scale. This book covers installing and setting up PySpark, RDD operations, big data cleaning and wrangling, and aggregating and summarizing data into useful reports. You will also learn how to implement some practical and proven techniques to improve certain aspects of programming and administration in Apache Spark. By the end of the book, you will be able to build big data analytical solutions using the various PySpark offerings and also optimize them effectively.

Preface

Who this book is for

What this book covers

To get the most out of this book

Free Chapter

Installing Pyspark and Setting up Your Development Environment

Installing Pyspark and Setting up Your Development Environment

An overview of PySpark

Setting up Spark on Windows and PySpark

Core concepts in Spark and PySpark

Getting Your Big Data into the Spark Environment Using RDDs

Getting Your Big Data into the Spark Environment Using RDDs

Loading data on to Spark RDDs

Parallelization with Spark RDDs

Basics of RDD operation

Big Data Cleaning and Wrangling with Spark Notebooks

Big Data Cleaning and Wrangling with Spark Notebooks

Using Spark Notebooks for quick iteration of ideas

Sampling/filtering RDDs to pick out relevant data points

Splitting datasets and creating some new combinations

Aggregating and Summarizing Data into Useful Reports

Aggregating and Summarizing Data into Useful Reports

Calculating averages with map and reduce

Faster average computations with aggregate

Pivot tabling with key-value paired data points

Powerful Exploratory Data Analysis with MLlib

Powerful Exploratory Data Analysis with MLlib

Computing summary statistics with MLlib

Using Pearson and Spearman correlations to discover correlations

Testing our hypotheses on large datasets

Putting Structure on Your Big Data with SparkSQL

Putting Structure on Your Big Data with SparkSQL

Manipulating DataFrames with Spark SQL schemas

Using Spark DSL to build queries

Transformations and Actions

Transformations and Actions

Using Spark transformations to defer computations to a later time

Avoiding transformations

Using the reduce and reduceByKey methods to calculate the results

Performing actions that trigger computations

Reusing the same rdd for different actions

Immutable Design

Immutable Design

Delving into the Spark RDD's parent/child chain

Using RDD in an immutable way

Using DataFrame operations to transform

Immutability in the highly concurrent environment

Using the Dataset API in an immutable way

Avoiding Shuffle and Reducing Operational Expenses

Avoiding Shuffle and Reducing Operational Expenses

Detecting a shuffle in a process

Testing operations that cause a shuffle in Apache Spark

Changing the design of jobs with wide dependencies

Using keyBy() operations to reduce shuffle

Using a custom partitioner to reduce shuffle

Saving Data in the Correct Format

Saving Data in the Correct Format

Saving data in plain text format

Leveraging JSON as a data format

Tabular formats – CSV

Using Avro with Spark

Columnar formats – Parquet

Working with the Spark Key/Value API

Working with the Spark Key/Value API

Available actions on key/value pairs

Using aggregateByKey instead of groupBy()

Actions on key/value pairs

Available partitioners on key/value data

Implementing a custom partitioner

Testing Apache Spark Jobs

Testing Apache Spark Jobs

Separating logic from Spark engine-unit testing

Integration testing using SparkSession

Mocking data sources using partial functions

Using ScalaCheck for property-based testing

Testing in different versions of Spark

Leveraging the Spark GraphX API

Leveraging the Spark GraphX API

Creating a graph from a data source

Using the Vertex API

Using the Edge API

Calculating the degree of the vertex

Calculating PageRank

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Summary

In this chapter, we learned summary statistics and computing the summary statistics with MLlib. We also learned about Pearson and Spearman correlations, and how we can discover these correlations in our datasets using PySpark. Finally, we learned one particular way of performing hypothesis testing, which is called the Pearson chi-square test. We then used PySpark's hypothesis-testing functions to test our hypotheses on large datasets.

In the next chapter, we're going to look at putting the structure on our big data with Spark SQL.