Hands-On Big Data Analytics with PySpark

By : Rudy Lai, Bartłomiej Potaczek

Hands-On Big Data Analytics with PySpark

By: Rudy Lai, Bartłomiej Potaczek

Overview of this book

Apache Spark is an open source parallel-processing framework that has been around for quite some time now. One of the many uses of Apache Spark is for data analytics applications across clustered computers. In this book, you will not only learn how to use Spark and the Python API to create high-performance analytics with big data, but also discover techniques for testing, immunizing, and parallelizing Spark jobs. You will learn how to source data from all popular data hosting platforms, including HDFS, Hive, JSON, and S3, and deal with large datasets with PySpark to gain practical big data experience. This book will help you work on prototypes on local machines and subsequently go on to handle messy data in production and at scale. This book covers installing and setting up PySpark, RDD operations, big data cleaning and wrangling, and aggregating and summarizing data into useful reports. You will also learn how to implement some practical and proven techniques to improve certain aspects of programming and administration in Apache Spark. By the end of the book, you will be able to build big data analytical solutions using the various PySpark offerings and also optimize them effectively.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Free Chapter

Installing Pyspark and Setting up Your Development Environment

An overview of PySpark

Setting up Spark on Windows and PySpark

Core concepts in Spark and PySpark

Summary

Getting Your Big Data into the Spark Environment Using RDDs

Loading data on to Spark RDDs

Parallelization with Spark RDDs

Basics of RDD operation

Summary

Big Data Cleaning and Wrangling with Spark Notebooks

Using Spark Notebooks for quick iteration of ideas

Sampling/filtering RDDs to pick out relevant data points

Splitting datasets and creating some new combinations

Summary

Aggregating and Summarizing Data into Useful Reports

Calculating averages with map and reduce

Faster average computations with aggregate

Pivot tabling with key-value paired data points

Summary

Powerful Exploratory Data Analysis with MLlib

Computing summary statistics with MLlib

Using Pearson and Spearman correlations to discover correlations

Testing our hypotheses on large datasets

Summary

Putting Structure on Your Big Data with SparkSQL

Manipulating DataFrames with Spark SQL schemas

Using Spark DSL to build queries

Summary

Transformations and Actions

Using Spark transformations to defer computations to a later time

Avoiding transformations

Using the reduce and reduceByKey methods to calculate the results

Performing actions that trigger computations

Reusing the same rdd for different actions

Summary

Immutable Design

Delving into the Spark RDD's parent/child chain

Using RDD in an immutable way

Using DataFrame operations to transform

Immutability in the highly concurrent environment

Using the Dataset API in an immutable way

Summary

Avoiding Shuffle and Reducing Operational Expenses

Detecting a shuffle in a process

Testing operations that cause a shuffle in Apache Spark

Changing the design of jobs with wide dependencies

Using keyBy() operations to reduce shuffle

Using a custom partitioner to reduce shuffle

Summary

Saving Data in the Correct Format

Saving data in plain text format

Leveraging JSON as a data format

Tabular formats – CSV

Using Avro with Spark

Columnar formats – Parquet

Summary

Working with the Spark Key/Value API

Available actions on key/value pairs

Using aggregateByKey instead of groupBy()

Actions on key/value pairs

Available partitioners on key/value data

Implementing a custom partitioner

Summary

Testing Apache Spark Jobs

Separating logic from Spark engine-unit testing

Integration testing using SparkSession

Mocking data sources using partial functions

Using ScalaCheck for property-based testing

Testing in different versions of Spark

Summary

Leveraging the Spark GraphX API

Creating a graph from a data source

Using the Vertex API

Using the Edge API

Calculating the degree of the vertex

Calculating PageRank

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Hands-On Big Data Analytics with PySpark

By : Rudy Lai, Bartłomiej Potaczek

Hands-On Big Data Analytics with PySpark

By: Rudy Lai, Bartłomiej Potaczek

Overview of this book

Related Content you might be interested in

Current Title:

Hands-On Big Data Analytics with PySpark

Apache Spark Quick Start Guide

Scala and Spark for Big Data Analytics

Apache Spark 2.x for Java Developers