Book Image

Essential PySpark for Scalable Data Analytics

By : Sreeram Nudurupati
Book Image

Essential PySpark for Scalable Data Analytics

By: Sreeram Nudurupati

Overview of this book

Apache Spark is a unified data analytics engine designed to process huge volumes of data quickly and efficiently. PySpark is Apache Spark's Python language API, which offers Python developers an easy-to-use scalable data analytics framework. Essential PySpark for Scalable Data Analytics starts by exploring the distributed computing paradigm and provides a high-level overview of Apache Spark. You'll begin your analytics journey with the data engineering process, learning how to perform data ingestion, cleansing, and integration at scale. This book helps you build real-time analytics pipelines that help you gain insights faster. You'll then discover methods for building cloud-based data lakes, and explore Delta Lake, which brings reliability to data lakes. The book also covers Data Lakehouse, an emerging paradigm, which combines the structure and performance of a data warehouse with the scalability of cloud-based data lakes. Later, you'll perform scalable data science and machine learning tasks using PySpark, such as data preparation, feature engineering, and model training and productionization. Finally, you'll learn ways to scale out standard Python ML libraries along with a new pandas API on top of PySpark called Koalas. By the end of this PySpark book, you'll be able to harness the power of PySpark to solve business problems.

Related Content you might be interested in

Current Title:

Small book image
Essential PySpark for Scalable Data Analytics
Sreeram Nudurupati
Oct, 2021 | 322 pages
Small book image
Optimizing Databricks Workloads
Sarthak Sarbahi | Anirudh Kala | Anshul Bhatnagar
Dec, 2021 | 230 pages
Small book image
Simplifying Data Engineering and Analytics with Delta
Anindita Mahapatra
Jul, 2022 | 334 pages
Small book image
Practical Machine Learning on Databricks
Debu Sinha
Nov, 2023 | 244 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share your thoughts
1
Section 1: Data Engineering
Section 1: Data Engineering
Free Chapter
2
Chapter 1: Distributed Computing Primer
Chapter 1: Distributed Computing Primer
Technical requirements
Distributed Computing
Distributed Computing with Apache Spark
Big data processing with Spark SQL and DataFrames
Summary
3
Chapter 2: Data Ingestion
Chapter 2: Data Ingestion
Technical requirements
Introduction to Enterprise Decision Support Systems
Ingesting data from data sources
Ingesting data into data sinks
Using file formats for data storage in data lakes
Building data ingestion pipelines in batch and real time
Unifying batch and real time using Lambda Architecture
Summary
4
Chapter 3: Data Cleansing and Integration
Chapter 3: Data Cleansing and Integration
Technical requirements
Transforming raw data into enriched meaningful data
Building analytical data stores using cloud data lakes
Consolidating data using data integration
Making raw data analytics-ready using data cleansing
Summary
5
Chapter 4: Real-Time Data Analytics
Chapter 4: Real-Time Data Analytics
Technical requirements
Real-time analytics systems architecture
Stream processing engines
Real-time analytics industry use cases
Simplifying the Lambda Architecture using Delta Lake
Change Data Capture
Handling late-arriving data
Multi-hop pipelines
Summary
6
Section 2: Data Science
Section 2: Data Science
7
Chapter 5: Scalable Machine Learning with PySpark
Chapter 5: Scalable Machine Learning with PySpark
Technical requirements
ML overview
Scaling out machine learning
Data wrangling with Apache Spark and MLlib
Summary
8
Chapter 6: Feature Engineering – Extraction, Transformation, and Selection
Chapter 6: Feature Engineering – Extraction, Transformation, and Selection
Technical requirements
The machine learning process
Feature extraction
Feature transformation
Feature selection
Feature store as a central feature repository
Delta Lake as an offline feature store
Summary
9
Chapter 7: Supervised Machine Learning
Chapter 7: Supervised Machine Learning
Technical requirements
Introduction to supervised machine learning
Regression
Classification
Tree ensembles
Real-world supervised learning applications
Summary
10
Chapter 8: Unsupervised Machine Learning
Chapter 8: Unsupervised Machine Learning
Technical requirements
Introduction to unsupervised machine learning
Clustering using machine learning
Building association rules using machine learning
Real-world applications of unsupervised learning
Summary
11
Chapter 9: Machine Learning Life Cycle Management
Chapter 9: Machine Learning Life Cycle Management
Technical requirements
Introduction to the ML life cycle
Tracking experiments with MLflow
Tracking model versions using MLflow Model Registry
Model serving and inferencing
Continuous delivery for ML
Summary
12
Chapter 10: Scaling Out Single-Node Machine Learning Using PySpark
Chapter 10: Scaling Out Single-Node Machine Learning Using PySpark
Technical requirements
Scaling out EDA
Scaling out model inferencing
Model training using embarrassingly parallel computing
Upgrading pandas to PySpark using Koalas
Summary
13
Section 3: Data Analysis
Section 3: Data Analysis
14
Chapter 11: Data Visualization with PySpark
Chapter 11: Data Visualization with PySpark
Technical requirements
Importance of data visualization
Techniques for visualizing data using PySpark
Considerations for PySpark to pandas conversion
Summary
15
Chapter 12: Spark SQL Primer
Chapter 12: Spark SQL Primer
Technical requirements
Introduction to SQL
Introduction to Spark SQL
Spark SQL language reference
Optimizing Spark SQL performance
Summary
16
Chapter 13: Integrating External Tools with Spark SQL
Chapter 13: Integrating External Tools with Spark SQL
Technical requirements
Apache Spark as a distributed SQL engine
Spark connectivity to SQL analysis tools
Spark connectivity to BI tools
Connecting Python applications to Spark SQL using Pyodbc
Summary
17
Chapter 14: The Data Lakehouse
Chapter 14: The Data Lakehouse
Moving from BI to AI
The data lakehouse paradigm
Advantages of data lakehouses
Summary
Why subscribe?
18
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share your thoughts

Chapter 1: Distributed Computing Primer

This chapter introduces you to the Distributed Computing paradigm and shows you how Distributed Computing can help you to easily process very large amounts of data. You will learn about the concept of Data Parallel Processing using the MapReduce paradigm and, finally, learn how Data Parallel Processing can be made more efficient by using an in-memory, unified data processing engine such as Apache Spark.

Then, you will dive deeper into the architecture and components of Apache Spark along with code examples. Finally, you will get an overview of what's new with the latest 3.0 release of Apache Spark.

In this chapter, the key skills that you will acquire include an understanding of the basics of the Distributed Computing paradigm and a few different implementations of the Distributed Computing paradigm such as MapReduce and Apache Spark. You will learn about the fundamentals of Apache Spark along with its architecture and core components, such as the Driver, Executor, and Cluster Manager, and how they come together as a single unit to perform a Distributed Computing task. You will learn about Spark's Resilient Distributed Dataset (RDD) API along with higher-order functions and lambdas. You will also gain an understanding of the Spark SQL Engine and its DataFrame and SQL APIs. Additionally, you will implement working code examples. You will also learn about the various components of an Apache Spark data processing program, including transformations and actions, and you will learn about the concept of Lazy Evaluation.

In this chapter, we're going to cover the following main topics:

  • Introduction Distributed Computing
  • Distributed Computing with Apache Spark
  • Big data processing with Spark SQL and DataFrames
Previous Section
End of Section 1
Next Section