Book Image

Optimizing Databricks Workloads

By : Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi
Book Image

Optimizing Databricks Workloads

By: Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi

Overview of this book

Databricks is an industry-leading, cloud-based platform for data analytics, data science, and data engineering supporting thousands of organizations across the world in their data journey. It is a fast, easy, and collaborative Apache Spark-based big data analytics platform for data science and data engineering in the cloud. In Optimizing Databricks Workloads, you will get started with a brief introduction to Azure Databricks and quickly begin to understand the important optimization techniques. The book covers how to select the optimal Spark cluster configuration for running big data processing and workloads in Databricks, some very useful optimization techniques for Spark DataFrames, best practices for optimizing Delta Lake, and techniques to optimize Spark jobs through Spark core. It contains an opportunity to learn about some of the real-world scenarios where optimizing workloads in Databricks has helped organizations increase performance and save costs across various domains. By the end of this book, you will be prepared with the necessary toolkit to speed up your Spark jobs and process your data more efficiently.

Related Content you might be interested in

Current Title:

Small book image
Optimizing Databricks Workloads
Anirudh Kala | Anshul Bhatnagar | Sarthak Sarbahi
Dec, 2021 | 230 pages
Small book image
Azure Databricks Cookbook
Phani Raj | Vinod Jaiswal
Sep, 2021 | 452 pages
Small book image
Distributed Data Systems with Azure Databricks
Alan Bernardo Palacio
May, 2021 | 414 pages
Small book image
Essential PySpark for Scalable Data Analytics
Sreeram Nudurupati
Oct, 2021 | 322 pages
Table of Contents (13 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
1
Section 1: Introduction to Azure Databricks
Section 1: Introduction to Azure Databricks
Free Chapter
2
Chapter 1: Discovering Databricks
Chapter 1: Discovering Databricks
Technical requirements
Introducing Spark fundamentals
Introducing Databricks
Learning about Delta Lake
Summary
3
Chapter 2: Batch and Real-Time Processing in Databricks
Chapter 2: Batch and Real-Time Processing in Databricks
Technical requirements
Differentiating batch versus real-time processing
Mounting Azure Data Lake in Databricks
Working with batch processing
Batch ETL process demo
Learning Structured Streaming in Azure Databricks
Summary
4
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Technical requirements
Learning about ML components in Databricks
Practicing ML in Databricks
Learning about MLflow
Learning about graph analysis in Databricks
Summary
5
Section 2: Optimization Techniques
Section 2: Optimization Techniques
6
Chapter 4: Managing Spark Clusters
Chapter 4: Managing Spark Clusters
Technical requirements
Designing Spark clusters
Learning about Databricks managed resource groups
Learning about Databricks Pools
Using spot instances
Following the Spark UI
Summary
7
Chapter 5: Big Data Analytics
Chapter 5: Big Data Analytics
Technical requirements
Understanding the collect() method
Understanding the use of inferSchema
Learning to differentiate CSV and Parquet
Learning to differentiate Pandas and Koalas
Understanding built-in Spark functions
Learning column predicate pushdown
Learning partitioning strategies in Spark
Understanding Spark SQL optimizations
Understanding bucketing in Spark
Summary
8
Chapter 6: Databricks Delta Lake
Chapter 6: Databricks Delta Lake
Technical requirements
Working with the OPTIMIZE and ZORDER commands
Using Auto Optimize
Learning about delta caching
Learning about dynamic partition pruning
Understanding bloom filter indexing
Summary
9
Chapter 7: Spark Core
Chapter 7: Spark Core
Technical requirements
Learning about broadcast joins
Learning about Apache Arrow in Pandas
Understanding shuffle partitions
Understanding caching in Spark
Learning about AQE
Summary
10
Section 3: Real-World Scenarios
Section 3: Real-World Scenarios
11
Chapter 8: Case Studies
Chapter 8: Case Studies
Learning case studies from the manufacturing industry
Learning case studies from the media and entertainment industry
Learning case studies from the retail and FMCG industry
Learning case studies from the pharmaceutical industry
Learning case studies from the e-commerce industry
Learning case studies from the logistics and supply chain industry
Summary
Why subscribe?
12
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Understanding shuffle partitions

Every time Spark performs a wide transformation or aggregations, shuffling of data across the nodes occurs. And during these shuffle operations, Spark, by default, changes the partitions of the DataFrame. For example, when creating a DataFrame, it may have 10 partitions, but as soon as the shuffle occurs, Spark may change the partitions of the DataFrame to 200. These are what we call the shuffle partitions.

This is a default behavior in Spark, but it can be altered to improve the performance of Spark jobs. We can also confirm the default behavior by running the following line of code:

spark.conf.get('spark.sql.shuffle.partitions')

This returns the output of 200. This means that Spark will change the shuffle partitions to 200 by default. To alter this configuration, we can run the following code, which configures the shuffle partitions to 8:

spark.conf.set('spark.sql.shuffle.partitions',8)

You may be wondering why we...

Previous Section
End of Section 5
Next Section