Book Image

Optimizing Databricks Workloads

By : Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi
Book Image

Optimizing Databricks Workloads

By: Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi

Overview of this book

Databricks is an industry-leading, cloud-based platform for data analytics, data science, and data engineering supporting thousands of organizations across the world in their data journey. It is a fast, easy, and collaborative Apache Spark-based big data analytics platform for data science and data engineering in the cloud. In Optimizing Databricks Workloads, you will get started with a brief introduction to Azure Databricks and quickly begin to understand the important optimization techniques. The book covers how to select the optimal Spark cluster configuration for running big data processing and workloads in Databricks, some very useful optimization techniques for Spark DataFrames, best practices for optimizing Delta Lake, and techniques to optimize Spark jobs through Spark core. It contains an opportunity to learn about some of the real-world scenarios where optimizing workloads in Databricks has helped organizations increase performance and save costs across various domains. By the end of this book, you will be prepared with the necessary toolkit to speed up your Spark jobs and process your data more efficiently.

Related Content you might be interested in

Current Title:

Small book image
Optimizing Databricks Workloads
Anirudh Kala | Anshul Bhatnagar | Sarthak Sarbahi
Dec, 2021 | 230 pages
Small book image
Azure Databricks Cookbook
Phani Raj | Vinod Jaiswal
Sep, 2021 | 452 pages
Small book image
Distributed Data Systems with Azure Databricks
Alan Bernardo Palacio
May, 2021 | 414 pages
Small book image
Essential PySpark for Scalable Data Analytics
Sreeram Nudurupati
Oct, 2021 | 322 pages
Table of Contents (13 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
1
Section 1: Introduction to Azure Databricks
Section 1: Introduction to Azure Databricks
Free Chapter
2
Chapter 1: Discovering Databricks
Chapter 1: Discovering Databricks
Technical requirements
Introducing Spark fundamentals
Introducing Databricks
Learning about Delta Lake
Summary
3
Chapter 2: Batch and Real-Time Processing in Databricks
Chapter 2: Batch and Real-Time Processing in Databricks
Technical requirements
Differentiating batch versus real-time processing
Mounting Azure Data Lake in Databricks
Working with batch processing
Batch ETL process demo
Learning Structured Streaming in Azure Databricks
Summary
4
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Technical requirements
Learning about ML components in Databricks
Practicing ML in Databricks
Learning about MLflow
Learning about graph analysis in Databricks
Summary
5
Section 2: Optimization Techniques
Section 2: Optimization Techniques
6
Chapter 4: Managing Spark Clusters
Chapter 4: Managing Spark Clusters
Technical requirements
Designing Spark clusters
Learning about Databricks managed resource groups
Learning about Databricks Pools
Using spot instances
Following the Spark UI
Summary
7
Chapter 5: Big Data Analytics
Chapter 5: Big Data Analytics
Technical requirements
Understanding the collect() method
Understanding the use of inferSchema
Learning to differentiate CSV and Parquet
Learning to differentiate Pandas and Koalas
Understanding built-in Spark functions
Learning column predicate pushdown
Learning partitioning strategies in Spark
Understanding Spark SQL optimizations
Understanding bucketing in Spark
Summary
8
Chapter 6: Databricks Delta Lake
Chapter 6: Databricks Delta Lake
Technical requirements
Working with the OPTIMIZE and ZORDER commands
Using Auto Optimize
Learning about delta caching
Learning about dynamic partition pruning
Understanding bloom filter indexing
Summary
9
Chapter 7: Spark Core
Chapter 7: Spark Core
Technical requirements
Learning about broadcast joins
Learning about Apache Arrow in Pandas
Understanding shuffle partitions
Understanding caching in Spark
Learning about AQE
Summary
10
Section 3: Real-World Scenarios
Section 3: Real-World Scenarios
11
Chapter 8: Case Studies
Chapter 8: Case Studies
Learning case studies from the manufacturing industry
Learning case studies from the media and entertainment industry
Learning case studies from the retail and FMCG industry
Learning case studies from the pharmaceutical industry
Learning case studies from the e-commerce industry
Learning case studies from the logistics and supply chain industry
Summary
Why subscribe?
12
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Using Auto Optimize

Auto Optimize is a feature that helps us automatically compact small files while an individual writes to a delta table. Unlike bin-packing, we do not need to run a command every time Auto Optimize is executed. It consists of two components:

  • Optimized Writes: Databricks dynamically optimizes Spark partition sizes to write 128 MB chunks of table partitions.
  • Auto Compaction: Here, Databricks runs an optimized job when the data writing process has been completed and compacts small files. It tries to coalesce small files into 128 MB files. This works on data that has the greatest number of small files.

Next, we will learn about the Spark configurations for Auto Optimize and go through a worked-out example to understand how it works:

  1. To enable Auto Optimize for all new tables, we need to run the following Spark configuration code:
    %sql
set spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite = true;
set spark.databricks.delta...
Previous Section
End of Section 4
Next Section