Book Image

Optimizing Databricks Workloads

By : Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi
Book Image

Optimizing Databricks Workloads

By: Anirudh Kala, Anshul Bhatnagar, Sarthak Sarbahi

Overview of this book

Databricks is an industry-leading, cloud-based platform for data analytics, data science, and data engineering supporting thousands of organizations across the world in their data journey. It is a fast, easy, and collaborative Apache Spark-based big data analytics platform for data science and data engineering in the cloud. In Optimizing Databricks Workloads, you will get started with a brief introduction to Azure Databricks and quickly begin to understand the important optimization techniques. The book covers how to select the optimal Spark cluster configuration for running big data processing and workloads in Databricks, some very useful optimization techniques for Spark DataFrames, best practices for optimizing Delta Lake, and techniques to optimize Spark jobs through Spark core. It contains an opportunity to learn about some of the real-world scenarios where optimizing workloads in Databricks has helped organizations increase performance and save costs across various domains. By the end of this book, you will be prepared with the necessary toolkit to speed up your Spark jobs and process your data more efficiently.

Related Content you might be interested in

Current Title:

Small book image
Optimizing Databricks Workloads
Anirudh Kala | Anshul Bhatnagar | Sarthak Sarbahi
Dec, 2021 | 230 pages
Small book image
Azure Databricks Cookbook
Phani Raj | Vinod Jaiswal
Sep, 2021 | 452 pages
Small book image
Distributed Data Systems with Azure Databricks
Alan Bernardo Palacio
May, 2021 | 414 pages
Small book image
Essential PySpark for Scalable Data Analytics
Sreeram Nudurupati
Oct, 2021 | 322 pages
Table of Contents (13 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
1
Section 1: Introduction to Azure Databricks
Section 1: Introduction to Azure Databricks
Free Chapter
2
Chapter 1: Discovering Databricks
Chapter 1: Discovering Databricks
Technical requirements
Introducing Spark fundamentals
Introducing Databricks
Learning about Delta Lake
Summary
3
Chapter 2: Batch and Real-Time Processing in Databricks
Chapter 2: Batch and Real-Time Processing in Databricks
Technical requirements
Differentiating batch versus real-time processing
Mounting Azure Data Lake in Databricks
Working with batch processing
Batch ETL process demo
Learning Structured Streaming in Azure Databricks
Summary
4
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Chapter 3: Learning about Machine Learning and Graph Processing in Databricks
Technical requirements
Learning about ML components in Databricks
Practicing ML in Databricks
Learning about MLflow
Learning about graph analysis in Databricks
Summary
5
Section 2: Optimization Techniques
Section 2: Optimization Techniques
6
Chapter 4: Managing Spark Clusters
Chapter 4: Managing Spark Clusters
Technical requirements
Designing Spark clusters
Learning about Databricks managed resource groups
Learning about Databricks Pools
Using spot instances
Following the Spark UI
Summary
7
Chapter 5: Big Data Analytics
Chapter 5: Big Data Analytics
Technical requirements
Understanding the collect() method
Understanding the use of inferSchema
Learning to differentiate CSV and Parquet
Learning to differentiate Pandas and Koalas
Understanding built-in Spark functions
Learning column predicate pushdown
Learning partitioning strategies in Spark
Understanding Spark SQL optimizations
Understanding bucketing in Spark
Summary
8
Chapter 6: Databricks Delta Lake
Chapter 6: Databricks Delta Lake
Technical requirements
Working with the OPTIMIZE and ZORDER commands
Using Auto Optimize
Learning about delta caching
Learning about dynamic partition pruning
Understanding bloom filter indexing
Summary
9
Chapter 7: Spark Core
Chapter 7: Spark Core
Technical requirements
Learning about broadcast joins
Learning about Apache Arrow in Pandas
Understanding shuffle partitions
Understanding caching in Spark
Learning about AQE
Summary
10
Section 3: Real-World Scenarios
Section 3: Real-World Scenarios
11
Chapter 8: Case Studies
Chapter 8: Case Studies
Learning case studies from the manufacturing industry
Learning case studies from the media and entertainment industry
Learning case studies from the retail and FMCG industry
Learning case studies from the pharmaceutical industry
Learning case studies from the e-commerce industry
Learning case studies from the logistics and supply chain industry
Summary
Why subscribe?
12
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

What this book covers

Chapter 1, Discovering Databricks, will help you learn the fundamentals of Spark and all the different features of the Databricks platform and workspace.

Chapter 2, Batch and Real-Time Processing in Databricks, will help you learn about the SQL/DataFrame API for processing batch loads and the Streaming API for processing real-time data streams.

Chapter 3, Learning about Machine Learning and Graph Processing in Databricks, will help you get an introduction to machine learning on big data using SparkML and the Spark Graph Processing API.

Chapter 4, Managing Spark Clusters, will help you learn to select the optimal Spark cluster configurations for running big data processing and workloads in Databricks.

Chapter 5, Big Data Analytics, will help you learn some very useful optimization techniques for Spark DataFrames.

Chapter 6, Databricks Delta Lake, will help you learn the best practices for optimizing Delta Lake workloads in Databricks.

Chapter 7, Spark Core, will help you learn techniques to optimize Spark jobs through a true understanding of Spark core.

Chapter 8, Case Studies, will look at a number of real-world case studies where Databricks played a crucial role in an organization's data journey. We will also learn how Databricks is helping drive innovation across various industries around the world.

Previous Section
Next Section