Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Azure Databricks Cookbook
  • Table Of Contents Toc
Azure Databricks Cookbook

Azure Databricks Cookbook

By : Raj, Jaiswal
4.4 (28)
Buy this Book
close
close
Azure Databricks Cookbook

Azure Databricks Cookbook

4.4 (28)
By: Raj, Jaiswal
Buy this Book

Overview of this book

Azure Databricks is a unified collaborative platform for performing scalable analytics in an interactive environment. The Azure Databricks Cookbook provides recipes to get hands-on with the analytics process, including ingesting data from various batch and streaming sources and building a modern data warehouse. The book starts by teaching you how to create an Azure Databricks instance within the Azure portal, Azure CLI, and ARM templates. You’ll work through clusters in Databricks and explore recipes for ingesting data from sources, including files, databases, and streaming sources such as Apache Kafka and EventHub. The book will help you explore all the features supported by Azure Databricks for building powerful end-to-end data pipelines. You'll also find out how to build a modern data warehouse by using Delta tables and Azure Synapse Analytics. Later, you’ll learn how to write ad hoc queries and extract meaningful insights from the data lake by creating visualizations and dashboards with Databricks SQL. Finally, you'll deploy and productionize a data pipeline as well as deploy notebooks and Azure Databricks service using continuous integration and continuous delivery (CI/CD). By the end of this Azure book, you'll be able to use Azure Databricks to streamline different processes involved in building data-driven apps.
Table of Contents (12 chapters)
close
close
close
Preface
Icon
Preface
Icon
Who this book is for
Icon
What this book covers
Icon
To get the most out of this book
Icon
Download the color images
Icon
Conventions used
Icon
Sections
Icon
Get in touch
Icon
Share Your Thoughts
1
Chapter 1: Creating an Azure Databricks Service
Icon
Chapter 1: Creating an Azure Databricks Service
Icon
Technical requirements
Icon
Creating a Databricks workspace in the Azure portal
Icon
Creating a Databricks service using the Azure CLI (command-line interface)
Icon
Creating a Databricks service using Azure Resource Manager (ARM) templates
Icon
Adding users and groups to the workspace
Icon
Creating a cluster from the user interface (UI)
Icon
Getting started with notebooks and jobs in Azure Databricks
Icon
Authenticating to Databricks using a PAT
Lock Free Chapter
2
Chapter 2: Reading and Writing Data from and to Various Azure Services and File Formats
chevron up
Icon
Chapter 2: Reading and Writing Data from and to Various Azure Services and File Formats
Icon
Technical requirements
Icon
Mounting ADLS Gen2 and Azure Blob storage to Azure DBFS
Icon
Reading and writing data from and to Azure Blob storage
Icon
Reading and writing data from and to ADLS Gen2
Icon
Reading and writing data from and to an Azure SQL database using native connectors
Icon
Reading and writing data from and to Azure Synapse SQL (dedicated SQL pool) using native connectors
Icon
Reading and writing data from and to Azure Cosmos DB
Icon
Reading and writing data from and to CSV and Parquet
Icon
Reading and writing data from and to JSON, including nested JSON
3
Chapter 3: Understanding Spark Query Execution
Icon
Chapter 3: Understanding Spark Query Execution
Icon
Technical requirements
Icon
Introduction to jobs, stages, and tasks
Icon
Checking the execution details of all the executed Spark queries via the Spark UI
Icon
Deep diving into schema inference
Icon
Looking into the query execution plan
Icon
How joins work in Spark
Icon
Learning about input partitions
Icon
Learning about output partitions
Icon
Learning about shuffle partitions
Icon
Storage benefits of different file types
4
Chapter 4: Working with Streaming Data
Icon
Chapter 4: Working with Streaming Data
Icon
Technical requirements
Icon
Reading streaming data from Apache Kafka
Icon
Reading streaming data from Azure Event Hubs
Icon
Reading data from Event Hubs for Kafka
Icon
Streaming data from log files
Icon
Understanding trigger options
Icon
Understanding window aggregation on streaming data
Icon
Understanding offsets and checkpoints
5
Chapter 5: Integrating with Azure Key Vault, App Configuration, and Log Analytics
Icon
Chapter 5: Integrating with Azure Key Vault, App Configuration, and Log Analytics
Icon
Technical requirements
Icon
Creating an Azure Key Vault to store secrets using the UI
Icon
Creating an Azure Key Vault to store secrets using ARM templates
Icon
Using Azure Key Vault secrets in Azure Databricks
Icon
Creating an App Configuration resource
Icon
Using App Configuration in an Azure Databricks notebook
Icon
Creating a Log Analytics workspace
Icon
Integrating a Log Analytics workspace with Azure Databricks
6
Chapter 6: Exploring Delta Lake in Azure Databricks
Icon
Chapter 6: Exploring Delta Lake in Azure Databricks
Icon
Technical requirements
Icon
Delta table operations – create, read, and write
Icon
Streaming reads and writes to Delta tables
Icon
Delta table data format
Icon
Handling concurrency
Icon
Delta table performance optimization
Icon
Constraints in Delta tables
Icon
Versioning in Delta tables
7
Chapter 7: Implementing Near-Real-Time Analytics and Building a Modern Data Warehouse
Icon
Chapter 7: Implementing Near-Real-Time Analytics and Building a Modern Data Warehouse
Icon
Technical requirements
Icon
Understanding the scenario for an end-to-end (E2E) solution
Icon
Creating required Azure resources for the E2E demonstration
Icon
Simulating a workload for streaming data
Icon
Processing streaming and batch data using Structured Streaming
Icon
Understanding the various stages of transforming data
Icon
Loading the transformed data into Azure Cosmos DB and a Synapse dedicated pool
Icon
Creating a visualization and dashboard in a notebook for near-real-time analytics
Icon
Creating a visualization in Power BI for near-real-time analytics
Icon
Using Azure Data Factory (ADF) to orchestrate the E2E pipeline
8
Chapter 8: Databricks SQL
Icon
Chapter 8: Databricks SQL
Icon
Technical requirements
Icon
How to create a user in Databricks SQL
Icon
Creating SQL endpoints
Icon
Granting access to objects to the user
Icon
Running SQL queries in Databricks SQL
Icon
Using query parameters and filters
Icon
Introduction to visualizations in Databricks SQL
Icon
Creating dashboards in Databricks SQL
Icon
Connecting Power BI to Databricks SQL
9
Chapter 9: DevOps Integrations and Implementing CI/CD for Azure Databricks
Icon
Chapter 9: DevOps Integrations and Implementing CI/CD for Azure Databricks
Icon
Technical requirements
Icon
How to integrate Azure DevOps with an Azure Databricks notebook
Icon
Using GitHub for Azure Databricks notebook version control
Icon
Understanding the CI/CD process for Azure Databricks
Icon
How to set up an Azure DevOps pipeline for deploying notebooks
Icon
Deploying notebooks to multiple environments
Icon
Enabling CI/CD in an Azure DevOps build and release pipeline
Icon
Deploying an Azure Databricks service using an Azure DevOps release pipeline
10
Chapter 10: Understanding Security and Monitoring in Azure Databricks
Icon
Chapter 10: Understanding Security and Monitoring in Azure Databricks
Icon
Technical requirements
Icon
Understanding and creating RBAC in Azure for ADLS Gen-2
Icon
Creating ACLs using Storage Explorer and PowerShell
Icon
How to configure credential passthrough
Icon
How to restrict data access to users using RBAC
Icon
How to restrict data access to users using ACLs
Icon
Deploying Azure Databricks in a VNet and accessing a secure storage account
Icon
Using Ganglia reports for cluster health
Icon
Cluster access control
Icon
Why subscribe?
11
Other Books You May Enjoy
Icon
Other Books You May Enjoy
Icon
Packt is searching for authors like you
Icon
Share Your Thoughts

Chapter 2: Reading and Writing Data from and to Various Azure Services and File Formats

Azure Databricks provides options for data engineers, data scientists, and data analysts to read and write data from and to various sources such as different file formats, databases, NoSQL databases, Azure Storage, and so on. Users get a lot of flexibility in ingesting and storing data in various formats as per business requirements, using Databricks. It also provides libraries to ingest data from streaming systems such as Events Hub and Kafka.

In this chapter, we will learn how we can read data from different file formats, such as comma-separated values (CSV), Parquet, and JavaScript Object Notation (JSON), and how to use native connectors to read and write data from and to Azure SQL Database and Azure Synapse Analytics. We will also learn how to read and store data in Azure Cosmos DB.

By the end of this chapter, you will have built a foundation for reading data from various sources that are required to work on the end-to-end (E2E) scenarios of a data ingestion pipeline. You will learn how and when to use JavaScript Database Connectivity (JDBC) drivers and Apache Spark connectors to ingest into the Azure SQL Database.

We're going to cover the following recipes in this chapter:

  • Mounting Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob storage to Azure Databricks File System (DBFS)
  • Reading and writing data from and to Azure Blob storage
  • Reading and writing data from and to ADLS Gen2
  • Reading and writing data from and to an Azure SQL database using native connectors
  • Reading and writing data from and to Azure Synapse Dedicated Structured Query Language (SQL) Pool using native connectors
  • Reading and writing data from and to the Azure Cosmos DB
  • Reading and writing data from and to CSV and Parquet
  • Reading and writing data from and to JSON, including nested JSON
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Azure Databricks Cookbook
End of Section 2
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options
font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon