Book Image

Cloud Scale Analytics with Azure Data Services

By : Patrik Borosch
Book Image

Cloud Scale Analytics with Azure Data Services

By: Patrik Borosch

Overview of this book

Azure Data Lake, the modern data warehouse architecture, and related data services on Azure enable organizations to build their own customized analytical platform to fit any analytical requirements in terms of volume, speed, and quality. This book is your guide to learning all the features and capabilities of Azure data services for storing, processing, and analyzing data (structured, unstructured, and semi-structured) of any size. You will explore key techniques for ingesting and storing data and perform batch, streaming, and interactive analytics. The book also shows you how to overcome various challenges and complexities relating to productivity and scaling. Next, you will be able to develop and run massive data workloads to perform different actions. Using a cloud-based big data-modern data warehouse-analytics setup, you will also be able to build secure, scalable data estates for enterprises. Finally, you will not only learn how to develop a data warehouse but also understand how to create enterprise-grade security and auditing big data programs. By the end of this Azure book, you will have learned how to develop a powerful and efficient analytical platform to meet enterprise needs.

Related Content you might be interested in

Current Title:

Small book image
Cloud Scale Analytics with Azure Data Services
Patrik Borosch
Jul, 2021 | 520 pages
Small book image
Limitless Analytics with Azure Synapse
Prashant Kumar Mishra
Jun, 2021 | 392 pages
Small book image
Azure Synapse Analytics Cookbook
Gaurav Agarwal | Meenakshi Muralidharan
Apr, 2022 | 238 pages
Small book image
Azure Data Engineering Cookbook
Nagaraj Venkatesan | Ahmad Osama
Sep, 2022 | 608 pages
Table of Contents (20 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Share Your Thoughts
1
Section 1: Data Warehousing and Considerations Regarding Cloud Computing
Section 1: Data Warehousing and Considerations Regarding Cloud Computing
Free Chapter
2
Chapter 1: Balancing the Benefits of Data Lakes Over Data Warehouses
Chapter 1: Balancing the Benefits of Data Lakes Over Data Warehouses
Distinguishing between Data Warehouses and Data Lakes
Understanding the opportunities of modern cloud computing
Exploring the benefits of AI and ML
Answering the question
Summary
3
Chapter 2: Connecting Requirements and Technology
Chapter 2: Connecting Requirements and Technology
Formulating your requirements
Understanding basic architecture patterns
Finding the right Azure tool for the right purpose
Understanding Industry Data Models
Thinking about different sizes
Understanding the supporting services
Summary
Questions
4
Section 2: The Storage Layer
Section 2: The Storage Layer
5
Chapter 3: Understanding the Data Lake Storage Layer
Chapter 3: Understanding the Data Lake Storage Layer
Technical requirements
Setting up your Cloud Big Data Storage
Organizing your data lake
Implementing a data model in your Data Lake
Monitoring your storage account
Talking about backups
Implementing access control in your Data Lake
Setting the networking options
Discovering additional knowledge
Summary
Further reading
6
Chapter 4: Understanding Synapse SQL Pools and SQL Options
Chapter 4: Understanding Synapse SQL Pools and SQL Options
Uncovering MPP in the cloud – the power of 60
Provisioning a Synapse dedicated SQL pool
Talking about partitioning
Implementing workload management
Scaling the database
Loading data
Understanding other SQL options in Azure
Summary
Further reading
7
Section 3: Cloud-Scale Data Integration and Data Transformation
Section 3: Cloud-Scale Data Integration and Data Transformation
8
Chapter 5: Integrating Data into Your Modern Data Warehouse
Chapter 5: Integrating Data into Your Modern Data Warehouse
Technical requirements
Setting up Azure Data Factory
Examining the authoring environment
Using wizards
Adding data transformation logic
Understanding integration runtimes
Integrating with DevOps
Summary
Further reading
9
Chapter 6: Using Synapse Spark Pools
Chapter 6: Using Synapse Spark Pools
Technical requirements
Setting up a Synapse Spark pool
Examining the Synapse Spark architecture
Programming with Synapse Spark pools
Using additional libraries with your Spark pool
Handling security
Monitoring your Synapse Spark pools
Summary
Further reading
10
Chapter 7: Using Databricks Spark Clusters
Chapter 7: Using Databricks Spark Clusters
Technical requirements
Provisioning Databricks
Examining the Databricks workspace
Understanding the Databricks components
Setting up security
Monitoring Databricks
Summary
Further reading
11
Chapter 8: Streaming Data into Your MDWH
Chapter 8: Streaming Data into Your MDWH
Technical requirements
Provisioning ASA
Implementing an ASA job
Understanding ASA SQL
Using Structured Streaming with Spark
Security in your streaming solution
Monitoring your streaming solution
Summary
Further reading
12
Chapter 9: Integrating Azure Cognitive Services and Machine Learning
Chapter 9: Integrating Azure Cognitive Services and Machine Learning
Technical requirements
Understanding Azure Cognitive Services
Using Cognitive Services with your data
Examining Azure Machine Learning
Using Azure Machine Learning with your modern data warehouse
Summary
Further reading
13
Chapter 10: Loading the Presentation Layer
Chapter 10: Loading the Presentation Layer
Technical requirements
Understanding the loading strategy with Synapse-dedicated SQL pools
Loading data into Synapse-dedicated SQL pools
Using Synapse serverless SQL pools
Integrating data with Synapse Spark pools
Exchanging metadata between computes
Summary
Further reading
14
Section 4: Data Presentation, Dashboarding, and Distribution
Section 4: Data Presentation, Dashboarding, and Distribution
15
Chapter 11: Developing and Maintaining the Presentation Layer
Chapter 11: Developing and Maintaining the Presentation Layer
Developing with Synapse Studio
Backing up and DR in Azure Synapse
Monitoring your MDWH
Understanding security in your MDWH
Summary
Further reading
16
Chapter 12: Distributing Data
Chapter 12: Distributing Data
Technical requirements
Building data marts with Power BI
Creating data models with Azure Analysis Services
Distributing data using Azure Data Share
Summary
Further reading
17
Chapter 13: Introducing Industry Data Models
Chapter 13: Introducing Industry Data Models
Understanding Common Data Model
Examining and leveraging predefined entities
Discovering Azure Industry Data Workbench
Summary
Further reading
18
Chapter 14: Establishing Data Governance
Chapter 14: Establishing Data Governance
Technical requirements
Discovering Azure Purview
Classifying data
Integrating with Azure services
Using data lineage
Discovering Insights
Discovering more Purview
Summary
Further reading
Why subscribe?
19
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Integrating with DevOps

Finally, you want to take care that work that was developed not only gets versioned and saved in a reliable manner; you also want to be able to deploy it automatically to a test and later to your production environment. This can be achieved by integrating your Data Factory or Synapse workspace with Azure DevOps, for example, or with GitHub.

Let's examine the Azure DevOps integration. When we created our data factory in the first section of this chapter (Setting up Azure Data Factory), we skipped the DevOps integration at that moment. Now we are going to finish that task as well:

Note

The Synapse workspace can also be integrated with either Azure DevOps or GitHub. We will further examine this in Chapter 11, Developing and Maintaining the Presentation Layer.

  1. First of all, create a new project in your Azure DevOps environment. You can create it as a private project, and please choose Git for the Version control setting:

    Figure 5.35 –...

Previous Section
End of Section 8
Next Section