Book Image

Azure Data Engineer Associate Certification Guide

By : Newton Alex
Book Image

Azure Data Engineer Associate Certification Guide

By: Newton Alex

Overview of this book

Azure is one of the leading cloud providers in the world, providing numerous services for data hosting and data processing. Most of the companies today are either cloud-native or are migrating to the cloud much faster than ever. This has led to an explosion of data engineering jobs, with aspiring and experienced data engineers trying to outshine each other. Gaining the DP-203: Azure Data Engineer Associate certification is a sure-fire way of showing future employers that you have what it takes to become an Azure Data Engineer. This book will help you prepare for the DP-203 examination in a structured way, covering all the topics specified in the syllabus with detailed explanations and exam tips. The book starts by covering the fundamentals of Azure, and then takes the example of a hypothetical company and walks you through the various stages of building data engineering solutions. Throughout the chapters, you'll learn about the various Azure components involved in building the data systems and will explore them using a wide range of real-world use cases. Finally, you’ll work on sample questions and answers to familiarize yourself with the pattern of the exam. By the end of this Azure book, you'll have gained the confidence you need to pass the DP-203 exam with ease and land your dream job in data engineering.

Related Content you might be interested in

Current Title:

Small book image
Azure Data Engineer Associate Certification Guide
Newton Alex
Feb, 2022 | 574 pages
Small book image
Azure Synapse Analytics Cookbook
Gaurav Agarwal | Meenakshi Muralidharan
Apr, 2022 | 238 pages
Small book image
Limitless Analytics with Azure Synapse
Prashant Kumar Mishra
Jun, 2021 | 392 pages
Small book image
Cloud Scale Analytics with Azure Data Services.
Patrik Borosch
Jul, 2021 | 520 pages
Table of Contents (23 chapters)
Preface
Preface
Who this book is for
What this book covers
Download the example code files
Download the color images
Get in touch
Reviews
Share Your Thoughts
1
Part 1: Azure Basics
Part 1: Azure Basics
Free Chapter
2
Chapter 1: Introducing Azure Basics
Chapter 1: Introducing Azure Basics
Technical requirements
Introducing the Azure portal
Exploring Azure accounts, subscriptions, and resource groups
Introducing Azure Services
Exploring Azure VMs
Exploring Azure Storage
Exploring Azure Networking (VNet)
Exploring Azure Compute
Summary
3
Part 2: Data Storage
Part 2: Data Storage
4
Chapter 2: Designing a Data Storage Structure
Chapter 2: Designing a Data Storage Structure
Technical requirements
Designing an Azure data lake
Selecting the right file types for storage
Choosing the right file types for analytical queries
Designing storage for efficient querying
Designing storage for data pruning
Designing folder structures for data transformation
Designing a distribution strategy
Designing a data archiving solution
Summary
5
Chapter 3: Designing a Partition Strategy
Chapter 3: Designing a Partition Strategy
Understanding the basics of partitioning
Designing a partition strategy for files
Designing a partition strategy for analytical workloads
Designing a partition strategy for efficiency/performance
Designing a partition strategy for Azure Synapse Analytics
Identifying when partitioning is needed in ADLS Gen2
Summary
6
Chapter 4: Designing the Serving Layer
Chapter 4: Designing the Serving Layer
Technical requirements
Learning the basics of data modeling and schemas
Designing Star and Snowflake schemas
Designing SCDs
Designing a solution for temporal data
Designing a dimensional hierarchy
Designing for incremental loading
Designing analytical stores
Designing metastores in Azure Synapse Analytics and Azure Databricks
Summary
7
Chapter 5: Implementing Physical Data Storage Structures
Chapter 5: Implementing Physical Data Storage Structures
Technical requirements
Getting started with Azure Synapse Analytics
Implementing compression
Implementing partitioning
Implementing horizontal partitioning or sharding
Implementing distributions
Implementing different table geometries with Azure Synapse Analytics pools
Implementing data redundancy
Implementing data archiving
Summary
8
Chapter 6: Implementing Logical Data Structures
Chapter 6: Implementing Logical Data Structures
Technical requirements
Building a temporal data solution
Building a slowly changing dimension
Building a logical folder structure
Implementing file and folder structures for efficient querying and data pruning
Building external tables
Summary
9
Chapter 7: Implementing the Serving Layer
Chapter 7: Implementing the Serving Layer
Technical requirements
Delivering data in a relational star schema
Implementing a dimensional hierarchy
Maintaining metadata
Summary
10
Part 3: Design and Develop Data Processing (25-30%)
Part 3: Design and Develop Data Processing (25-30%)
11
Chapter 8: Ingesting and Transforming Data
Chapter 8: Ingesting and Transforming Data
Technical requirements
Transforming data by using Apache Spark
Transforming data by using T-SQL
Transforming data by using ADF
Transforming data by using Azure Synapse pipelines
Transforming data by using Stream Analytics
Cleansing data
Splitting data
Shredding JSON
Encoding and decoding data
Configuring error handling for the transformation
Normalizing and denormalizing values
Transforming data by using Scala
Performing Exploratory Data Analysis (EDA)
Summary
12
Chapter 9: Designing and Developing a Batch Processing Solution
Chapter 9: Designing and Developing a Batch Processing Solution
Technical requirements
Designing a batch processing solution
Developing batch processing solutions by using Data Factory, Data Lake, Spark, Azure Synapse Pipelines, PolyBase, and Azure Databricks
Creating data pipelines
Integrating Jupyter/Python notebooks into a data pipeline
Designing and implementing incremental data loads
Designing and developing slowly changing dimensions
Handling duplicate data
Handling missing data
Handling late-arriving data
Upserting data
Regressing to a previous state
Introducing Azure Batch
Configuring the batch size
Scaling resources
Configuring batch retention
Designing and configuring exception handling
Handling security and compliance requirements
Summary
13
Chapter 10: Designing and Developing a Stream Processing Solution
Chapter 10: Designing and Developing a Stream Processing Solution
Technical requirements
Designing a stream processing solution
Developing a stream processing solution using ASA, Azure Databricks, and Azure Event Hubs
Processing data using Spark Structured Streaming
Monitoring for performance and functional regressions
Processing time series data
Designing and creating windowed aggregates
Configuring checkpoints/watermarking during processing
Replaying archived stream data
Transformations using streaming analytics
Handling schema drifts
Processing across partitions
Processing within one partition
Scaling resources
Handling interruptions
Designing and configuring exception handling
Upserting data
Designing and creating tests for data pipelines
Optimizing pipelines for analytical or transactional purposes
Summary
14
Chapter 11: Managing Batches and Pipelines
Chapter 11: Managing Batches and Pipelines
Technical requirements
Triggering batches
Handling failed Batch loads
Validating Batch loads
Scheduling data pipelines in Data Factory/Synapse pipelines
Managing data pipelines in Data Factory/Synapse pipelines
Managing Spark jobs in a pipeline
Implementing version control for pipeline artifacts
Summary
15
Part 4: Design and Implement Data Security (10-15%)
Part 4: Design and Implement Data Security (10-15%)
16
Chapter 12: Designing Security for Data Policies and Standards
Chapter 12: Designing Security for Data Policies and Standards
Technical requirements
Introducing the security and privacy requirements
Designing and implementing data encryption for data at rest and in transit
Designing and implementing a data auditing strategy
Designing and implementing a data masking strategy
Designing and implementing Azure role-based access control and a POSIX-like access control list for Data Lake Storage Gen2
Designing and implementing row-level and column-level security
Designing and implementing a data retention policy
Designing to purge data based on business requirements
Managing identities, keys, and secrets across different data platform technologies
Implementing secure endpoints (private and public)
Implementing resource tokens in Azure Databricks
Loading a DataFrame with sensitive information
Writing encrypted data to tables or Parquet files
Designing for data privacy and managing sensitive information
Summary
17
Part 5: Monitor and Optimize Data Storage and Data Processing (10-15%)
Part 5: Monitor and Optimize Data Storage and Data Processing (10-15%)
18
Chapter 13: Monitoring Data Storage and Data Processing
Chapter 13: Monitoring Data Storage and Data Processing
Technical requirements
Implementing logging used by Azure Monitor
Configuring monitoring services
Understanding custom logging options
Interpreting Azure Monitor metrics and logs
Measuring the performance of data movement
Monitoring data pipeline performance
Monitoring and updating statistics about data across a system
Measuring query performance
Interpreting a Spark DAG
Monitoring cluster performance
Scheduling and monitoring pipeline tests
Summary
19
Chapter 14: Optimizing and Troubleshooting Data Storage and Data Processing
Chapter 14: Optimizing and Troubleshooting Data Storage and Data Processing
Technical requirements
Compacting small files
Rewriting user-defined functions (UDFs)
Handling skews in data
Handling data spills
Tuning shuffle partitions
Finding shuffling in a pipeline
Optimizing resource management
Tuning queries by using indexers
Tuning queries by using cache
Optimizing pipelines for analytical or transactional purposes
Optimizing pipelines for descriptive versus analytical workloads
Troubleshooting a failed Spark job
Troubleshooting a failed pipeline run
Summary
20
Part 6: Practice Exercises
Part 6: Practice Exercises
21
Chapter 15: Sample Questions with Solutions
Chapter 15: Sample Questions with Solutions
Exploring the question formats
Case study-based questions
Scenario-based questions
Direct questions
Ordering sequence questions
Code segment questions
Sample questions from the Design and Implement Data Storage section
Sample questions from the Design and Develop Data Processing section
Sample questions from the Design and Implement Data Security section
Sample questions from the Monitor and Optimize Data Storage and Data Processing section
Summary
Why subscribe?
22
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Troubleshooting a failed Spark job

There are two aspects to troubleshooting a failed Spark job in a cloud environment: environmental issues and job issues. Let's look at both of these factors in detail.

Debugging environmental issues

Here are some of the steps involved in checking environmental issues:

  1. Check the health of Azure services in the region where your Spark clusters are running by using this link: https://status.azure.com/en-us/status.
  2. Next, check whether your Spark cluster itself is fine. You can do this for your HDInsight clusters by checking the Ambari home page. We saw how to check Ambari for the status in Chapter 13, Monitoring Data Storage and Data Processing, in the Monitoring overall cluster performance section. Here is the Ambari screen home page again for your reference:

Figure 14.23 – Ambari home page showing the status of the cluster

  1. Check to see whether any service is down or whether any of the resources...
Previous Section
End of Section 14
Next Section