Book Image

Serverless ETL and Analytics with AWS Glue

By : Vishal Pathak, Subramanya Vajiraya, Noritaka Sekiyama, Tomohiro Tanaka, Albert Quiroga, Ishan Gaur
Book Image

Serverless ETL and Analytics with AWS Glue

By: Vishal Pathak, Subramanya Vajiraya, Noritaka Sekiyama, Tomohiro Tanaka, Albert Quiroga, Ishan Gaur

Overview of this book

Organizations these days have gravitated toward services such as AWS Glue that undertake undifferentiated heavy lifting and provide serverless Spark, enabling you to create and manage data lakes in a serverless fashion. This guide shows you how AWS Glue can be used to solve real-world problems along with helping you learn about data processing, data integration, and building data lakes. Beginning with AWS Glue basics, this book teaches you how to perform various aspects of data analysis such as ad hoc queries, data visualization, and real-time analysis using this service. It also provides a walk-through of CI/CD for AWS Glue and how to shift left on quality using automated regression tests. You’ll find out how data security aspects such as access control, encryption, auditing, and networking are implemented, as well as getting to grips with useful techniques such as picking the right file format, compression, partitioning, and bucketing. As you advance, you’ll discover AWS Glue features such as crawlers, Lake Formation, governed tables, lineage, DataBrew, Glue Studio, and custom connectors. The concluding chapters help you to understand various performance tuning, troubleshooting, and monitoring options. By the end of this AWS book, you’ll be able to create, manage, troubleshoot, and deploy ETL pipelines using AWS Glue.

Related Content you might be interested in

Current Title:

Small book image
Serverless ETL and Analytics with AWS Glue
Vishal Pathak | Subramanya Vajiraya | Noritaka Sekiyama | Tomohiro Tanaka | Albert Quiroga | Ishan Gaur
Aug, 2022 | 434 pages
No titles found
Table of Contents (20 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1 – Introduction, Concepts, and the Basics of AWS Glue
Section 1 – Introduction, Concepts, and the Basics of AWS Glue
Free Chapter
2
Chapter 1: Data Management – Introduction and Concepts
Chapter 1: Data Management – Introduction and Concepts
Types of data processing – OLTP and OLAP
Data warehouses and data marts
Data lakes
Data lakehouse
Data mesh
Distributed computing for big data
AWS Glue
Summary
3
Chapter 2: Introduction to Important AWS Glue Features
Chapter 2: Introduction to Important AWS Glue Features
Data integration
Integrating data with AWS Glue
Features of AWS Glue
Summary
4
Chapter 3: Data Ingestion
Chapter 3: Data Ingestion
Technical requirements
Data ingestion from file/object stores
Data ingestion from JDBC data stores
Data ingestion from streaming data sources
Data ingestion from SaaS data stores
Summary
5
Section 2 – Data Preparation, Management, and Security
Section 2 – Data Preparation, Management, and Security
6
Chapter 4: Data Preparation
Chapter 4: Data Preparation
Technical requirements
Introduction to data preparation
Data preparation using AWS Glue
Selecting the right service/tool
Summary
7
Chapter 5: Data Layouts
Chapter 5: Data Layouts
Technical requirements
Why do we need to pay attention to data layout?
Key techniques to optimally storing data
Optimizing the number of files and each file size
Optimizing your storage with Amazon S3
Summary
Further reading
8
Chapter 6: Data Management
Chapter 6: Data Management
Technical requirements
Normalizing data
Deduplicating records
Denormalizing tables
Securing data content
Managing data quality
Summary
9
Chapter 7: Metadata Management
Chapter 7: Metadata Management
Technical requirements
Populating metadata
Maintaining metadata
Partition management
Versioning and rollback
Lineage
Summary
10
Chapter 8: Data Security
Chapter 8: Data Security
Technical requirements
Access control
Encryption
Network
Summary
11
Chapter 9: Data Sharing
Chapter 9: Data Sharing
Technical requirements
Overview of data sharing strategies
Sharing data with multiple AWS accounts using S3 bucket policies and Glue catalog policies
Sharing data with multiple AWS accounts using AWS Lake Formation permissions
Summary
12
Chapter 10: Data Pipeline Management
Chapter 10: Data Pipeline Management
Technical requirements
What are data pipelines?
Selecting the appropriate data processing services for your analysis
Orchestrating your pipelines with workflow tools
utomating how you provision your pipelines with provisioning tools
Developing and maintaining your data pipelines
Summary
Further reading
13
Section 3 – Tuning, Monitoring, Data Lake Common Scenarios, and Interesting Edge Cases
Section 3 – Tuning, Monitoring, Data Lake Common Scenarios, and Interesting Edge Cases
14
Chapter 11: Monitoring
Chapter 11: Monitoring
Defining an SLA for a data platform
Monitoring the SLA of a data platform
Monitoring the components of a data platform
Analyzing usage
Summary
15
Chapter 12: Tuning, Debugging, and Troubleshooting
Chapter 12: Tuning, Debugging, and Troubleshooting
Tuning AWS Glue workloads
Troubleshooting and debugging common issues in AWS Glue ETL
Summary
16
Chapter 13: Data Analysis
Chapter 13: Data Analysis
Creating Marketplace connections
Creating the CloudFormation stack
The benefit of ad hoc analysis and how a data lake enables it
Creating and updating Hudi tables using Glue
Creating and updating Delta Lake tables using Glue
Inserting data into Lake Formation governed tables
Consuming streaming data using Glue
Glue’s integration with OpenSearch
Cleaning up
Summary
17
Chapter 14: Machine Learning Integration
Chapter 14: Machine Learning Integration
Technical requirements
Glue ML transformations
SageMaker integration
Developing ML pipelines with Glue
Summary
18
Chapter 15: Architecting Data Lakes for Real-World Scenarios and Edge Cases
Chapter 15: Architecting Data Lakes for Real-World Scenarios and Edge Cases
Technical requirements
Running a highly selective query on a big fact table using AWS Glue
Dealing with Join performance issues with big fact and small dimension tables in ETL workloads
Solving Join problems involving big fact and big dimension tables using AWS Glue
Reducing time on read operations using AWS Glue grouping
Solving S3 eventual consistency problems using AWS Glue
Summary
Why subscribe?
19
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Tuning AWS Glue workloads

Based on our discussions in the previous chapter, we already know that AWS Glue is a serverless data integration service wherein different components are bundled with a number of optimizations that cover most use cases—most being the operative word here. The optimizations already in place may not be the perfect fit for our use case, and they can be further improved to get the most out of the resources we are allocating.

It is still up to us to monitor workloads and implement optimizations where necessary to ensure that we are making use of resources efficiently. The performance of any Glue component is dependent on a number of factors such as input data, resources allocated, configuration, and the actual workflow itself.

Now, let’s discuss some of the tuning mechanisms we can use to optimize different components of AWS Glue.

Tuning AWS Glue crawlers

As discussed in the previous section, the performance of a Glue component depends...

Previous Section
End of Section 2
Next Section