Book Image

Data Engineering with AWS

By : Gareth Eagar
Book Image

Data Engineering with AWS

By: Gareth Eagar

Overview of this book

Written by a Senior Data Architect with over twenty-five years of experience in the business, Data Engineering for AWS is a book whose sole aim is to make you proficient in using the AWS ecosystem. Using a thorough and hands-on approach to data, this book will give aspiring and new data engineers a solid theoretical and practical foundation to succeed with AWS. As you progress, you’ll be taken through the services and the skills you need to architect and implement data pipelines on AWS. You'll begin by reviewing important data engineering concepts and some of the core AWS services that form a part of the data engineer's toolkit. You'll then architect a data pipeline, review raw data sources, transform the data, and learn how the transformed data is used by various data consumers. You’ll also learn about populating data marts and data warehouses along with how a data lakehouse fits into the picture. Later, you'll be introduced to AWS tools for analyzing data, including those for ad-hoc SQL queries and creating visualizations. In the final chapters, you'll understand how the power of machine learning and artificial intelligence can be used to draw new insights from data. By the end of this AWS book, you'll be able to carry out data engineering tasks and implement a data pipeline on AWS independently.

Related Content you might be interested in

Current Title:

Small book image
Data Engineering with AWS
Gareth Eagar
Dec, 2021 | 482 pages
Small book image
Modern Data Architecture on AWS
Behram Irani
Aug, 2023 | 420 pages
Small book image
Data Wrangling on AWS
Sankar M | Sampat Palani | Navnit Shukla
Jul, 2023 | 420 pages
Small book image
Serverless Analytics with Amazon Athena
Anthony Virtuoso | Aaron Wishnick | Mert Hocanin
Nov, 2021 | 438 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1: AWS Data Engineering Concepts and Trends
Section 1: AWS Data Engineering Concepts and Trends
Free Chapter
2
Chapter 1: An Introduction to Data Engineering
Chapter 1: An Introduction to Data Engineering
Technical requirements
The rise of big data as a corporate asset
The challenges of ever-growing datasets
Data engineers – the big data enablers
The benefits of the cloud when building big data analytic solutions
Hands-on – creating and accessing your AWS account
Summary
3
Chapter 2: Data Management Architectures for Analytics
Chapter 2: Data Management Architectures for Analytics
Technical requirements
The evolution of data management for analytics
Understanding data warehouses and data marts – fountains of truth
Building data lakes to tame the variety and volume of big data
Bringing together the best of both worlds with the lake house architecture
Hands-on – configuring the AWS Command Line Interface tool and creating an S3 bucket
Summary
4
Chapter 3: The AWS Data Engineer's Toolkit
Chapter 3: The AWS Data Engineer's Toolkit
Technical requirements
AWS services for ingesting data
AWS services for transforming data
AWS services for orchestrating big data pipelines
AWS services for consuming data
Hands-on – triggering an AWS Lambda function when a new file arrives in an S3 bucket
Summary
5
Chapter 4: Data Cataloging, Security, and Governance
Chapter 4: Data Cataloging, Security, and Governance
Technical requirements
Getting data security and governance right
Cataloging your data to avoid the data swamp
The AWS Glue/Lake Formation data catalog
AWS services for data encryption and security monitoring
AWS services for managing identity and permissions
Hands-on – configuring Lake Formation permissions
Summary
6
Section 2: Architecting and Implementing Data Lakes and Data Lake Houses
Section 2: Architecting and Implementing Data Lakes and Data Lake Houses
7
Chapter 5: Architecting Data Engineering Pipelines
Chapter 5: Architecting Data Engineering Pipelines
Technical requirements
Approaching the data pipeline architecture
Identifying data consumers and understanding their requirements
Identifying data sources and ingesting data
Identifying data transformations and optimizations
Loading data into data marts
Wrapping up the whiteboarding session
Hands-on – architecting a sample pipeline
Summary
8
Chapter 6: Ingesting Batch and Streaming Data
Chapter 6: Ingesting Batch and Streaming Data
Technical requirements
Understanding data sources
Ingesting data from a relational database
Ingesting streaming data
Hands-on – ingesting data with AWS DMS
Hands-on – ingesting streaming data
Summary
9
Chapter 7: Transforming Data to Optimize for Analytics
Chapter 7: Transforming Data to Optimize for Analytics
Technical requirements
Transformations – making raw data more valuable
Types of data transformation tools
Data preparation transformations
Business use case transforms
Working with change data capture (CDC) data
Hands-on – joining datasets with AWS Glue Studio
Summary
10
Chapter 8: Identifying and Enabling Data Consumers
Chapter 8: Identifying and Enabling Data Consumers
Technical requirements
Understanding the impact of data democratization
Meeting the needs of business users with data visualization
Meeting the needs of data analysts with structured reporting
Meeting the needs of data scientists and ML models
Hands-on – creating data transformations with AWS Glue DataBrew
Summary
11
Chapter 9: Loading Data into a Data Mart
Chapter 9: Loading Data into a Data Mart
Technical requirements
Extending analytics with data warehouses/data marts
What not to do – anti-patterns for a data warehouse
Redshift architecture review and storage deep dive
Designing a high-performance data warehouse
Moving data between a data lake and Redshift
Hands-on – loading data into an Amazon Redshift cluster and running queries
Summary
12
Chapter 10: Orchestrating the Data Pipeline
Chapter 10: Orchestrating the Data Pipeline
Technical requirements
Understanding the core concepts for pipeline orchestration
Examining the options for orchestrating pipelines in AWS
Hands-on – orchestrating a data pipeline using AWS Step Function
Summary
13
Section 3: The Bigger Picture: Data Analytics, Data Visualization, and Machine Learning
Section 3: The Bigger Picture: Data Analytics, Data Visualization, and Machine Learning
14
Chapter 11: Ad Hoc Queries with Amazon Athena
Chapter 11: Ad Hoc Queries with Amazon Athena
Technical requirements
Amazon Athena – in-place SQL analytics for the data lake
Tips and tricks to optimize Amazon Athena queries
Federating the queries of external data sources with Amazon Athena Query Federation
Managing governance and costs with Amazon Athena Workgroups
Hands-on – creating an Amazon Athena workgroup and configuring Athena settings
Hands-on – switching Workgroups and running queries
Summary
15
Chapter 12: Visualizing Data with Amazon QuickSight
Chapter 12: Visualizing Data with Amazon QuickSight
Technical requirements
Representing data visually for maximum impact
Understanding Amazon QuickSight's core concepts
Ingesting and preparing data from a variety of sources
Creating and sharing visuals with QuickSight analyses and dashboards
Understanding QuickSight's advanced features – ML Insights and embedded dashboards
Hands-on – creating a simple QuickSight visualization
Summary
16
Chapter 13: Enabling Artificial Intelligence and Machine Learning
Chapter 13: Enabling Artificial Intelligence and Machine Learning
Technical requirements
Understanding the value of ML and AI for organizations
Exploring AWS services for ML
Exploring AWS services for AI
Hands-on – reviewing reviews with Amazon Comprehend
Summary
Further reading
17
Chapter 14: Wrapping Up the First Part of Your Learning Journey
Chapter 14: Wrapping Up the First Part of Your Learning Journey
Technical requirements
Looking at the data analytics big picture
Examining examples of real-world data pipelines
Imagining the future – a look at emerging trends
Hands-on – cleaning up your AWS account
Summary
Why subscribe?
18
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

To get the most out of this book

Basic knowledge of computer systems and concepts, and how these are used within large organizations, is helpful prerequisite knowledge for this book. However, no data engineering-specific skills or knowledge is required. Also, a familiarity with cloud computing fundamentals and core AWS systems will make it easier to follow along, especially with the hands-on exercises, but detailed step-by-step instructions are included for each task.

All hands-on exercises make use of cloud-based services, so beyond using a supported web browser with a stable internet connection, there are no additional hardware or software requirements.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Things change fast in the computing industry, and this is clearly seen within the cloud industry. AWS is constantly rolling out new services, as well as improvements for existing services, and some of these improvements lead to changes in the user interface provided via the AWS console.

As a result, some of the screenshots included in this book may not look identical to what you are seeing in the AWS console when completing hands-on exercises. Or, you may find that a specific screen has additional options beyond what is shown in the screenshot in this book. It is unlikely that these changes will prevent you from following along with the step-by-step instructions in this book, but anything that may significantly impact a hands-on exercise will be addressed with a note for that chapter in this book's GitHub repository. Therefore, please refer to the GitHub repository as you complete the hands-on exercises for each chapter. In addition to notes about any significant console changes, the GitHub repository also includes copies of code contained in this book and other useful resources.

Previous Section
Next Section