Book Image

Modern Data Architecture on AWS

By : Behram Irani
5 (1)
Book Image

Modern Data Architecture on AWS

5 (1)
By: Behram Irani

Overview of this book

Many IT leaders and professionals are adept at extracting data from a particular type of database and deriving value from it. However, designing and implementing an enterprise-wide holistic data platform with purpose-built data services, all seamlessly working in tandem with the least amount of manual intervention, still poses a challenge. This book will help you explore end-to-end solutions to common data, analytics, and AI/ML use cases by leveraging AWS services. The chapters systematically take you through all the building blocks of a modern data platform, including data lakes, data warehouses, data ingestion patterns, data consumption patterns, data governance, and AI/ML patterns. Using real-world use cases, each chapter highlights the features and functionalities of numerous AWS services to enable you to create a scalable, flexible, performant, and cost-effective modern data platform. By the end of this book, you’ll be equipped with all the necessary architectural patterns and be able to apply this knowledge to efficiently build a modern data platform for your organization using AWS services.
Table of Contents (24 chapters)
1
Part 1: Foundational Data Lake
5
Part 2: Purpose-Built Services And Unified Data Access
17
Part 3: Govern, Scale, Optimize And Operationalize

Data Processing

In this chapter, we will look at the following key topics:

  • Challenges with data processing platforms
  • Data processing using Amazon EMR
  • Data processing using AWS Glue
  • Data processing using AWS Glue DataBrew

Let’s quickly recap what we have covered so far in this book. We set the foundation by creating the layers of a data lake on Amazon S3. The layers represent distinct storage areas where all the data can exist in a centralized location. The next piece of the puzzle we solved was to get data from disparate sources into the raw layer of the data lake in S3. Then, we spent the whole of Chapter 3 looking at batch data ingestion mechanisms, followed by Chapter 4, where we discussed streaming data ingestion mechanisms.

So, till this point, all the data is in the raw layer of S3; of course, it can also go directly to the conformed layer, if you have processed and optimized the data on the fly during the ingestion process. If you recall...