What this book covers
Chapter 1, An Overview of Amazon EMR, will give you an overview of Amazon EMR and its benefits compared to on-premises Hadoop clusters. Also, we will look at how EMR compares with other Spark-based AWS services such as AWS Glue and AWS Glue DataBrew.
Chapter 2, Exploring the Architecture and Deployment Options, will dive into EMR architecture; its life cycle; types of clusters; deployment options such as EMR on Elastic Compute Cloud (EC2), EMR on Elastic Kubernetes Service (EKS), and EMR on AWS Outposts; and the pricing models for each.
Chapter 3, Common Use Cases and Architecture Patterns, will explain some popular big data use cases for EMR and cover how you can build an end-to-end architecture for batch or real-time streaming and interactive analytics use cases.
Chapter 4, Big Data Applications and Notebooks Available in Amazon EMR, will give you an overview of a few of the popular Hadoop ecosystem services in EMR, such as Hive, Presto, and Spark. We will also look at a few popular machine learning frameworks, such as TensorFlow and MXNet, as well as the different notebook options available for interactive development.
Chapter 5, Setting Up and Configuring EMR Clusters, is where you will learn how to set up an EMR cluster, dive deep into its advanced configurations, learn how you can debug and troubleshoot cluster failures, and then get an overview of its SDKs and APIs.
Chapter 6, Monitoring, Scaling, and High Availability, is where you will learn about cluster monitoring, cloning, and high availability. Then, you will dive deep into different scaling aspects and understand the difference between managed and auto scaling.
Chapter 7, Understanding Security in Amazon EMR, will explain how you can make your cluster secure, covering authentication and authorization with AWS IAM, data encryption at rest and in transit, and how to leverage AWS security groups to control connectivity to your cluster.
Chapter 8, Understanding Data Governance in Amazon EMR, covers external Hive Metastore and Glue Data Catalog integration and how you can implement granular permission management with AWS Lake Formation and Apache Ranger.
Chapter 9, Implementing Batch ETL Pipeline with Amazon EMR and Apache Spark, takes you through a step-by-step guide to implementing a batch Extract, Transform, and Load (ETL) pipeline with Amazon EMR and Apache Spark.
Chapter 10, Implementing Real-Time Streaming with Amazon EMR and Spark Streaming, contains a step-by-step guide to implementing a real-time streaming ETL pipeline with Kinesis Data Stream, Amazon EMR, and Spark Streaming.
Chapter 11, Implementing UPSERT on S3 Data Lake with Apache Spark and Apache Hudi, teaches you how to do interactive development with an EMR notebook, as well as how to integrate UPSERT on an S3 data lake with Apache Hudi and Spark.
Chapter 12, Orchestrating Amazon EMR Jobs with AWS Step Functions and Apache Airflow/MWAA, gives you an overview of AWS Step Functions, Amazon-managed Airflow, and how to integrate them to build a workflow for your EMR-based data pipelines.
Chapter 13, Migrating On-Premises Hadoop Workloads to Amazon EMR, discusses how you can migrate your on-premises Hadoop workloads to Amazon EMR, covering migrating data, catalog metadata, and ETL jobs. Also, you will learn about some of the best practices you can follow during the migration process.
Chapter 14, Best Practices and Cost Optimization Techniques, discusses some of the best practices you can follow related to EMR cluster configuration, ETL processes, file storage, and security. Then, you will learn about some of the cost optimization techniques you can follow, the limitations of EMR, and some workarounds for them.