Book Image

Implementing Cloud Design Patterns for AWS - Second Edition

By : Sean Keery, Clive Harber, Marcus Young
Book Image

Implementing Cloud Design Patterns for AWS - Second Edition

By: Sean Keery, Clive Harber, Marcus Young

Overview of this book

Whether you're just getting your feet wet in cloud infrastructure or already creating complex systems, this book will guide you through using the patterns to fit your system needs. Starting with patterns that cover basic processes such as source control and infrastructure-as-code, the book goes on to introduce cloud security practices. You'll then cover patterns of availability and scalability and get acquainted with the ephemeral nature of cloud environments. You'll also explore advanced DevOps patterns in operations and maintenance, before focusing on virtualization patterns such as containerization and serverless computing. In the final leg of your journey, this book will delve into data persistence and visualization patterns. You'll get to grips with architectures for processing static and dynamic data, as well as practices for managing streaming data. By the end of this book, you will be able to design applications that are tolerant of underlying hardware failures, resilient against an unexpected influx of data, and easy to manage and replicate.
Table of Contents (20 chapters)
Title Page
Dedication
About Packt
Contributors
Preface
Free Chapter
1
Introduction to Amazon Web Services
Index

Queuing


Queues are used to collect intra-system messages and/or external events. Processing of these messages can happen sequentially or in parallel, depending on how many consumers are available to process them. Messages can also be routed to different parts of your system based on various criteria (as dictated by the queuing engine), to make sure that it is handled appropriately.

 

 

AWS provides different queues and queue-like services for different purposes:

  • Managed Streaming for Kafka (MSK)
  • Kinesis (data streams and firehose)
  • Amazon MQ
  • Amazon Simple Queue Service (SQS)

We will be discussing Kafka in this section.

MSK

MSK is a managed service. Apache Kafka is a platform for building real-time streaming pipelines that can be used to populate data lakes, stream system changes, and events, moving data to and from databases. Machine learning and analytics applications can also benefit from using MSK.

There is usually a significant challenge in setting up Kafka clusters; using MSK saves you from having...