There are typically two ways that the incoming messages can be processed. The most common way is to process each message one by one. And this is the way we have discussed throughout in this chapter. One-message-at-a-time stream processing is very low latency and obviously simple to handle, because you deal with a single message at a time. The downside with one-at-a-time processing is that a lot of time may be wasted by the underlying processors and stream systems to perform non-business-related tasks, such as fetching the individual message from the kernels page-cache, loading it in memory, transferring it across the network, and passing it down the underlying system. All this can significantly increase the load on the overall system, in terms of resources and time. Thus, it may be beneficial to consider a micro-batching strategy where the idea is to group small messages of the same type together and send them to the underlying processing framework as one large...
Architecting Data-Intensive Applications
By :
Architecting Data-Intensive Applications
By:
Overview of this book
<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p>
<p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Related Content you might be interested in
Current Title:
Architecting Data-Intensive Applications
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Free Chapter
Exploring the Data Ecosystem
Defining a Reference Architecture for Data-Intensive Systems
Patterns of the Data Intensive Architecture
Discussing Data-Centric Architectures
Understanding Data Collection and Normalization Requirements and Techniques
Creating a Data Pipeline for Consistent Data Collection, Processing, and Dissemination
Building a Robust and Fault-Tolerant Data Collection System
Challenges of Data Processing
Let Us Process Data in Batches
Handling Streams of Data
Let Us Store the Data
When Data Dissemination is as Important as Data Itself
Other Books You May Enjoy
Index
Customer Reviews