Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

Chapter 9. Let Us Process Data in Batches

We have come a long way in our endeavor to understand the nuances of data-intensive systems.

At a high level, in the last eight chapters, through various discussions and examples, we have tried to understand the challenges surrounding data-intensive systems and some of the different ways to handle those challenges.

As you can imagine, at its core, the goal of any data-intensive system is to answer questions that are relevant in the context of your organization. Any question that you need answered can essentially be represented or implemented as a (complex) computation being performed on your entire dataset. This dataset could be in a relational database or a filesystem, or on your hard disk. That is irrelevant to the underlying system. What is important is the mechanism by which you arrive at an answer and the confidence with which you support the answer. Confidence in the result grows when it is derived from another set of high-confidence data. In...