Chapter 9. Let Us Process Data in Batches
We have come a long way in our endeavor to understand the nuances of data-intensive systems.
At a high level, in the last eight chapters, through various discussions and examples, we have tried to understand the challenges surrounding data-intensive systems and some of the different ways to handle those challenges.
As you can imagine, at its core, the goal of any data-intensive system is to answer questions that are relevant in the context of your organization. Any question that you need answered can essentially be represented or implemented as a (complex) computation being performed on your entire dataset. This dataset could be in a relational database or a filesystem, or on your hard disk. That is irrelevant to the underlying system. What is important is the mechanism by which you arrive at an answer and the confidence with which you support the answer. Confidence in the result grows when it is derived from another set of high-confidence data. In...