Summary
This chapter is the backbone for the chapters to come. We discussed a lot of challenges that every person/architect should be aware of when creating a data processing system within their organization. We discussed how to enable the large scale processing of data while keeping the overall system costs lower and how to keep the overall processing time within the defined SLA as the load on the processing system increases. Further, we learned that moving to the scale out architecture for data processing requires significant engineering effort as well as a shift in your perspective, and that the underlying components of the data processing system should not share resources but should be independent of each other. We covered the idea that failure is not an exception—the system should be built to handle failures. We also learned that the processing of data doesn't have to be always stream-like. The use case defines the processing pipeline design and not the other way around. We looked at...