Sharing resources among processing applications
Now that we have identified that a scale out approach to building a data processing system is a more reasonable approach, we need to understand that scaling out is achieved by adding more hosts to the overall system. These hosts run only a part of the overall processing logic and thus need to communicate with other components that are handling some other part of the processing logic. There may be cases where a piece of data is critical for every underlying component and thus needs to be passed across. At the same time, you may not want to send all the data to all the components of the system as it will overwhelm the component, thereby potentially bringing it down.
Thus, we identify another challenge in building the data processing system.
The underlying components of the data processing system should not share resources, but instead should be independent of each other.
Adopting such an approach helps isolate failures in the system by clearly...