When designing any system, the first thing to do is to list down exactly what is required from that system. This not only focuses the design effort, but also helps eliminate any differences between various stakeholders.
So, let's list some of the major functional and nonfunctional requirements for the Data-Collection System:
- The System should support both Pull-based as well as Push-based data collection
With pull-based data collection, we should be able to pull data, at regular intervals, from a given source. Most of the systems we collect data from are usually pull-based. They expose APIs that our data-collection system can utilize to pull data in an efficient manner.
Push-based data-collection requires the data-collection system to expose an API that the end users can use to push the data into the System. This kind of system is much more complicated to implement if the API must be exposed to the outside world. The reason is that you then need to implement...