Chapter 5. Understanding Data Collection and Normalization Requirements and Techniques
So far, we have mostly discussed, at various levels of granularity, the architectural patterns that are important to keep in mind when designing a data-intensive system. As you will realize, knowledge of these Architectural Patterns will come in handy when we start to define the architecture of various functional components of a data-intensive system. This chapter starts diving deep in the first functional component that sits at the edge of any Data-Intensive System.
In this chapter, we will cover the following:
- Understanding, with examples of use cases, how to approach pinning down the requirements for your data-collection system
- Functional characteristics of a data collection System, such as keeping track of data lineage and preserving the quality of the data
- Various types of data sources (this was also discussed in Chapter 4, Discussing Data-Centric Architectures)
- Various requirements (both functional and...