Let's look at the journey of data from its creation to its usage:
- Raw data creation: The observation, event, action, and manual entry are the key elements that contribute to data creation. This data is typically persisted as a raw data source for future usage. The persistent storage could be flat files, a database, a Kafka topic, AWS Kinesis Data Streams (KDS), or any other suitable storage.
- Raw data extraction: Raw data extraction is the act of receiving or fetching raw data from a source. In an enterprise, raw data sources are internal as well as external. Some examples of commonly used external sources are currency exchange rates, stock prices, and weather data. A company's transactional data is an example of internal data.
- Raw data ingestion: Raw data ingestion refers to the act of storing raw data in an organized form to support orderly data extraction...