Understanding the data loading process
In Chapter 4, Real-World Usage Scenarios, we discussed sample architectures that you can use as blueprints in your own projects. For every single one of them, on the left-hand side of the diagrams, you could see data sources, followed by the Ingest stage. Your data load strategy, and the services you will use to perform data ingestion, will depend on your data source types and your latency requirements.
Simply put, the data loading process can be summarized into four steps:
- Defining a retention policy.
- Choosing a data load strategy.
- Creating destination tables and data mappings.
- Performing data ingestion.
Note that before you perform the data ingestion task, you should create your destination tables and data mappings. Since we will explore different ways to perform data ingestion, the steps to create destination tables and data mappings in this chapter (when needed) will be presented in topics where we will perform...