Most data warehousing projects follow a standard process. This process involves the extraction of data from a data source, the transformation of this data to both the standards of the data warehouse and the requirements of the end user, and the loading of data into the resulting database table. This process is more commonly known as the Extract, Transform, Load Process, or ETL for short.
The transformation step is important for a few reasons:
- Decoding of encoded values (that is, converting values of W and E to West and East)
- Calculation of values (that is, calculating the average order value by dividing the revenue by the count of orders)
- Splitting separated lists into individual columns
- Aggregation
- Data validation, either in the form of invalidating incorrect values or as reprocessing of incorrect data
If you are using files to load data into BigQuery then...