Implementing file and folder structures for efficient querying and data pruning
The concepts we explored in the previous section applies here too. Once we have a date-based hierarchical folder structure, query performance can be improved via data partitioning. If we divide the data into partitions and if we ensure that the partitions are stored in different folder structures, then the queries can skip scanning the irrelevant partitions altogether. This concept, as we already know, is called data pruning.
Another benefit of partitioning is the increased efficiency of data loading and deletion by performing partition switching and partition deletion. Here, instead of reading each row and updating it, huge partitions of data can be added or deleted with simple metadata operations. Chapter 2, Designing a Data Storage Structure, already covered examples of how queries can benefit from data pruning by skipping reading from unnecessary partitions. In this section, we'll learn how...