Implementing partitioning
In Chapter 3, Designing a Partition Strategy, we covered the basics of partitioning. In this section, we will be learning how to implement the different types of partitioning. We will start with partitioning on Azure data stores and then look into partitioning for analytical workloads.
For storage-based partitioning, the main technique is to partition the data into the correct folder structure. In the previous chapters, we learned about how to store the data in a date-based format. The recommendation from Azure is to use the following pattern:
{Region}/{SubjectMatter(s)}/{yyyy}/{mm}/{dd}/{hh}/
Let's learn how to implement folder creation in an automated manner using ADF.
Using ADF/Synapse pipelines to create data partitions
You can use ADF or Synapse Pipelines, as both use the same ADF technology. In this example, I'm using Synapse Pipelines. Let's look at the steps to partition data in an automated fashion:
- In your Synapse...