Identifying when partitioning is needed in ADLS Gen2
As we have learned in the previous chapter, we can partition data according to our requirements—such as performance, scalability, security, operational overhead, and so on—but there is another reason why we might end up partitioning our data, and that is the various I/O bandwidth limits that are imposed at subscription levels by Azure. These limits apply to both Blob storage and ADLS Gen2.
The rate at which we ingest data into an Azure Storage system is called the ingress rate, and the rate at which we move the data out of the Azure Storage system is called the egress rate.
The following table shows a snapshot of some of the limits enforced by Azure Blob storage. This table is just to give you an idea of the limits that Azure Storage imposes. When we design our data lake applications, we need to take care of such restrictions as part of our design itself:
Figure 3.4 – Some of the...