Validating Batch loads
Batch jobs are usually run as part of Azure Data Factory (ADF). ADF provides functionalities for validating the outcome of jobs. Let's learn how to use the Validation activity in ADF to check the correctness of Batch loads:
- The Validation activity of ADF can be used to check for a file's existence before proceeding with the rest of the activities in the pipeline. The validation pipeline will look similar to the following:
Figure 11.4 – ADF Validation activity
- Once we have validated that the files exist, we can use the Get Metadata activity to get more information about the output files. In the following screenshot, we output Column count, which we'll check later using an If Condition activity to decide if the output files are any good:
Figure 11.5 – Configuring the Get Metadata activity to publish the Column count