Calling Databricks notebook execution in ADF
We now have laid down everything to trigger the notebook execution in ADF. Going back to the factory, we're going to add a linked service. So far, all the linked services we created in this book were connected to a data store: SQL Server, blob storage, and so on. This time, we're going to use a computation linked service: Azure Databricks.
As shown in the following screenshot, add a linked service. Click on the Compute tab, select Azure Databricks, and click on Continue:
We'll now enter the details of the cluster in the next step. We used Azure Databricks for the Type property. The following screenshot shows the properties to set:
The properties are explained as follows:
Connect via integration runtime: We use theDefaultone. It has access to all Azure resources.Account selection method:From Azure subscription.Select cluster: We're going to create a cluster on the fly and it will be running only for the duration of our job; therefore, we select...