Adding data transformation logic
Up to now, we have examined the ELT mode of Synapse pipelines/Azure Data Factory. But like other data integration tools, Synapse pipelines and Data Factory offer the option to create logical dependencies in your pipeline run. You can put together different activities in a single pipeline and connect them to form your loading logic.
Figure 5.28 shows the possibilities for adding dependencies between two activities. In this way, you can set up complex loading logic. The displayed dependency would run the ForEach activity only when the Copy data activity has been run successfully:
Figure 5.25 – Adding dependencies to your pipeline
But what about transformation logic? What about the need to calculate, filter, or aggregate data and add other information from elsewhere to your data?
You can use data flows to do so. Data flows will give you a graphical interface to "program" your transformation logic and you...