Creating validation functions
One of the ways to implement the data validation process in Data Services is to use validation functions along with the Validation transform in your dataflow to split the flow of data into two: records that pass the defined validation rule and those that do not. Those validation rules can be combined into validation function objects for your convenience and traceability.
In this recipe, we will create a standard but quite simple validation function. We will deploy it in our dataflow, which extracts the address data from the source system into a staging area. The Validation
function will check to see whether the city in the migrated record has Paris
as a value, and if it does, it will send the records to a separate reject table.
Getting ready
First, we need to create another schema in our STAGE
database to contain reject tables. Creating the Reject
schema to store these tables allows the keeping of the original table names; that makes writing queries and reporting...