Based on the concepts that we recently introduced about Pipe-and-Filter in the field of enterprise applications, we use this kind of architecture in several scenarios in order to process a large amount of data (or large files) that trigger several steps (or tasks) that need to be processed. This architecture is highly beneficial when we need to perform a lot of transformations in the data.
To understand how Pipe-and-Filter works, we are going to review a classic example of processing payroll records. In this example, a message is being sent through a sequence of filters, where each filter processes the message in different transactions.
When we apply a Pipe-and-Filter approach, we decompose the whole process into a series of separate tasks that can be reused. Using these tasks, we can change the format of the received message, and then we can split it to execute separate transactions. As a benefit of doing this, we improve the performance, scalability...