Often, we feel the need to work on some data and pass it along a series of steps, transforming it along the way before it arrives at its destination. We come across these sort of processes occurring in real-life scenarios, especially in factory assembly line environments.
In this chapter, we will see how the pipeline patterns can be used to build component-based applications. We'll see how we can use function composition data flow programming techniques to create flexible solutions that are not only robust, but also performant in today's distributed processing environments.
Our goal in this chapter is to do the following:
- Be able to identify when to use the pipeline pattern
- Learn how to build a pipeline
- Understand how we can leverage buffering to increase throughput
- Use Goroutines and channels to process data faster
- Improve API readability using interfaces
- Implement useful filters
- Build a flexible pipeline
- See what happens when you change the order...