In this chapter we learned how Elixir elegantly tackles the problem of processing events in stages by analyzing the GenStage
and Flow
abstractions:
- We implemented our media upload and download pipelines with
GenStage
, putting to good use its elegant back-pressure mechanism. To achieve this, we had to implement the needed callbacks on the different modules behind the pipeline stages and start each pipeline supervisor on the application initialization. - We then examined
Flow
, an abstraction based onGenStage
that lets you process collections in a lazy and concurrent fashion, while keeping much of the semantics one associates with theEnum
andStream
modules.
The advantage of using GenStage
is that no step in the media pipeline will ever be overwhelmed with requests coming from upstream. If you want to process large datasets or when the bulk of the processing is CPU- or I/O-bound, that's when Flow
shines. In this day and age, where multi-core processors are the norm, having a tool like...