Summary

In this chapter, we walked through how runners execute pipelines in both classic mode and using the portability layer. We have seen that classic runners are suitable only for cases where a particular underlying technology – for instance, Apache Flink – has an API in the same language as the pipeline SDK. The most practical cases for this include using the Java SDK for both the runner and the pipeline.

In cases where the language of the runner and the pipeline SDK differ, we have to use portability (Fn API), which brings some overhead. We have seen how pipeline fusion is used to reduce this overhead as much as possible. We have also discussed situations where we want to prevent fusion and how to do this by inserting a shuffle boundary.

Next, we discussed the responsibilities of a runner with regard to state management. We saw how the runner ensures fault tolerance and correctness upon failures. We outlined two basic types of fault-tolerant states: local state...