Task 13 – Writing a reusable PTransform – StreamingInnerJoin

Our goal in this task is to create a reusable PTransform that will do the fully stream-oriented join. For simplicity, we will constrain ourselves to the inner join as the implementation of the full outer join would become a little lengthy. We will mention the differences at the end of this section, though.

Problem definition

Implement a PTransform that will be applied to a PCollectionTuple with exactly two TupleTags (leftHandTag and rightHandTag) and will produce a streaming join operation on top of them. The transform needs to take functions to extract join keys and both sides' primary keys. The output will contain the (inner) joined values and any retractions.

Problem decomposition discussion

We described the theoretical properties of a streaming join in the previous section, so we will focus on describing the semantics in practical examples. We will then use these examples to test our solution...