Introducing the primitive PTransform object – stateful ParDo
This section will focus on a theoretical description of the stateful ParDo transform, which is the most complex transform that a typical user of Beam needs to understand. We will divide this section into two subsections. First, we will describe its theoretical properties and the differences from the stateless version of ParDo. Then, we will see how this theoretical knowledge applies to API changes.
Describing the theoretical properties of the stateful ParDo object
As we have seen, the main difference between a stateful ParDo object and a stateless ParDo object is – as the name suggests – the presence of user state or timers. This alone brings one important requirement: every meaningful access to a state must be keyed. That is to say, the PCollection object we apply to a stateful ParDo object must be of the KV<K, V> type. We must assign a key to every element of a PCollection object (or use...