Task 9 – Separating droppable data from the rest of the data processing
Under normal circumstances, data flowing in a pipeline does not change its status regarding being late, droppable, or on time. However, the exceptions to this are as follows:
- Data could change its status if we change our
WindowFnobject and re-window our stream, thereby producing different points in time that define the window GC time. - Data could change its status if we apply logic with a more sensitive definition of droppable data – this specifically applies to
@RequiresTimeSortedInput, where droppable data becomes every data element that is – at any point in time – more behind the watermark than the defined allowed lateness.
We can rephrase these conditions so that as long as we do not change the window function and do not apply logic with specific requirements, the droppable status of an element should not change between transforms. We will use this property to...