In this section, we will take a detailed look at an example application that illustrates the use of dynamic partitioning of an operator. It uses an input operator that generates random numbers and outputs them to a DevNull
library operator (which, as the name suggests, simply discards them). The input operator starts out with two partitions; after some tuples have been processed, a dynamic repartition is triggered via the StatsListener
interface discussed above to increase the number of partitions to four. The source code is available atthe following link: https://github.com/apache/apex-malhar/tree/master/examples/dynamic-partition.
The populateDAG()
method is, as expected, very simple:
@Override public void populateDAG(DAG dag, Configuration conf) { Gen gen = dag.addOperator("gen", Gen.class); DevNull devNull = dag.addOperator("devNull", DevNull.class); dag.addStream("data", gen.out, devNull.data); }
The interesting code...