The following figure provides the data flow between the Spark driver, workers, streaming sources and targets:
It all starts with the Spark Streaming Context, represented by ssc.start() in the preceding figure:
When the Spark Streaming Context starts, the driver will execute a long-running task on the executors (that is, the Spark workers).
The Receiver on the executors (Executor 1 in this diagram) receives a data stream from the Streaming Sources. With the incoming data stream, the receiver divides the stream into blocks and keeps these blocks in memory.
These blocks are also replicated to another executor to avoid data loss.
The block ID information is transmitted to the Block Management Master on the driver.
For every batch interval configured within Spark Streaming Context (commonly this is every 1 second), the driver will launch Spark tasks to process the blocks. Those blocks are then persisted to any number of target data stores, including...