Spark Streaming is built on top of Spark core engine and can be used to develop a fast, scalable, high throughput, and fault tolerant real-time system. Streaming data can come from any source, such as production logs, click-stream data, Kafka, Kinesis, Flume, and many other data serving systems.
Spark streaming provides an API to receive this data and apply complex algorithms on top of it to get business value out of this data. Finally, the processed data can be put into any storage system. We will talk more about Spark Streaming integration with Kafka in this section.
Basically, we have two approaches to integrate Kafka with Spark and we will go into detail on each:
- Receiver-based approach
- Direct approach
The receiver-based approach is the older way of doing integration. Direct API integration provides lots of advantages over the receiver-based approach...