The log event processor example covered in this chapter was listening to a TCP socket for the stream of messages to be processed by the Spark Streaming data processing application. But in real-world use cases, this is not going to be the case.
Message queueing systems with publish-subscribe capability are generally used for processing messages. The traditional message queueing systems failed to perform because of the huge volume of messages to be processed per second for the needs of large-scale data processing applications.
Kafka is a publish-subscribe messaging system used by many IoT applications to process a huge number of messages. The following capabilities of Kafka made it one of the most widely used messaging systems:
Extremely fast: Kafka can process huge amounts of data by handling reading and writing in short intervals of time from many application clients
Highly scalable: Kafka is designed to scale up and scale out to form a cluster using commodity hardware...