Building publish/subscribe messaging using Kafka
As with RabbitMQ, Apache Kafka is an asynchronous messaging tool used by applications to send and store messages between producers and consumers. However, it is faster than RabbitMQ because it uses topics with partitions where producers can append various types of messages across these minute folder-like structures. In this architecture, the consumers can consume all these messages in a parallel mode, unlike in queue-based messaging, which enables producers to send multiple messages to a queue that can only allow message consumption sequentially. Within this publish/subscribe architecture, Kafka can handle an exchange of large quantities of data per second in continuous and real-time mode.
There are three Python extensions that we can use to integrate the FastAPI services with Kafka, namely the kafka-python, confluent-kafka, and pykafka extensions. Our online newsstand prototype will use kafka-python, so we need to install it using...