Producing and consuming with Python
You can create producers and consumers for Kafka using Python. There are multiple Kafka Python libraries – Kafka-Python, PyKafka, and Confluent Python Kafka. In this section, I will use Confluent Python Kafka, but if you want to use an open source, community-based library, you can use Kafka-Python. The principles and structure of the Python programs will be the same no matter which library you choose.
To install the library, you can use pip. The following command will install it:
pip3 install confluent-kafka
Once the library has finished installing, you can use it by importing it into your applications. The following sections will walk through writing a producer and consumer.
Writing a Kafka producer in Python
To write a producer in Python, you will create a producer, send data, and listen for acknowledgements. In the previous examples, you used Faker to create fake data about people. You will use it again to generate the data...