Kafka has an architecture that differs significantly from other messaging systems. Kafka is a peer-to-peer system in which each node is called a broker. The brokers coordinate their actions with the help of a ZooKeeper ensemble.
A Kafka cluster
The following are the important components of Kafka.
In Kafka, messages are published by a producer to named entities called topics. A topic is a queue that can be consumed by multiple consumers. For parallelism, a Kafka topic can have multiple partitions. Reads and writes can happen to each partition in parallel. Data for each partition of a topic is stored in a different directory on the disk. Each of these directories can be on different disks, allowing us to overcome the I/O limitations of a single disk. Also, two partitions of a single topic can be allocated on different brokers, thus increasing throughput as each partition is independent of each other. Each message in a partition has a unique sequence number...