Before we talk about replication in Kafka, let's talk about message partitioning. In Kafka, a message partitioning strategy is used at the Kafka broker end. The decision about how the message is partitioned is taken by the producer, and the broker stores the messages in the same order as they arrive. The number of partitions can be configured for each topic within the Kafka broker.

Kafka replication is one of the very important features introduced in Kafka 0.8. Though Kafka is highly scalable, for better durability of messages and high availability of Kafka clusters, replication guarantees that the message will be published and consumed even in the case of broker failure, which may be caused by any reason. Both producers and consumers are replication-aware in Kafka. The following diagram explains replication in Kafka:

Let's discuss the preceding diagram in detail.

In replication, each partition of a message has n replicas and can afford n-1 failures to guarantee message...