Understanding how Kafka uses logs
Apache Kafka uses logs to store data – records. Logs in Kafka are called topics. A topic is like a table in a database. In the previous chapter, you tested your Kafka cluster by creating a topic named
dataengineering. The topic is saved to disk as a log file. Topics can be a single log, but usually they are scaled horizontally into partitions. Each partition is a log file that can be stored on another server. In a topic with partitions, the message order guarantee no longer applies to the topic, but only each partition. The following diagram shows a topic split into three partitions:
The preceding topic – Transactions – has three partitions labeled P1, P2, and P3. Within each partition, the...