This chapter explains the basic recipes to get started with Apache Kafka. It discusses how to install, configure, and run Kafka. It also discusses how to make basic operations with a Kafka broker.
Kafka can run in several operating systems: Mac, Linux, and even Windows. As it usually runs in production on Linux servers, the recipes in this book are designed to run in Linux environments. This book also considers the bash environment usage.
Kafka scales very well in a horizontal fashion without compromising speed and efficiency.
This chapter explains how to install, configure, and run Kafka. As this is a practical recipes book, it does not cover the theoretical details of Kafka. These three things are enough theory for the moment:
- To connect heterogeneous applications, we need to implement a mechanism for message publishing by sending and receiving messages among them. A message router is known as a message broker. Kafka is a software solution to deal with routing messages among consumers in a quick way.
- The message broker has two directives: the first is to not block the producers, and the second is to isolate producers and consumers (the producers should not know who their consumers are).
- Kafka is two things: a real-time, publish-subscribe solution, and a messaging system. Moreover, it is a solution: open source, distributed, partitioned, replicated, commit-log based, with a publish-subscribe schema.
Before we begin it is relevant to mention some concepts and nomenclature in Kafka:
- Broker: A server process
- Cluster: A set of brokers
- Topic: A queue (that has log partitions)
- Offset: A message identifier
- Partition: An ordered and immutable sequence of records continually appended to a structured commit log
- Producer: Those who publish data to topics
- Consumer: Those who process the feed
- ZooKeeper: The coordinator
- Retention period: The time to keep messages available for consumption
In Kafka, there are three types of clusters:
- Single node: Single broker
- Single node: Multiple Broker
- Multiple node: Multiple Broker
There are three ways to deliver messages:
- Never redelivered: The messages may be lost
- May be redelivered: The messages are never lost
- Delivered once: The message is delivered exactly once
There are two types of log compaction:
- Coarse grained: By time
- Finer grained: By message
The next six recipes contain the necessary steps to make a full Kafka test from zero.