At the time of this writing, the stable version of Kafka is 0.8.1. The prerequisites for running Kafka is a ZooKeeper ensemble and Java Version 1.6 or above. Kafka comes with a convenience script that can start a single-node ZooKeeper, but it is not recommended to use it in a production environment. We will be using the ZooKeeper cluster we deployed in the Setting up a ZooKeeper cluster section of Chapter 2, Setting Up a Storm Cluster.
We will see both how to set up a single-node Kafka cluster first and how to add two more nodes to it to run a full-fledged three-node Kafka cluster with replication enabled.
The following are the steps to set up a single-node Kafka cluster:
Download the Kafka 0.8.1.1 binary distribution named
kafka_2.8.0-0.8.1.1.tgz
from http://kafka.apache.org/downloads.html.Extract the archive to where you want to install Kafka with the following command:
tar -xvzf kafka_2.8.0-0.8.1.1.tgz cd kafka_2.8.0-0.8.1.1
We will...