Most of Apache Kafka's magic is achieved through configuration. As with all the intensive messaging systems, the success factor is to configure them well. In this point, Kafka is highly configurable. In practice, most of the systems have average performance with the default settings, but in production, it is required to configure it to achieve optimal performance. Sometimes, finding the right configuration is a test and error task; there is no such thing as a configuration silver bullet.
The rest of the chapter is about Kafka broker fine tuning.
In previous recipes, it was explained how to install and run Kafka. Now, make a copy of the server.properties template in the config directory and open the copy with a text editor.
- Configure the basic settings in the configuration file.
- Set each one of the following parameters with these values:
broker.id=0 listeners=PLAINTEXT://localhost:9093 log.dirs=/tmp/kafka-logs
As shown in the previous recipes, all of the broker definition is contained in the configuration file. The rest is to pass the configuration file as an argument to the server-start command.
A detailed explanation of every parameter is as follows:
broker.id: A non-negative integer; the default value is 0. The name should be unique in the cluster. The important point here is to assign a name to the broker, so when it is moved to a different host or to a different port, no change is made in the consumer's side.listeners: A comma-separated list of URIs the broker will listen on and the listener names. Examples of legal listener lists are:PLAINTEXT://127.0.0.1:9092,SSL://:9091,CLIENT://0.0.0.0:9092, andREPLICATION://localhost:9093.host.name: DEPRECATED. A string; the default value is null. If it is not specified, Kafka will bind all the interfaces on the system. If it is specified, it will bind only to the specified address. Set this name if you want the clients to connect to a particular interface.port: DEPRECATED. A non-negative integer; the default value is9092. It is the TCP port in which listen connections. Note that in the file template this value is not set.log.dir: A String; the default value is/tmp/kafka-logs. This is the directory where Kafka persists the messages locally. This parameter tells the directory where Kafka will store the data. It is very important that the user that runs the start command have write permissions on that directory.log.dirs: A String; the default value is null. This is the directory where Kafka persists the messages locally. If not set, the value inlog.diris used. There can be more than one location specified, separating the directories with a comma.
If bridged connections are used, it means that when the internal host.name and port are different from the ones which external parties need to connect to, this parameter is used:
advertised.listeners: The hostname given to producers, consumers, and other brokers specified to connect to. If it is not specified, it is the same ashost.name.