No parameter should be left at default when optimal behavior is desired. These parameters should be taken into consideration to achieve the best performance.
Adjust the following parameters:
auto.create.topics.enable=true controlled.shutdown.enable=true controlled.shutdown.max.retries=3 controlled.shutdown.retry.backoff.ms=5000 auto.leader.rebalance.enable=true leader.imbalance.per.broker.percentage=10 leader.imbalance.check.interval.seconds=300 offset.metadata.max.bytes=4096 max.connections.per.ip=Int.MaxValue connections.max.idle.ms=600000 unclean.leader.election.enable=true offsets.topic.num.partitions=50 offsets.topic.retention.minutes=1440 offsets.retention.check.interval.ms=600000 offsets.topic.replication.factor=3 offsets.topic.segment.bytes=104857600 offsets.load.buffer.size=5242880 offsets.commit.required.acks=-1 offsets.commit.timeout.ms=5000
Here is an explanation of these settings:
auto.create.topics.enable: Default value: true. Suppose that metadata is fetched or a message is produced for a nonexistent topic; if this value is true, the topic will automatically be created. In production environments, this value should be false.controlled.shutdown.enable: Default value: true. If this value is true, when a shutdown is called on the broker, the leader will gracefully move all the leaders to a different broker. When it is true, the availability is increased.controlled.shutdown.max.retries: Default value: 3. This is the maximum number of retries the broker tries a controlled shutdown before making a forced and unclean shutdown.controlled.shutdown.retry.backoff.ms: Default value: 5000. Suppose that a failure happens (controller fail over, replica lag, and so on); this value determines the time to wait before recovery from the state that caused the failure.auto.leader.rebalance.enable: Default value: true. If this value is true, the broker will automatically try to balance the partition leadership among the brokers. At regular intervals, a background thread checks and triggers leader balance if required, setting the leadership to the preferred replica of each partition if available.leader.imbalance.per.broker.percentage: Default value: 10. This value is specified in percentages and is the leader imbalance allowed per broker (the leader imbalance will be explained later). The cluster will rebalance the leadership if this percentage goes above the set value.leader.imbalance.check.interval.seconds: Default value: 300. This value is the frequency at which to check the leader imbalance by the controller.offset.metadata.max.bytes: Default value: 4096. This is the maximum size allowed to the client for a metadata to be stored with an offset commit.max.connections.per.ip: Default value: 2 147 483 647. This is the maximum number of connections that the broker accepts from each IP address.connections.max.idle.ms: Default value: 600 000. This is the idle connection's timeout. The server socket processor threads close the connections that idle more than this value.unclean.leader.election.enable: Default value: true. If this value is true, the replicas that are not ISR can become leaders. Doing so may result in data loss.offsets.topic.num.partitions: Default value: 50. This is the number of partitions for the offset commit topic. This value cannot be changed post deployment.offsets.retention.minutes: Default value: 1440. This is the log retention window for the offsets topic. This is the time to keep the offsets. Passed this, the offsets will be marked for deletion.offsets.retention.check.interval.ms: Default value: 60 000. This is the frequency at which to check for stale offsetsoffsets.topic.replication.factor: Default value: 3. This is the number of replicas for the offset commit topic. The higher this value, the higher the availability. As shown in the previous recipes, if the number of brokers is lower than the replication factor, the number of replicas will be equal to the number of brokers.offsets.topic.segment.bytes: Default value: 104 857 600. This is the segment size for the offsets topic. The lower this value, the faster the log compaction and cache loading are.offsets.load.buffer.size: Default value: 5 242 880. This is the batch size to be used for reading offset segments when loading offsets into the cache.offsets.commit.required.acks: Default value: -1. This is the number of acknowledgements required before the offset commit can be accepted. It is recommended to not override the default value of -1, meaning no acknowledgements required.offsets.commit.timeout.ms: Default value: 5000. This is the time that an offset commit will be delayed until all replicas for the offsets topic receive the commit or this time value is reached. This value is similar to the producerrequest.timeout.ms.
- There are more broker configurations that are available. Read more about them at: http://kafka.apache.org/documentation.html#brokerconfigs