No parameter should be left by default when the optimal performance is desired. These parameters should be taken into consideration to achieve the best behavior.
Adjust the following parameters:
message.max.bytes=1000000 num.network.threads=3 num.io.threads=8 background.threads=10 queued.max.requests=500 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.partitions=1
With these changes, the network and performance configurations have been set to achieve optimum levels for the application. Again, every system is different, and you might need to experiment a little to come up with the optimal one for a specific configuration.
Here is an explanation of every parameter:
message.max.bytes: Default value: 10 00 000. This is the maximum size, in bytes, for each message. This is designed to prevent any producer from sending extra large messages and saturating the consumers.num.network.threads: Default value: 3. This is the number of simultaneous threads running to handle a network's request. If the system has too many simultaneous requests, consider increasing this value.num.io.threads: Default value: 8. This is the number of threads for Input Output operations. This value should be at least the number of present processors.background.threads: Default value: 10. This is the number of threads for background jobs. For example, old log files deletion.queued.max.requests: Default value: 500. This is the number of messages queued while the other messages are processed by the I/O threads. Remember, when the queue is full, the network threads will not accept more requests. If your application has erratic loads, set this to a value at which it will not throttle.socket.send.buffer.bytes: Default value: 102 400. This isSO_SNDBUFFbuffer size, used for socket connections.socket.receive.buffer.bytes: Default value: 102 400. This isSO_RCVBUFFbuffer size, also used for socket connections.socket.request.max.bytes: Default value: 104 857 600. This is the maximum request size, in bytes, that the server can accept. It should always be smaller than the Java heap size.num.partitions: Default value: 1. This is the number of default partitions of a topic, without giving any partition size.