The traffic on production servers is supposed to be high at times. When this happens, log entries and statistics data become very critical and the amount of such data in total is also high. All of the Beats will be doing their work and will be sending respective data to Elasticsearch, but it is possible that some of the packets/data are lost while processing. It may happen because of a network failure, very high peaks of data, or any other possible reason. The point is, the data being indexed must not be lost in any case.
To address this problem, using a message broker or buffer is a good choice. There are many tools that can be evaluated for your choice of the message broker. For Open Source, there are two good tools available:
Redis: http://redis.io/
Kafka: https://kafka.apache.org/
Sometimes, it seems that message broker is a must for a production environment, but there might be cases when we don't need any message broker at all. When we use Filebeat, it acts as...