In most cases, the data distribution of every Forwarder is handled automatically within any Forwarder; however in some instances it is necessary to tell Splunk to force appropriate data distribution across the indexing layer. This is especially necessary when dealing with extremely large events and multiple indexing peers.

Let's take a clustered environment as an example:

Let's say that two of our API's are pulling data from systems that have events in excess of 10,000 lines per event, and they are updating pulling this data at 60-second intervals.

While Splunk can certainly do this, and a Heavy Forwarder can handle the data throughput, there are a few settings that need to be in place to make sure not only that performance is optimized, but also that the data is being load-balanced properly, is not being truncated before the data can finish.

There are a few things to know when consuming very large datasets at fast intervals: