Now that we know what types of data inputs there are, let's say that you have 500 Forwarders and they are different parts of unique systems. How do you manage all of that?
I've got three words for you: Splunk deployment server.
If you're not familiar with Splunk deployment server, I highly recommend you become familiar. With a large deployment of Splunk it's surely the easiest way to manage all of your data inputs for your various systems:
Basics: As a general rule of thumb, in Splunk best practices, in Splunk architecture, there should be at least one deployment server. That deployment server would sit behind a load balancing device (let's use F5) and have its own DNS address.
Reason: Because if anything ever happens to your DS, and it has a catastrophic failure, what happens when you need to spin up a new one and you can't have the same IP address? Assuming that you don't have a system such as Puppet, Chef, or StackIQ to use to manage your infrastructure, that means you...