As previously mentioned, logging presents a challenge to many organizations. Gathering and processing log files is required for a number of reasons. It is used to watch for anomalous behavior as well as look for unauthorized activity.

For many years, a centralized syslog host was the most common method used to implement centralized logging. All of the hosts would ship their logs to one place and analysis was done there.

This worked OK for systems that used syslog for all logging. However, syslog has some drawbacks. It lacks a good way to deal with multiline records. Additionally, it only has a limited number of granularity levels so everything ended up logged in several giant log files. Attempts were made with various syslog agents to overcome some of these challenges, but there had to be a more complete way to handle the problem.

Enter Logstash. Logstash is nothing more than a system that takes input from multiple sources, parses it, and stores that output elsewhere...