OSSEC treats command output as log entries. OSSEC has two options for command monitoring: command and full_command. The difference is how OSSEC handles the output. When using the command variation, every line of output is treated as an individual log entry and analyzed independently. When using the full_command variation, the entire output is regarded as a single log entry.

OSSEC's internal rules match the command output using the source ID 530 and prefix each log entry with the alias or command that is run as follows:

ossec: output: 'df -h' ...
ossec: output: 'my-command-alias' ...

We'll use this knowledge to write rules to handle the output of two commands: one to monitor changes to listening ports with netstat and another to monitor disk usage with df.

With this rule configured, if the listening ports change, we'll receive an e-mail informing us of the current and previous output of the netstat command.

The commands are defined in the ossec.conf file, similar to logfiles, except they require a command attribute to tell OSSEC what command and arguments to execute. They use the log_format element having value as the command or full_command variation so OSSEC knows how to execute the command and how to handle the output. We're using an alias for these declarations so the rules will be easier to write.

In both our command declarations, we used command pipes, |, to either exclude or format the output of our commands to make alerts more relevant. The df example uses awk to completely rewrite the output of the df command; this makes it easier to read and match using OSSEC's regex/match capabilities. Let's break it down:

df -l -x tmpfs |grep -v '^Filesystem'|awk '{print $1 " mounted as " $6 " usage is " $5 }'

We use the df command to list the capacity and usage of our locally mounted filesystems. We skip the nonlocally mounted filesystems because if an NFS volume fills up, it will alert potentially every server in our infrastructure. We're also excluding the shared memory filesystem, tmpfs. The output of this command on my server is as follows:

$ df -l -x tmpfs
Filesystem   1K-blocks     Used Available Use% Mounted on
/dev/xvda     20421052  9134172  10458508  47% /
/dev/xvdc     16513960  7980512   7694588  51% /home

Since we're using the command option, each line is treated as its own log entry. If this is the case, the header to the df command isn't really worth analyzing, so we remove it using grep -v '^Filesystem' to exclude the lines starting with the word 'Filesystem'.

$ df -l -x tmpfs |grep -v '^Filesystem'
/dev/xvda     20421052  9134172  10458508  47% /
/dev/xvdc     16513960  7980512   7694588  51% /home

The output is usable, but it's not as simple as it could be. So using awk we rewrite each line by printing the first word ($1), " mounted as ", the sixth word ($6), " usage is ", and finally the fifth word ($5). The awk command's defaults interpret the white space as word separators.

 $ df -l -x tmpfs |grep -v '^Filesystem'\
 >   |awk '{print $1 " mounted as " $6 " usage is " $5 }'
/dev/xvda mounted as / usage is 47%
/dev/xvdc mounted as /home usage is 51%

Using this format, it's easier to write rules and easier for administrators receiving alerts to understand what they mean. We can then start writing rules to notify administrators as follows:

<rule id="100100" level="2">
  <if_sid>530</if_sid>
  <match>ossec: output: 'disk-usage'</match>
  <group>system_availability,</group>
</rule>

The first rule for the disk-usage command anchors the rest of the rules as their parent. It also appends the system_availability group to the alert. We could use this group to route alerts to different e-mails, active responses, or look at aggregates. The next rule that we'll write looks at 90 to 99 percent disk usage:

<rule id="100101" level="12">
  <if_sid>100100</if_sid>
  <regex>usage is 9\d</regex>
  <description>Critically high disk usage.</description>
</rule>

This rule will alert a high-importance event at level 12. To match this, the rule anchors to the rule 100100 with the if_sid declaration. The regex element is used to look for usage is 9 followed by the other digit \d—when the usage is 90 to 99 percent. Once the disk goes to 100 percent, this rule will stop matching; so, we'll need another rule to handle this special case:

<rule id="100102" level="13">
  <if_sid>100100</if_sid>
  <match>usage is 100</match>
  <options>alert_by_email</options>
  <description>Disk is full, availability in jeopardy.</description>
</rule>

This rule again anchors itself to the parent rule, 100100. We can't have a disk at 101 percent usage or higher so we look for 100 percent only. The alert level is raised to 13 (unusual error), and we explicitly set the alert_by_email option so we can be assured that this alert will always generate an e-mail regardless of our other e-mail and report settings.

Our next command monitors the listening of the TCP and UDP ports using netstat.

 netstat -nltu 

This calls the netstat command with the following options: do not look up hostnames (-n), show only listening sockets (-l), show TCP sockets (-t), and show UDP sockets (-u).

To accomplish our goal, it's not necessary to transform or decode the output of the netstat command. A simple difference between the current and previous run is sufficient. We can achieve this with a single rule:

<rule id="100200" level="2">
  <if_sid>530</if_sid>
  <match>ossec: output: 'netstat-listening'</match>
  <check_diff/>
  <options>alert_by_email</options>
  <group>network_services,</group>
</rule>

We check the source ID 530, the source ID for the command output, and the match using the alias for the command netstat-listening. We add the network_services group to the alerts. To check for differences, you need only specify the check_diff attribute to the rule. The alert level is 2 (system information), but we really want to know about these events so we set the alert_by_email flag on the alert.