As a deployment of OSSEC expands and matures, so does the volume of alerts. Normal activities, such as system packages updates, updates to critical configuration files, and users' interaction with those systems, generate alerts. This section will explore the possibility of using active response to assist with the verification of changes to a user's list of scheduled tasks (crontab).
Shortly after enabling OSSEC on my network, I began receiving alerts for "normal" changes to the environment. Our configuration management software, Puppet, was generating alerts by updating the crontab of the root user. When Puppet manages a user's crontab, it prepends a header to the file as follows:
# HEADER: This file was autogenerated at 2013-03-01 13:30:53 +0100 by puppet.
Implementing a script to check Puppet's modification time against the crontab modification time isn't difficult. We just need to know how to leverage the entire OSSEC infrastructure.
Unfortunately, OSSEC does not provide a decoder to extract fields from crontab log messages. Luckily, we can easily extend the decoders! We'll add a small decoder to handle crontab messages and extract the username.
To verify alerts with active response, perform the following steps:
Set up decoders for our crontab log entries and our new logging target
ossec-ar-verify. Then place them in the/var/ossec/etc/decoders.ddirectory and tell OSSEC to look there for our custom decoders. Also, add thecommandandactive-responseelements to theossec.conffiles:<command> <name>ossec-ar-verify-crontab</name> <executable>ossec-ar-verify-crontab.sh</executable> <expect>username</expect> <timeout_allowed>no</timeout_allowed> </command> <active-response> <command>ossec-ar-verify-crontab</command> <location>local</location> <rules_id>106001</rules_id> </active-response> <rules> <!-- OSSEC Decoders --> <decoder>etc/decoder.xml</decoder> <!-- Load my custom decoders --> <decoder_dir>etc/decoders.d</decoder_dir> <!-- Rest of rules --> </rules>Next, set up the
crontab.xmldecoder:<decoder name="crontab"> <program_name>crontab</program_name> </decoder> <decoder name="crontab-user"> <parent>crontab</parent> <regex>^\((\S+)\) </regex> <order>user</order> </decoder>
After that, set up the
ossec-ar-verify.xmldecoder:<decoder name="ossec-ar-verify"> <program_name>ossec-ar-verify</program_name> </decoder> <decoder name="ossec-ar-verify-crontab"> <parent>ossec-ar-verify</parent> <prematch>^crontab </prematch> <regex offset="after_prematch">^(\S+)</regex> <order>user</order> </decoder>
Then define the rules to handle the triggering of the active response and then interpret the results in our
local_rules.xmlfile:<rule id="106001" level="1"> <if_sid>2833</if_sid> <options>no_email_alert</options> <description>Verify the crontab modification</description> </rule> <rule id="107000" level="11"> <decoded_as>ossec-ar-verify</decoded_as> <description>OSSEC ActiveResponse Verification</description> <group>verified,</group> </rule> <rule id="107001" level="12"> <if_sid>107000</if_sid> <match>^crontab </match> <options>alert_by_email</options> <description>Verified: Crontab changed outside of Puppet</description> <group>oob_change,scheduled_tasks</group> </rule>Also, distribute a Perl script to handle your date math with a wrapper shell script. The wrapper script
ossec-ar-verify.shreferenced in our command declaration is as follows:#!/bin/sh USER=$2 CRONTAB="/var/spool/cron/$USER" if [ -f "$CRONTAB" ]; then /usr/local/bin/ossec-ar-verify-crontab.pl $CRONTAB rc=$? if [ "$rc" != "0" ]; then logger -t ossec-ar-verify "crontab $USER change outside Puppet" fi fi
OSSEC includes the rule 2833 to identify when the root's crontab changes. We need to take action on this alert and generate another more serious alert if the validation against Puppet fails. The steps to verify the alert are as follows:
If the rule
2833fires, silence it.Run an active response command on the agent generating the alert.
If validation fails, inform OSSEC via syslog.
OSSEC should understand the new notification and alert via e-mail.
For step 1, rule 2833 uses the alert_by_email option to always send an e-mail regardless of the level of the alert. Since we're going to be verifying this alert, we need to disable that e-mail. The best way to do this is to create a new rule (100601) in our local_rules.xml file to turn off the e-mail alert.
To enable the active response required in step 2, we need three things: a script to run, a command declaration in the ossec.conf file, and an active-response declaration in ossec.conf. In simple words, the script we're calling is a wrapper around another script that checks the file modification time of the crontab file against the date of the header provided by Puppet. If they vary by more than 1 minute, the script exits with an error; if the times match, the script exits successfully. Our wrapper script uses these exit codes to determine whether or not to generate a new syslog message by calling the logger program. If the dates match, no message is generated; if they don't, the following command is called:
logger -t ossec-ar-verify "crontab root changed outside puppet"
This may be the first of many verification scripts, so we create a namespace for this purpose; ossec-ar-verify is descriptive and succinct. We name this command ossec-ar-verify-crontab with an absolute path of /var/ossec/active-response/bin/ossec-ar-verify-crontab.sh. This script requires a username to be set so we know whose crontab to verify. Since it's only being run once per alert, there's no need to use the timeout features; so we've disabled them. Next we have to configure the active-response element to use this command.
We created a child of rule 2833, and the source ID of that new rule is 106001. This script needs to be run at the source of the event, so we configure the location of this active response to be local. The command variation is set to ossec-ar-verify-crontab.
We need events to pass a username to the active response, but OSSEC doesn't ship with a decoder for crontab log messages. Without it, our active response would never fire and we'd silently discard all these messages. We create a crontab.xml decoder file and place it in the /var/ossec/etc/decoder.d directory we configured to be loaded in the ossec.conf file. This decoder extracts the username from the crontab log messages to pass to the active response.
For step 3, we're going to simply send a new log message via syslog if the verification fails, which will look similar to the following:
Jun 2 12:57:38 host ossec-ar-verify[640]: crontab changed outside Puppet
We create an ossec-ar-verify.xml file in our /var/ossec/etc/decoder.d directory to extract fields from messages to our new syslog target ossec-ar-verify. The decoder parent will be named ossec-ar-verify, so we can use the high-speed decoded_as attribute for anchoring rules.
The two decoders work together to set the decoded_as attribute in the event and extract the user attribute from the crontab alerts. The regex attribute in our second decoder line extracts the first word after the prematch element (crontab in our example). The order element assigns names to the variable extracted using grouping parentheses. We're only extracting a single value named user.
If the modification times don't match, step 4 of our model is triggered by our log messages. We create a rule 107000 to anchor any current and future ossec-ar-verify messages; this rule also appends a verified group to the alert that we can use for reporting. The rule 107001 matches messages starting with the word crontab. It sets the alert_by_email option to notify the administrators. It also appends the oob_change (out of band change) and scheduled_tasks groups to the alert for reporting.
Hopefully, this example will make you wonder, "Couldn't I extend this same pattern to verify file checksum changes against our package manager and/or configuration engine?". Yes! Given the scriptability of the active response system, you are not limited by vendor lock-in and can incorporate data sources for verification from anywhere in your organization. You are limited only by your own creativity!