OSSEC ships with a rootkit detection module that looks specifically for traces of rootkits, malware, and Trojans on configured systems. This recipe looks briefly at configuring the rootcheck module.
The OSSEC syscheck daemon runs the rootkit module. There are a few components of the rootkit detection module, all configurable individually. They are as follows:
File detection: This component looks for malicious files at known locations
/dev check: This component looks in
/devfor executable filesPID check: This component looks for processes hidden from
psPort check: This component looks for open ports hidden from
netstatInterface check: This component looks for an interface in promiscuous mode
System scan: This component looks for anomalies in the filesystem, that is, for bad permissions and strange SUID (set user ID) files
Policy checks: This component looks for weird items in files; its submodules are as follows:
Trojans check: This module looks for well-known Trojans on the system
Windows malware: This module looks for well-known malware
Windows application: This module looks for bad applications
System audit: This module looks for bad configurations that would be a violation of the best practices according to industry standards
Some of these checks, like the interface check, are really simple and quick while others like the policy check can be quite resource intensive.
Here are some guidelines for detecting rootkits and anomalies:
Check OSSEC's default configuration of the
rootcheckdaemon inossec.conf:<rootcheck> <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files> <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans> </rootcheck>To completely disable the rootkit module, change the
rootchecksection to the following:<rootcheck> <disabled>yes</disabled> </rootcheck>Disabling the rootkit detection module isn't ideal, but there are some exceptional cases we might need to address. A better configuration for a CPU-bound system is as follows:
<rootcheck> <frequency>172800</frequency> <!-- Disable CPU heavy checks due to network connections --> <check_ports>no</check_ports> <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files> <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans> </rootcheck>If we had a disk-bound system, it might be wise to disable the intensive disk checks as follows:
<rootcheck> <frequency>172800</frequency> <!-- Disable high disk IO Checks --> <check_policy>no</check_policy> <check_sys>no</check_sys> <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files> <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans> </rootcheck>
OSSEC's default configuration enables all the rootkit checks. With the rest of the modules running, we adjust the frequency attribute from the default value of every 20 hours to every 48 hours.
There are a few heavy checks that are enabled depending on your system profiles. On systems that use network filesystems (NFS), the rootcheck daemon may cause performance problems while running checks that read data from the filesystem.
The check_policy attribute actually controls the policy checks, including the trojans, winmalware, winapps, unixaudit, and winaudit checks. These checks inspect the contents of files on the filesystem, looking for misconfiguration. You can disable one or more of these checks individually, but as they scan the same places, it's best to disable the entire set.
The system scan can also lead to high-disk input/output and degrade system performance, though it's very rare. We can disable it by setting check_sys to no in the rootcheck section as we previously saw. This check isn't opening a lot of files but is using the stat system call to look at permissions. You may have guessed that this is more expensive on network filesystems but also on systems with a large number of files.
For systems that have a large number of network connections open at the same time, you may find that the port check component can take a lot of CPU. Port check uses netstat to determine either currently open ports or closed ports with live connections. On systems with several thousand concurrent, open connections, this check can actually impact production performance. You can disable this check by setting check_ports to no in the rootcheck section.
There may be an overlap where you have both a large number of connections as well as network-mounted filesystems. Disabling individual checks is not exclusive, so you can disable as many as you need to get the performance to reasonable levels.
The rootcheck daemon can be configured to do more precise or more sweeping scans.
The rootcheck module uses a file configuration syntax that can and has been extended to provide the system audit capability. It provides a notation for "if this directory contains an executable" and "if this file contains this line". It can audit the configuration of critical services and identify violations of corporate or security policy.
There are some servers that keep administrators up at night. If these systems have resources to spare, you can enable a "paranoid" mode on the rootcheck module. This setting does away with the notion of "look in well-known places for bad things" and enables "look everywhere on the system for bad things". While this sounds spectacular, it is very resource intensive and should be used only where necessary.
To enable the "paranoid" mode, configure the rootcheck module as follows:
<rootcheck>
<!-- Enable Scanning of ALL THE THINGS! -->
<scanall>yes</scanall>
<frequency>86400</frequency>
<!-- etc, .. -->
</rootcheck>I recommend setting a custom frequency attribute when you enable scanall, if only to make yourself aware. Use periods that are common to monitoring systems: days, weeks, or months. The default of 20 hours can drive administrators insane trying to track down why the server is under high load at seemingly random times on their graphs.