Product SiteDocumentation Site

14.3. Supervision: Prevention, Detection, Deterrence

Monitoring is an integral part of any security policy for several reasons. Among them, that the goal of security is usually not restricted to guaranteeing data confidentiality, but it also includes ensuring availability of the services. It is therefore imperative to check that everything works as expected, and to detect in a timely manner any deviant behavior or change in quality of the service(s) rendered. Monitoring activity can enable the detection of intrusion attempts and enable a swift reaction before they cause grave consequences. This section reviews some tools that can be used to monitor several aspects of a Debian system. As such, it completes the section dedicated to generic system monitoring in Chapter 12, Advanced Administration.

14.3.1. Monitoring Logs with logcheck

The logcheck program monitors log files every hour by default. It sends unusual log messages in emails to the administrator for further analysis.
The list of monitored files is stored in /etc/logcheck/logcheck.logfiles; the default values work fine if the /etc/syslog.conf file has not been completely overhauled.
logcheck can work in one of three more or less detailed modes: paranoid, server and workstation. The first one is very verbose, and should probably be restricted to specific servers such as firewalls. The second (and default) mode is recommended for most servers. The last one is designed for workstations, and is even terser (it filters out more messages).
In all three cases, logcheck should probably be customized to exclude some extra messages (depending on installed services), unless the admin really wishes to receive hourly batches of long uninteresting emails. Since the message selection mechanism is rather complex, /usr/share/doc/logcheck-database/README.logcheck-database.gz is a required — if challenging — read.
The applied rules can be split into several types:
  • those that qualify a message as a cracking attempt (stored in a file in the /etc/logcheck/cracking.d/ directory);
  • those canceling such a qualification (/etc/logcheck/cracking.ignore.d/);
  • those classifying a message as a security alert (/etc/logcheck/violations.d/);
  • those canceling this classification (/etc/logcheck/violations.ignore.d/);
  • finally, those applying to the remaining messages (considered as system events).
A system event is always signaled unless a rule in one of the /etc/logcheck/ignore.d.{paranoid,server,workstation}/ directories states the event should be ignored. Of course, the only directories taken into account are those corresponding to verbosity levels equal or greater than the selected operation mode.

14.3.2. Monitoring Activity

14.3.2.1. In Real Time

top is an interactive tool that displays a list of currently running processes. The default sorting is based on the current amount of processor use and can be obtained with the P key. Other sort orders include a sort by occupied memory (M key), by total processor time (T key) and by process identifier (N key). The k key allows killing a process by entering its process identifier. The r key allows renicing a process, i.e. changing its priority.
When the system seems to be overloaded, top is a great tool to see which processes are competing for processor time or consume too much memory. In particular, it is often interesting to check if the processes consuming resources match the real services that the machine is known to host. An unknown process running as the www-data user should really stand out and be investigated, since it's probably an instance of software installed and executed on the system through a vulnerability in a web application.
top is a very flexible tool and its manual page gives details on how to customize its display and adapt it to one's personal needs and habits.
The gnome-system-monitor and qps graphical tools are similar to top and they provide roughly the same features.

14.3.2.2. History

Processor load, network traffic and free disk space are information that are constantly varying. Keeping a history of their evolution is often useful in determining exactly how the computer is used.
There are many dedicated tools for this task. Most can fetch data via SNMP (Simple Network Management Protocol) in order to centralize this information. An added benefit is that this allows fetching data from network elements that may not be general-purpose computers, such as dedicated network routers or switches.
This book deals with Munin in some detail (see Section 12.4.1, “Setting Up Munin”) as part of Chapter 12: “Advanced Administration. Debian also provides a similar tool, cacti. Its deployment is slightly more complex, since it is based solely on SNMP. Despite having a web interface, grasping the concepts involved in configuration still requires some effort. Reading the HTML documentation (/usr/share/doc/cacti/html/index.html) should be considered a prerequisite.

14.3.3. Detecting Changes

Once the system is installed and configured, and barring security upgrades, there's usually no reason for most of the files and directories to evolve, data excepted. It is therefore interesting to make sure that files actually do not change: any unexpected change would therefore be worth investigating. This section presents a few tools able to monitor files and to warn the administrator when an unexpected change occurs (or simply to list such changes).

14.3.3.1. Auditing Packages: debsums and its Limits

debsums is an interesting tool since it allows finding what installed files have been modified (potentially by an attacker), but this should be taken with a grain of salt. First, because not all Debian packages provide the fingerprints required by this program (they can be found in /var/lib/dpkg/info/package.md5sums when they exist). As a reminder: a fingerprint is a value, often a number (even though in hexadecimal notation), that contains a kind of signature for the contents of a file. This signature is calculated with an algorithm (MD5 or SHA1 being well-known examples) that more or less guarantee that even the tiniest change in the file contents implies a change in the fingerprint; this is known as the “avalanche effect”. This allows a simple numerical fingerprint to serve as a litmus test to check whether the contents of a file have been altered. These algorithms are not reversible; in other words, for most of them, knowing a fingerprint doesn't allow finding the corresponding contents. Recent mathematical advances seem to weaken the absoluteness of these principles, but their use is not called into question so far, since creating different contents yielding the same fingerprint still seems to be quite a difficult task.
In addition, the md5sums files are stored on the hard disk; a thorough attacker will therefore update these files so they contain the new control sums for the subverted files.
The first drawback can be avoided by asking debsums to base its checks on a .deb package instead of relying on the md5sums file. But that requires downloading the matching .deb files first:
# apt-get --reinstall -d install `debsums -l`
[ ... ]
# debsums -p /var/cache/apt/archives -g
It is also worth noting that, in its default configuration, debsums automatically generates the missing md5sums files whenever a package is installed using APT.
The other problem can be avoided in a similar fashion: the check must simply be based on a pristine .deb file. Since this implies having all the .deb files for all the installed packages, and being sure of their integrity, the simplest way is to grab them from a Debian mirror. This operation can be slow and tedious, and should therefore not be considered a proactive technique to be used on a regular basis.
# apt-get --reinstall -d install `grep-status -e 'Status: install ok installed' -n -s Package`
[ ... ]
# debsums -p /var/cache/apt/archives --generate=all
Note that this example uses the grep-status command from the dctrl-tools package, which is not installed by default.

14.3.3.2. Monitoring Files: AIDE

The AIDE tool (Advanced Intrusion Detection Environment) allows checking file integrity, and detecting any change against a previously recorded image of the valid system. This image is stored as a database (/var/lib/aide/aide.db) containing the relevant information on all files of the system (fingerprints, permissions, timestamps and so on). This database is first initialized with aideinit; it is then used daily (by the /etc/cron.daily/aide script) to check that nothing relevant changed. When changes are detected, AIDE records them in log files (/var/log/aide/*.log) and sends its findings to the administrator by email.
Many options in /etc/default/aide can be used to tweak the behavior of the aide package. The AIDE configuration proper is stored in /etc/aide/aide.conf and /etc/aide/aide.conf.d/ (actually, these files are only used by update-aide.conf to generate /var/lib/aide/aide.conf.autogenerated). Configuration indicates which properties of which files need to be checked. For instance, the contents of log files changes routinely, and such changes can be ignored as long as the permissions of these files stay the same, but both contents and permissions of executable programs must be constant. Although not very complex, the configuration syntax is not fully intuitive, and reading the aide.conf(5) manual page is therefore recommended.
A new version of the database is generated daily in /var/lib/aide/aide.db.new; if all recorded changes were legitimate, it can be used to replace the reference database.

14.3.4. Detecting Intrusion (IDS/NIDS)

snort (in the Debian package of the same name) is a NIDS — a Network Intrusion Detection System. Its function is to listen to the network and try to detect infiltration attempts and/or hostile acts (including denial of service attacks). All these events are logged, and a daily email is sent to the administrator with a summary of the past 24 hours.
Its configuration requires describing the range of addresses that the local network covers. In practice, this means the set of all potential attack targets. Other important parameters can be configured with dpkg-reconfigure snort, including the network interface to monitor. This will often be eth0 for an Ethernet connection, but other possibilities exist such as ppp0 for an ADSL or PSTN (Public Switched Telephone Network, or good old dialup modem), or even wlan0 for some wireless network cards.
The snort configuration file (/etc/snort/snort.conf) is very long, and the abundant comments describe each directive with much detail. Getting the most out of it requires reading it in full and adapting it to the local situation. For instance, indicating which machine hosts which service can limit the number of incidents snort will report, since a denial of service attack on a desktop machine is far from being as critical as one on a DNS server. Another interesting directive allows storing the mappings between IP addresses and MAC addresses (these uniquely identify a network card), so as to allow detecting ARP spoofing attacks by which a compromised machine attempts to masquerade as another such as a sensitive server.