Odd System Behavior

Linux systems vary greatly in how they operate. One system might normally run with a high CPU load, whereas another might normally run with little or no load. One computer might normally see little network traffic, whereas another might regularly transfer gigabytes of data every day. As a system administrator, you should have a good feel for how a system normally runs. You should investigate any deviation from normal behavior. Even if the deviation isn't due to a break-in, it may represent a problem that you should correct. Some possible oddities you should investigate include:

Unusual Load Averages A load average is a measure of the demand for CPU time placed on the system by all the programs it runs. Chapter 14, "Programs and Processes," describes this concept in more detail. You can check your load average with uptime or top. Many GUI environments also provide load average meters. If you see the load average change from its usual range, a cracker could have broken in and be running CPU-intensive programs or could have caused CPU-intensive programs to crash. On the other hand, legitimate users could be demanding more CPU time than normal for legitimate reasons, or a CPU-intensive program might have crashed for mundane reasons or run out of data to process.

Unusual Network Traffic Just as with load averages, a system normally sees a certain range of network traffic. One possible cause of changes in network traffic patterns is an intrusion. A cracker might be using your system as a node in a distributed denial-of-service (DDoS) attack, for instance, chewing up your network bandwidth. On the other hand, you could be seeing an innocent spike in demand for a server's services, or a user could be downloading unusually large files for legitimate reasons. One way to monitor network traffic is to watch the LEDs on a NIC—but they are usually not very visible, as they're on the back of most computers. Hubs and switches also have activity LEDs, and these LEDs may be more accessible. Similar comments apply to external modems—both telephone modems and broadband modems. Typing ifconfig ethO (or a similar command for an interface other than ethO) produces a measure of transmitted (TX) and received (RX) packets.

Strange Running Processes If you use ps, top, or some other process-monitoring tool, you may see processes with which you're not familiar. If you are familiar with all of the processes that run on the computer, seeing something new can be a sign of a problem—or it could be something innocent, such as a user running a perfectly legitimate program that's not often run on your system. You should definitely take the time to peruse your process list to become familiar with what's normal for your system.

Odd Program Crashes Unfortunately, programs sometimes crash, even on Linux. After you've used Linux for a while, you should be familiar with which programs are likely to crash on your system. If another program starts crashing, it could be that an intruder has changed a configuration file, support libraries, or even the program executable file itself, thereby causing the crashes. Alternatively, it could be that a routine system upgrade or an error on your part caused these problems. Program crashes can also be early signs of deteriorating hardware, such as RAM or a hard disk starting to go bad.

Changes in Program Behavior In most cases, programs should behave in a very predictable manner. For instance, if your bash shell presents a plain dollar sign ($) prompt today, it should do the same tomorrow unless you change the configuration file. If a program's behavior changes suddenly and without your having updated it or changed its configuration file, an intruder might have fumbled a takeover by inadvertently altering a configuration file detail or by changing a program file. On the other hand, you might have forgotten a legitimate upgrade or change, or the program might be responding to some other change in the system that you don't realize is related. If you share administrative responsibility for the system, perhaps another administrator has made a change.

Peculiar Log Entries All Linux systems maintain log files, most of which reside in /var/log and its subdirectories. These log files may contain clues concerning an attempted or successful intrusion, such as repeated login failures or reports of unusual server stop/start cycles. Crackers often try to cover their tracks by editing or deleting log files, so even missing log files or gaps in log files can be signs of trouble.

Filled Filesystems If you notice that a filesystem has filled up, you should investigate the matter. Even if it's due to innocent causes, such as increased user demand for storage space, you must correct the problem. Sometimes an intruder might inadvertently or intentionally fill a filesystem. One type of DoS attack involves causing your system to create huge log files, filling your log file partition. Such attacks aren't intrusions perse, but they do require your attention.

Unfamiliar Usernames If you notice unfamiliar usernames in ps or top listings or in /etc/passwd or other system configuration files, investigate immediately. Some servers require special accounts to operate correctly, but for the most part, if an account wasn't present when you installed the system and wasn't added by you or another authorized administrator, it's suspect. Make a backup of/etc/passwd right after you install the system, and keep it for reference.

Complaints from Other System Administrators In many ways the worst-case scenario is receiving a complaint from another system administrator. You might receive a phone call ore-mail complaining of suspicious access attempts from your computer. Another common complaint concerns spam originating from your system. Sometimes these complaints are signs of an intrusion; the intruder is using your system to attack others. Other times these complaints indicate that you have a local user who's a "bad apple" and is attacking others. Sometimes (particularly with spam and e-mail worms) the complaint is spurious; it's easy to forge e-mail headers to indicate a false return address, and a spammer may be attacking you indirectly in this way.

All of these indications demand further investigation on your part. How you proceed depends on the nature of the peculiarity. Sometimes reading a few man page entries or contacting a user about an unusual process will clear up the situation. Other times you may need to do extensive web searches. Using chkrootkit, as described in the upcoming section, "Looking for Root Kits with chkrootkit," may be in order.

Warning When dealing with a potential system intrusion, you shouldn't trust anything on the system that's under investigation. For instance, a cracker might have planted false man page entries to explain away an unusual program; and a compromised system's e-mail might not be trustworthy. Intruders sometimes alter common system tools such as Is to hide their tracks. Perform your investigations from another computer that displays no unusual symptoms.

0 0

Post a comment