M3rry Chr1stm4s, y0u ar3 0wn3d
Note to archeologists and readers: this was written a long time ago and is maintained simply for historical reasons.
Had a bit of a scare last night. I was logged in to one of my Debian Linux servers just making a short configuration change to something when, for no apparent reason, I ran chkrootkit (checks for the presence of rootkits… a positive result indicates that the host has been broken in to), which came back with output like this:
…
Checking `asp’… not infected
Checking `bindshell’… not infected
**Checking `lkm’… You have 3 process hidden for readdir command
You have 3 process hidden for ps command
Warning: Possible LKM Trojan installed**
Checking `rexedcs’… not found
…
Um, ok. That’s not good. That’s also entirely unexpected, since I am vigilant about handling security related issues. I ran chkrootkit again, almost in disbelief. It returned no such warning this time.
About this time, I started thinking about Murphy. Christmas Eve. Family in town. Big dinners. Presents. Relaxing. Server possibly compromised. Spending endless hours in a datacenter hunched over a keyboard while my skin slowly goes dry. Headaches and tense shoulders. Why now, why me???
I sat on the sofa in shock for about 10 minutes.
But it doesn’t make sense. I pulled down the latest version of chkrootkit and started going through the source code. What I realised is that the part that the lkm tests are actually using pretty simple code. What’s happening is that chkrootkit (which is a shell script) uses chkproc to see if there are entries in the /proc filesystem that do not exist in the process table.
But there’s a not-so-subtle race condition that can easily arise, especially on a server that creates lots of processes. Between the time that chkproc can parse and store the output of ‘ps’ and start the process of comparing it to the entries in /proc, new processes can be created. Since the original ps entries are cached, the new process will falsely trigger an alert.
LKM compromises are the hardest to detect and fix. I wasn’t looking forward to handling that. But it seems like all is well and I can now reproduce this problem to my satisfaction.
Now Christmas can indeed be Merry…