M3rry Chr1stm4s, y0u ar3 0wn3d

Had a bit of a scare last night. I was logged in to one of my Debian Linux servers just making a short configuration change to something when, for no apparent reason, I ran chkrootkit (checks for the presence of rootkits… a positive result indicates that the host has been broken in to), which came back with output like this:

…

Checking `asp’… not infected

Checking `bindshell’… not infected

**Checking `lkm’… You have 3 process hidden for readdir command

You have 3 process hidden for ps command

Warning: Possible LKM Trojan installed**

Checking `rexedcs’… not found

…

Um, ok. That’s not good. That’s also entirely unexpected, since I am vigilant about handling security related issues. I ran chkrootkit again, almost in disbelief. It returned no such warning this time.

About this time, I started thinking about Murphy. Christmas Eve. Family in town. Big dinners. Presents. Relaxing. Server possibly compromised. Spending endless hours in a datacenter hunched over a keyboard while my skin slowly goes dry. Headaches and tense shoulders. Why now, why me???

I sat on the sofa in shock for about 10 minutes.

But it doesn’t make sense. I pulled down the latest version of chkrootkit and started going through the source code. What I realised is that the part that the lkm tests are actually using pretty simple code. What’s happening is that chkrootkit (which is a shell script) uses chkproc to see if there are entries in the /proc filesystem that do not exist in the process table.

But there’s a not-so-subtle race condition that can easily arise, especially on a server that creates lots of processes. Between the time that chkproc can parse and store the output of ‘ps’ and start the process of comparing it to the entries in /proc, new processes can be created. Since the original ps entries are cached, the new process will falsely trigger an alert.

LKM compromises are the hardest to detect and fix. I wasn’t looking forward to handling that. But it seems like all is well and I can now reproduce this problem to my satisfaction.

Now Christmas can indeed be Merry…