M3rry Chr1stm4s, y0u ar3 0wn3d
Had a bit of a scare last night. I was logged in to one of my Debian Linux servers just making a short configuration change to something when, for no apparent reason, I ran chkrootkit (checks for the presence of rootkits… a positive result indicates that the host has been broken in to), which came back with output like this:
…
Checking `asp’… not infected
imdur
Checking `bindshell’… not infected
Checking `lkm’… You have 3 process hidden for readdir command
You have 3 process hidden for ps command
Warning: Possible LKM Trojan installed
Checking `rexedcs’… not found
…
Um, ok. That’s not good. That’s also entirely unexpected, since I am vigilant about handling security related issues. I ran chkrootkit again, almost in disbelief. It returned no such warning this time.
About this time, I started thinking about Murphy. Christmas Eve. Family in town. Big dinners. Presents. Relaxing. Server possibly compromised. Spending endless hours in a datacenter hunched over a keyboard while my skin slowly goes dry. Headaches and tense shoulders. Why now, why me???
I sat on the sofa in shock for about 10 minutes.
But it doesn’t make sense. I pulled down the latest version of chkrootkit and started going through the source code. What I realised is that the part that the lkm tests are actually using pretty simple code. What’s happening is that chkrootkit (which is a shell script) uses chkproc to see if there are entries in the /proc filesystem that do not exist in the process table.
But there’s a not-so-subtle race condition that can easily arise, especially on a server that creates lots of processes. Between the time that chkproc can parse and store the output of ‘ps’ and start the process of comparing it to the entries in /proc, new processes can be created. Since the original ps entries are cached, the new process will falsely trigger an alert.
LKM compromises are the hardest to detect and fix. I wasn’t looking forward to handling that. But it seems like all is well and I can now reproduce this problem to my satisfaction.
Now Christmas can indeed be Merry…
Thanks for the LKM post. I was speechless and did a google on it and found your explanation. Ran chkrootkit again and voila, no false positives.
I work in a datacenter and I was running chkrootkit on some Korean guy’s RHEL web server, and I kept getting the same “Possible LKM trojan installed” message. Then I read your blog about the race condition in the code, so I typed in:
/etc/init.d/httpd stop
to stop Apache and then ran chkrootkit again and now the race condition was gone and the “Possible LKM trojan installed” message disappeared with it.
Thanks!