Rootkit hunters caught in cat-and-mouse game

No malware, let alone a virtualised rootkit, is undetectable.That was the message delivered loud and clear at the Black Hat USA Briefings.

A team of well-known security researchers led a session on the methods they believe would be effective in finding virtualised rootkits, such as Joanna Rutkowska's infamous Blue Pill or Dino Dai Zovi's Vitriol. The researchers outlined a number of techniques for detecting traces of such a rootkit's activity, including side-channel attacks, finding hypervisor bugs and looking for errors caused by the malware.

"You're basically stuck in a cat-and-mouse game in which the attacker designs some code, you look for characteristics of that code and detect it, and then it all repeats in a big cycle," said Nate Lawson, principal at Oakland, Calif.-based Root Labs, and one of the co-presenters of the session. "We've seen this before and people have always found countermeasures, and we expect that will continue the same way."

From left: Peter Ferrie, Symantec; Nate Lawson, Root Labs; Tom Ptacek, Matasano Security. Click to enlarge.

"We're really interested in debunking that claim," Ptacek said.

The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable. Nate Lawson Root Labs

One of the methods Lawson outlined for detecting a virtualised rootkit involves observing changes in the Translation Lookaside Buffer (TLB), a cache in the CPU. When something causes a virtual machine to exit, the hypervisor leaves traces of its presence in the TLB. So, Lawson said, one way to detect a hypervisor rootkit would be to cause it to somehow exit, and then read the TLB and look for changes.

But, Lawson and Ptacek conceded, there's nothing stopping the malware author from writing a feature to detect the "rootkit detector," which Lawson said leads back to the familiar attacker-defender cat-and-mouse game.

"What you end up with is the same cycle that we see with AV engines and viruses, where I look at the latest version of your code, find ways to detect it and then you write a new version and we start all over again," Lawson said. "The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable."

Special Black Hat coverage Check out more of SearchSecurity.com's special news coverage of Black Hat USA 2007.

The rootkit session originally was intended to be a live demo in which Rutkowska would load Blue Pill onto one of several clean Vista machines, and Ptacek and his co-presenters would load Samsara onto all of the PCs and try to detect the rootkit. But Rutkowska declined the offer and instead sat in the audience.