Rootkit hunters caught in cat-and-mouse game

No malware, let alone a virtualised rootkit, is undetectable.That was the message delivered loud and clear at the Black Hat USA Briefings.

A team of well-known security researchers led a session on the methods they believe would be effective in finding virtualised rootkits, such as Joanna Rutkowska's infamous Blue Pill or Dino Dai Zovi's Vitriol. The researchers outlined a number of techniques for detecting traces of such a rootkit's activity, including side-channel attacks, finding hypervisor bugs and looking for errors caused by the malware.

Download this free guide

The importance of web security

Join us as we take a look at the different approaches you can take in order to bolster your web security. We find out how to identify and address overlooked web security vulnerabilities, how security controls affect web security assessment results and why web opportunities must be met with appropriate security controls.

Start Download

• Corporate E-mail Address:

  You forgot to provide an Email Address.

  This email address doesn’t appear to be valid.

  This email address is already registered. Please login.

  You have exceeded the maximum character limit.

  Please provide a Corporate E-mail Address.

By submitting my Email address I confirm that I have read and accepted the Terms of Use and Declaration of Consent.

By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.

You also agree that your personal information may be transferred and processed in the United States, and that you have read and agree to the Terms of Use and the Privacy Policy.

"You're basically stuck in a cat-and-mouse game in which the attacker designs some code, you look for characteristics of that code and detect it, and then it all repeats in a big cycle," said Nate Lawson, principal at Oakland, Calif.-based Root Labs, and one of the co-presenters of the session. "We've seen this before and people have always found countermeasures, and we expect that will continue the same way."

From left: Peter Ferrie, Symantec; Nate Lawson, Root Labs; Tom Ptacek, Matasano Security. Click to enlarge.

"We're really interested in debunking that claim," Ptacek said.

The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable. Nate Lawson Root Labs

One of the methods Lawson outlined for detecting a virtualised rootkit involves observing changes in the Translation Lookaside Buffer (TLB), a cache in the CPU. When something causes a virtual machine to exit, the hypervisor leaves traces of its presence in the TLB. So, Lawson said, one way to detect a hypervisor rootkit would be to cause it to somehow exit, and then read the TLB and look for changes.

But, Lawson and Ptacek conceded, there's nothing stopping the malware author from writing a feature to detect the "rootkit detector," which Lawson said leads back to the familiar attacker-defender cat-and-mouse game.

"What you end up with is the same cycle that we see with AV engines and viruses, where I look at the latest version of your code, find ways to detect it and then you write a new version and we start all over again," Lawson said. "The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable."

Special Black Hat coverage Check out more of SearchSecurity.com's special news coverage of Black Hat USA 2007.

The rootkit session originally was intended to be a live demo in which Rutkowska would load Blue Pill onto one of several clean Vista machines, and Ptacek and his co-presenters would load Samsara onto all of the PCs and try to detect the rootkit. But Rutkowska declined the offer and instead sat in the audience.