Rootkit and malware detection and removal guide

Malware and other security threats plague every type of Windows user, and that includes even the most advanced technical IT professional. Infections caused by rootkits, spyware, viruses and any other conceivable type of malware have become inevitable in the enterprise and, as a Windows security professional, you need to know how to prevent these threats from completely corrupting your systems.

Know thy malware enemy

The first step to combating a malware infestation is understanding and identifying what type of security threat has invaded your Windows shop. Do you have the right tools to clean up a computer virus? Do you know how to root out a rootkit? Can you identify that a malicious hacker has broken through your security defenses quickly enough to prevent them from doing serious damage? In this guide, learn about anti-malware strategies and disaster recovery strategies and save yourself the hassle of being yet another hacker's victim.

Windows Security Threats

The fight against security threats in your Windows shop is a part of everyday life. Help yourself to be as well-equipped as possible to fight that fight with this All-in-one Guide on Windows Security Threats. Here you will find expert advice, columns and tips on malware (including spyware and bots), prevention planning and tools, and information about removal.

Table of contents

Rootkit prevention and detection
Prevent and defend against spyware infection
Tools for virus removal and detection

In this section, learn about one of today's most ferocious breeds of malware: The rootkit. Find information about what a rootkit is, how to locate one on your Windows network, how to remove it and how to assemble a proper rootkit defense tool belt.

Rootkits

What is a rootkit?

A rootkit is a collection of tools (programs) that enable administrator-level access to a computer or computer network. Typically, a cracker installs a rootkit on a computer after first obtaining user-level access, either by exploiting a known vulnerability or cracking a password. Once the rootkit is installed, it allows the attacker to mask intrusion and gain root or privileged access to the computer and, possibly, other machines on the network.

A rootkit may consist of spyware and other programs that: monitor traffic and keystrokes; create a "backdoor" into the system for the hacker's use; attack other machines on the network; and alter existing system tools to escape detection.

Many experts have theorized that rootkits will soon be thought of as equally troublesome as viruses and spyware, if they aren't already. Rootkits have become more common and their sources more surprising. In late October of 2005, security expert Mark Russinovich of Sysinternals discovered that he had a rootkit on his own computer that had been installed as part of the digital rights management (DRM) component on a Sony audio CD. Experts worry that the practice may be more widespread than the public suspects and that attackers could exploit existing programs like the Sony rootkit. "This creates opportunities for virus writers," said Mikko Hypponen, director of AV research for Finnish firm F-Secure Corp. "These rootkits can be exploited by any malware, and when it's used this way, it's harder for firms like ours to distinguish the malicious from the legitimate."

Rootkit detection

Rootkit technologies are rapidly cropping up in a variety of places, including commercial security products and seemingly benign, third-party application extensions. Finding and removing rootkit installations is not an exact science. Rootkits can be installed on a computer in many ways. No single tool (and no combination of tools) can correctly identify all rootkits and rootkit-like behavior.

1. Search your system memory. Monitor all ingress points for a process as it is invoked, keeping track of imported library calls (from DLLs) that may be hooked or redirected to other functions, loading device drivers, etc. The drawback to this approach is that it is tedious, time-consuming and cannot account for all possible avenues in which a rootkit can be introduced into the system.
2. Seek the truth -- expose API dishonesty. One good rootkit detection application for Windows is the RootkitRevealer by Windows security analysts Bryce Cogswell and Mark Russinovich. This tiny (190 KB) binary scouts out file system locations and registry hives, looking for information kept hidden from the Windows API, the Master File Table, and directory index. In addition, Jamie Butler, author of the highly recommended trade book Subverting the Windows Kernel: Rootkits, has created a tool called VICE, which systematically hunts down hooks in APIs, call tables and function pointers.

   RootkitRevealer may take a while to complete because it performs an exhaustive search. First it dumps the registry hives, then it examines the C: directory tree for known rootkit sources and signatures, and finally performs a cursory analysis of the entire C: volume.

3. Keep abreast of the latest antivirus and malware protection software from leading antivirus and security vendors. Sysinternals and F-Secure offer standalone rootkit detection tools (RootkitRevealer and Blacklight, respectively). Even Microsoft has implemented rootkit detection features in its own Malicious software removal tool.
4. Update your firewall protection. Remember, for the concealment process to be effective to a potential attacker, it is vital that the hacker can get back into a machine once it's been compromised. Although firewalls do nothing to mitigate application-level risks, they can pose a significant challenge to attackers when they prohibit re-entry into a victim machine.
5. If possible, harden your workstation or server against attack.This proactive step prevents an attacker from installing a rootkit in the first place. The National Security Agency publishes a guideline for hardening Windows environments, which is a great jump-off point for educating yourself on preventive actions against system intrusion.

Rootkit removal

Rootkits are relatively easy to install on victim hosts. To upload a rootkit, a determined attacker can do everything from exploit a Windows vulnerability to crack a password or even obtain physical system access. They can even execute a phishing attack, where a hacker cons a user into running an executable file in an email attachment or via a hyperlink distributed via email or instant messaging. Once they're in place, as you're likely to find out, rootkits aren't so easy to find or get rid of.

The rootkit threat is not as widespread as viruses and spyware. Given this fact, and the lack of a truly effective rootkit prevention solution, removing rootkits is largely a reactive process.

Is there a rootkit problem?

First, you need to determine if there is a problem. To determine if there is truly a rootkit operating behind the scenes, use a system process analyzer such as Sysinternals' ProcessExplorer or, better yet, a network analyzer. By using these tools, you'll likely be surprised to find what programs are doing and what's going in and out of your network adapter. You may also discover that you simply have an over-taxed system running with too little memory or a severely fragmented hard drive. With that in mind, I recommend checking your system configuration and defragmenting your drive(s). Remember, though, that it's better to be safe than sorry, so run a rootkit scan as well.

Choosing the right rootkit detection tool

To get started scanning, you need the right tools. There are several rootkit scanning tools available. A popular free scanner I mention often is Sysinternals' RootkitRevealer. It works by comparing the services running at the Windows API level with what's showing up at the raw data level on the computer's hard drive. The only negative aspect of RootkitRevealer is that it doesn't clean what it finds. Its instructions tell you to search the Web for removal instructions or reformat your drive and reinstall Windows. Ouch. Another free (at least until January of 2007) tool for scanning is F-Secure BlackLight.

Using BlackLight is simply a matter of downloading it and running the executable file. It will scan your local drives, highlight what it found and allow you to clean what it finds. It hides almost everything from the user, but it is very fast and very easy to use. You should definitely check it out.

Another rootkit scanning tool by an F-Secure competitor is Sophos Anti-Rootkit. Anti-Rootkit has an install routine and you have to manually run the executable afterwards. It allows for more user interactivity than BlackLight, but it is slower to scan your system. In Figure 3, notice how Anti-Rootkit easily uncovered the Hacker Defender as well -- including its installation files I intentionally left behind.

There are various other rootkit scanners including Rootkit Hook Analyzer, VICE, and RAIDE. I encourage you to try all of them to see which one(s) best suit your needs.

Clean up the rootkits

It's one thing to find a rootkit, but quite another to remove it and any malware it's hiding. It may or may not be possible -- again, you'll never really know since a rootkit can interfere with your scanning and removal program. You still need to try.

Security threats expert Kevin Beaver says, "I had good luck with both BlackLight and Anti-Rootkit in my test environment. Before you start cleaning house, though, make sure you have a backup of any important data files." Removing a rootkit with cleaning tools may actually leave Windows in an unstable or inoperable state depending on which files were infected and subsequently cleaned. Or, worse, a well-coded rootkit could conceivably detect the removal process and self-destruct taking your data out with it.

Defenses against rootkits

To truly bulletproof your rootkit detection and cleanup process, make sure you always read the current user instructions for your scanning tools to see what special steps you need to take before, during and after the clean-up process. Then, after you've found and cleaned a rootkit, rescan the system once you reboot to double-check that it was fully cleaned and the malware hasn't returned.

As of now, rootkit infections typically occur in targeted attacks, but given the way things have progressed with malware in the past decade, I wouldn't be surprised to see this as a widespread problem in the future. As always, the bad guys are using their knowledge and technical skills to stay a step or two ahead. Still a little paranoid about rootkit infections? Want to be sure your system is truly clean? The best and most reliable method is to repartition, reformat and reload Windows. It's painful, but it's really the best way to go if you really need some closure.