Malicious software – or malware, as it is more popularly known — has a long history, and continues to evolve in terms of sophistication and stealth. Over the past two decades, replication and spreading mechanisms used in malware have advanced considerably, as have techniques used to prevent analysis and detection. Such techniques include anti-debugging, encryption, packing and entry-point obscuring, among others. It is important to be aware or malware analysis procedures, using the available tools and a bit of ingenuity.
Static analysis of malware
A general, static malware analysis does not require knowledge of reverse engineering or complex programming techniques. However, detailed static analysis of the malware requires a disassembler and analysis of assembly-level code. This is beyond the scope of our tip.
Static analysis is relatively safe, since the malware payload is not live. There is no threat of deletion or modification of data or system files. Another advantage is that malware designed for, say, Windows, can safely be analyzed statically on an OS X system. Here are some of the common methods for static malware analysis:
This method includes creating file hashes. It can be achieved using the message-digest algorithm MD5, and secure hash algorithms such as SHA-1 and SHA-256, as established by security researchers. This helps analyze whether the file has been modified by any other program or has modified itself. For malware analysis, there are tools available to create file hashes; one such tool is md5deep (http://md5deep.sourceforge.net) developed by Jesse Kornblum.
Online virus scanners such as Virustotal (www.virustotal.com) or Jotti’s malware scan (virusscan.jotti.org) can facilitate file fingerprinting, since any detection of malware code in the file under analysis provides ready information regarding its behavior.
A packer is a program that obfuscates an executable program’s content. This complicates matters for a reverse engineer doing detailed malware analysis. PEiD is a free program that contains signatures of numerous compilers and packers. It can be used to detect known packers or the presence of an unknown packer.
Figure 1. Screenshot of PEiD with unknown packer. Courtesy: Kendall McMillan
Dynamic malware analysis
In dynamic analysis of malware, a fishbowl is created for executing the malware code and observing its behavior. Dynamic analysis requires an environment that can be sacrificed, and is logically partitioned from other hosts on the network. On a Windows platform, the malware behavior can be analyzed by monitoring its interactions with the file system, registry and other processes. This can be done using free tools such as Process Monitor from Sysinternals or the open-source Wireshark.
Figure 2. Process Monitor screenshot. Courtesy Kendall McMillan
The above methods of static and dynamic malware analysis are not completely effective for analyzing full-fledged backdoors or botnet clients using custom encoded data. The best approach is clubbing static analysis with IDA and dynamic analysis with OllyDbg or WinDbg. The use of scriptable debuggers facilitates automated malware analysis, thereby eliminating tedious manual tasks in the analysis.
Polymorphism and detection methodologies
In an attempt to defeat detection by signatures, malware writers began implementing code obfuscation techniques. Modern malware is designed such that the keys as well as the encryption algorithms can change upon replication. Such polymorphic malware requires special methods of malware analysis and detection.
This malware analysis method uses known decryption algorithms and fragmented decrypted code. Equations are generated for each key; solving them reveals the malware’s identity. The use of dedicated decryption algorithms (DDAs) can be very useful in such malware analysis. However, to write a successful DDA, one needs to have complete knowledge of the malware.
Use of emulation eliminates all problems posed by other methodologies. Potential malicious code runs in a simulated environment. Emulation can encrypt and decrypt any code, with minimal overheads. An emulator in the AV engine works in a finite period of time during which malicious signature detection is operational. If nothing is found during that period, malware analysis emulation stops, and the given file is tagged as non-malicious.
It’s always a challenge for malware analysis experts to stay ahead of the curve and devise programs to filter ever-evolving malware. New methods of encryption and evasion are discovered, and one needs to be alert and up-to-date with the latest preventive measures and precautions to ensure that malware intrusions are minimized or eliminated completely.
About the author: Karthik R is a member of the NULL community. He completed his training for EC-council CEH in December 2010, and is at present pursuing his final year of B.Tech in Information Technology, from National Institute of Technology, Surathkal. Karthik can be contacted on firstname.lastname@example.org.