Techniques
 | This section needs expansionwith: explaining more obfuscation techniques in general. You can help by adding to it. (March 2023) |
Types of obfuscations include simple keyword substitution, use or non-use of whitespace to create artistic effects, and self-generating or heavily compressed programs.
According to Nick Montfort, techniques may include:
- naming obfuscation, which includes naming variables in a meaningless or deceptive way;
- data/code/comment confusion, which includes making some actual code look like comments or confusing syntax with data;
- double coding, which can be displaying code in poetry form or interesting shapes. [9]
Payload encoding for malware evasion
XOR encryption and Base64 encoding are two common methods used to hide malware from antivirus detection. Both work by changing how malicious code appears in its file form, which prevents security software from recognizing dangerous patterns.
In XOR obfuscation, an attacker chooses a secret key and applies the XOR bitwise operation to the malware binary. This transforms the executable into what looks like random data. Function names in the import table vanish, PE headers become corrupted, and the entire file loses its structure. The obfuscated payload then gets embedded into a dropper, which is a normal-looking executable that contains the hidden malware as a resource or data section. When a user runs the dropper, it performs the XOR operation again with the same key to reconstruct the original malware, then either executes it directly from memory or writes it to disk before running it. [10]
This process removes several indicators that antivirus software relies on. The MZ header, which marks the beginning of every Windows executable, gets completely obscured by the XOR operation. Security programs frequently scan for this two-byte signature when searching for embedded executables. Base64 encoding achieves similar results through a different method. It converts binary data into ASCII text, so an executable file ends up looking like plain text rather than a program.
Research from the 2020 Machine Learning Security Evasion Competition showed that these methods can bypass modern detection systems. Participants used combinations of XOR encoding, Base64 encoding, and dead code insertion to evade all three competition models with fewer than five attempts per sample. Entropy-based detection also failed, and in some cases Base64 encoding actually lowered the entropy compared to the original malware files. [11]
The simplicity of these techniques is what makes them particularly dangerous. XOR and Base64 encoding require only basic programming skills to implement, yet they proved effective against advanced machine learning classifiers. This has pushed security researchers toward new defenses, including automated XOR key recovery tools and deeper analysis of embedded resources in executable files.
Automated tools
A variety of tools exist to perform or assist with code obfuscation. These include experimental research tools developed by academics, hobbyist tools, commercial products written by professionals, and open-source software. Additionally, deobfuscation tools exist, aiming to reverse the obfuscation process.
While most commercial obfuscation solutions transform either program source code or platform-independent bytecode, i.e. portable code (as used by Java and .NET), some also work directly on compiled binaries.
Recreational
Writing and reading obfuscated source code can be a brain teaser. A number of programming contests reward the most creatively obfuscated code, such as the International Obfuscated C Code Contest and the Obfuscated Perl Contest.
Short obfuscated Perl programs may be used in signatures of Perl programmers. These are JAPHs ("Just another Perl hacker"). [18]
Cryptographic
Cryptographers have explored the idea of obfuscating code so that reverse-engineering the code is cryptographically hard. This is formalized in the many proposals for indistinguishability obfuscation, a cryptographic primitive that, if possible to build securely, would allow one to construct many other kinds of cryptography, including completely novel types that no one knows how to make. (A stronger notion, black-box obfuscation, is known to be impossible in general.) [19] [20]
