antidebug-vm-scanner

Static analysis CLI that scans a PE (EXE/DLL) file for anti-debug and anti-VM techniques and obfuscation indicators, without requiring IDA, x64dbg, or any other disassembler installed.

It performs, from scratch, its own:

• PE32/PE32+ parser (headers, sections, import table)
• x86/x64 length-disassembler covering common legacy, SSE, and VEX (AVX) encodings, walked linearly over executable sections
• Shannon entropy calculation per section
• signature databases for anti-debug APIs, anti-VM APIs, VM/sandbox artifact strings, and known packer section names

What it detects

• Anti-debug API imports: IsDebuggerPresent, NtQueryInformationProcess, CheckRemoteDebuggerPresent, OutputDebugString, timing APIs, etc.
• Anti-debug/anti-VM opcodes found while walking executable sections: RDTSC/RDTSCP, CPUID, SLDT/STR/SGDT/SIDT/SMSW, ICEBP/INT1, INT 2D, VMCALL/VMREAD/VMWRITE
• VM I/O backdoor probes: IN/OUT port instructions (e.g. the VMware 0x5658 backdoor port)
• VM/sandbox artifact strings: VirtualBox, VMware, QEMU, Xen, Parallels, Hyper-V, Sandboxie, Wine registry/service/driver names
• Known packer section names: UPX, ASPack, Petite, PKLite, Themida, VMProtect, Enigma, MPRESS, kkrunchy
• High-entropy sections (>= 7.2 bits/byte), a proxy for packed or encrypted code/strings
• Indirect-dispatch density per section (ratio of indirect jmp/call to total decoded instructions), a proxy for control-flow flattening / VM-style dispatcher obfuscation

Each finding feeds a weighted confidence score (0-100) and a verdict string.

Build

Requires a C++17 compiler and CMake. No third-party dependencies.

cmake -S . -B build -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4

The advscan binary is statically linked against the C++ runtime so it runs standalone on any Windows machine.

Run

build/advscan.exe path/to/sample.exe
build/advscan.exe path/to/sample.exe --json
build/advscan.exe path/to/sample.exe --json report.json

JSON shape:

{
  "file": "sample.exe",
  "architecture": "x64",
  "image_base": 5368709120,
  "confidence_score": 22.0,
  "verdict": "Limited anti-analysis techniques detected",
  "findings": [
    { "category": "anti_debug_import", "description": "KERNEL32.dll!IsDebuggerPresent", "address": null },
    { "category": "anti_debug_opcode", "description": "RDTSC (timing-based anti-debug)", "address": 5368717056 }
  ]
}

address is an absolute virtual address (image base + RVA) when the finding has one, otherwise null.

category	meaning
anti_debug_import	Imports a known anti-debug API
anti_debug_opcode	Decoded a known anti-debug/anti-VM opcode (RDTSC, CPUID, SIDT, ICEBP, INT 2D, ...)
anti_vm_import	Imports a known VM/firmware enumeration API
anti_vm_opcode	Decoded VMCALL/VMREAD/VMWRITE
anti_vm_string	Embedded string matching a known VM/sandbox artifact
vm_backdoor_io	IN/OUT port instruction, consistent with the VMware I/O backdoor
packer_section	Section name matches a known packer (UPX, ASPack, VMProtect, ...)
high_entropy_section	Section entropy >= 7.2 bits/byte, likely packed/encrypted
indirect_dispatch	High ratio of indirect jmp/call, consistent with control-flow flattening

Test

cd build
ctest --output-on-failure

unit_tests checks the length-disassembler against known instruction encodings and the entropy function against known distributions. integration_smoke compiles a small anti-debug C sample with gcc (skipped automatically if gcc is not on PATH) and verifies advscan flags its IsDebuggerPresent import and its RDTSC/CPUID opcodes.

Limitations

This is a linear-sweep disassembler, not a recursive-descent one: it walks executable sections byte-by-byte without following control flow, so jump tables, exception-handling data, and other non-code bytes embedded in .text can be misdecoded as instructions. On large, heavily optimized binaries this produces some false-positive opcode findings (observed mostly as spurious ICEBP). Treat opcode-based findings as leads for manual triage, not proof. The import, string, packer-section, and entropy signals are exact and do not share this limitation.

Confidence scoring is a simple weighted sum capped at 100; it is most useful as a relative signal on small/targeted samples (loaders, droppers, single-purpose tools) rather than an absolute score on large general-purpose applications with hundreds of imports.