Lifting the VM and deriving a keygen

James · 28 July 2026

This article covers the static analysis of a 64 bit Windows crackme whose debugger gate protects a small register VM that turns its final register state into the repeating key used to decrypt the success message.

1Initial triage

Initial triage shows an unpacked PE32+ image built with the MSVC runtime whose .text section spans 0x32A1C bytes while .rdata begins at RVA 0x34000 and the import table exposes GetThreadContext, IsDebuggerPresent, Toolhelp enumeration and OutputDebugStringA.

The wide string table mentions far more analysis checks than the registration path really invokes but following the xref to the name prompt at 0x140034DA8 lands inside the application routine at 0x1400065A0 where the active gate below executes.

1400065D8  xor     r12d, r12d
1400065E0  call    sub_14000AA40
1400065E5  test    al, al
1400065E7  jne     loc_1400072F8
1400065ED  call    sub_14000A710
1400065F2  test    al, al
1400065F4  jne     loc_1400072F8
1400065FA  call    sub_14000A630
1400065FF  test    al, al
140006601  jne     loc_1400072F8
140006607  call    sub_14000C290
140006614  call    sub_14000C340
140006621  call    sub_14000B390
14000662E  call    sub_14000B660

1. Active debugger checks on the registration path

Every positive result reaches 0x1400072F8 which calls ExitProcess(0) without printing anything while the three checks read PEB.BeingDebugged through gs:[60h] then call IsDebuggerPresent and request CONTEXT_DEBUG_REGISTERS before checking DR0 through DR3.

The remaining live routines enumerate processes windows and loaded modules while the routine at 0x14000C290 is chaff that allocates three bytes RWX then writes 74 00 C3 before freeing the allocation and returning false.

Another state check surrounding the global object at 0x14004DA50 scans a vector whose records each occupy 0x78 bytes after VM execution and overwrites register 3 with DEADC0DE whenever it finds a suspicious record which matters because that register later becomes the decryption key so forcing the final branch cant repair the corrupted state.

2Serial parser and name hash

The parser at 0x140003FA0 requires exactly 0x1A characters then checks for hyphens at offsets 8 and 17 before parsing three hexadecimal fields containing eight characters each as shown underneath.

AAAAAAAA-BBBBBBBB-CCCCCCCC

I refer to the resulting dwords as A B and C during the lift while the supplied name goes through the ordinary 32 bit FNV-1a hash.

140006B9D  mov     ebx, 811C9DC5h
140006BC2  movzx   eax, byte ptr [rdx]
140006BC5  xor     eax, ebx
140006BC7  imul    ebx, eax, 1000193h
140006BCD  inc     rdx
140006BD0  cmp     rdx, r8
140006BD3  jne     loc_140006BC2

140006BE9  mov     [rbp+190h], ebx
140006BF3  mov     [rbp+194h], eax
140006BFD  mov     [rbp+198h], eax
140006C07  mov     [rbp+19Ch], eax
140006C19  call    sub_14000E630

2. FNV-1a and the VM input block

The caller zeroes the remainder of VM memory before execution so no uninitialised state participates and every operation performed by the hash wraps naturally at 32 bits.

3Rebuilding the bytecode

The application routine constructs 0x110 bytes directly on its stack before copying them into a vector whose instruction records each contain the four dwords shown underneath.

struct vm_insn {
    uint32_t opcode;
    uint32_t dst;
    uint32_t src;
    uint32_t imm;
};

3. VM instruction format

The compiler constructs this array through mixed qword and dword stores and my first pass shifted one boundary far enough to misread the third instruction as an unaligned load from mem+6.

That incorrect boundary produced a dead serial but reconstructing the literal buffer and decoding records along 16 byte boundaries showed a zero immediate which confirmed that the instruction actually loads the name hash from mem+0.

0000  01  r0, -, 00000004
0010  08  r0, -, 12345678
0020  01  r3, -, 00000000
0030  06  r0, r3, -
0040  01  r1, -, 00000008
0050  04  r2, -, DEADBEEF
0060  06  r1, r2, -
0070  01  r2, -, 00000004
0080  09  r2, -, 00000003
0090  06  r1, r2, -
00A0  01  r2, -, 0000000C
00B0  08  r2, -, C0FFEE42
00C0  03  r3, r0, -
00D0  05  r3, r1, -
00E0  05  r3, r2, -
00F0  08  r3, -, 7F3A9C10
0100  FF  -,  -, -

4. Reconstructed registration bytecode

4The VM dispatcher

The interpreter at 0x14000E350 owns four dword registers and masks both register indices with 3 before translating each opcode through the byte table at 0x14000E51C and selecting a handler RVA from the jump table at 0x14000E4E8.

14000E380  mov     r8d, [vm_ip]
14000E387  inc     [vm_ip]
14000E38D  shl     r8, 4
14000E391  add     r8, rdx
14000E394  mov     edx, [r8+4]
14000E398  mov     r10d, [r8+8]
14000E39C  and     edx, 3
14000E39F  mov     eax, [r8]
14000E3A2  and     r10d, 3
14000E3A6  dec     eax
14000E3AD  ja      loc_14000E4CA
14000E3B3  movzx   eax, byte ptr [r11+rax+0E51Ch]
14000E3BC  mov     ecx, [r11+rax*4+0E4E8h]
14000E3C4  add     rcx, r11
14000E3C7  jmp     rcx

5. Remapped VM dispatch

The useful handlers implement load store move immediate move addition subtraction XOR immediate XOR and shifts so naming those handlers turns the registration bytecode into the linear program below where every arithmetic operation wraps modulo 2³².

r0  = A;
r0 ^= 0x12345678;
r3  = fnv1a(name);
r0 -= r3;

r1  = B;
r1 -= 0xDEADBEEF;
r2  = A << 3;
r1 -= r2;

r2  = C ^ 0xC0FFEE42;
r3  = r0 + r1 + r2;
r3 ^= 0x7F3A9C10;

6. Lifted VM program

5The check is the decryption key

After the VM returns there isn't a conventional serial comparison because register 3 becomes a repeating dword key for the success buffer whose bytes are first XORed with 0x7F3A9C10 and then XORed with that final register value.

140006F50  lea     r8, [rbp-18h]
140006F66  mov     edx, 7F3A9C10h
140006F6B  shr     edx, cl
140006F6D  xor     dl, [r8+r9]
140006F71  mov     [rbp+r9+30h], dl

140006FA9  mov     edx, edi
140006FAB  shr     edx, cl
140006FAD  xor     dl, [rbp+rbx+30h]

140006FCA  cmp     edi, 7F3A9C10h
140006FD7  je      loc_140006FE0

7. Success buffer XOR passes

The comparison only controls the printed prefix while register 3 determines whether the message body decrypts into readable text which means a forced jump cant reconstruct the missing plaintext.

6Solving the serial

A clean run requires VM register 3 to equal the value below because the success buffer decryption loop later uses that same dword as its repeating key.

r₃ = 0x7F3A9C10.

The final VM instruction XORs register 3 with that same constant which means the accumulated value immediately before the instruction must equal zero under wrapping 32 bit arithmetic.

((A ⊕ 0x12345678) − H) + (B − 0xDEADBEEF − (A ≪ 3)) + (C ⊕ 0xC0FFEE42) ≡ 0 (mod 2³²),

Here H denotes FNV1a(name) and after choosing arbitrary dwords A and B the remaining serial field follows directly from the three equations underneath.

r₀ = (A ⊕ 0x12345678) − H, r₁ = B − 0xDEADBEEF − (A ≪ 3), C = (−(r₀ + r₁) mod 2³²) ⊕ 0xC0FFEE42.

7Keygen

MASK = 0xFFFFFFFF

def u32(value):
    return value & MASK

def fnv1a(data):
    value = 0x811C9DC5
    for byte in data:
        value = u32((value ^ byte) * 0x01000193)
    return value

def serial(name, a, b):
    h = fnv1a(name)
    r0 = u32((a ^ 0x12345678) - h)
    r1 = u32(b - 0xDEADBEEF - u32(a << 3))
    c = u32(-u32(r0 + r1)) ^ 0xC0FFEE42
    return f"{a:08X}-{b:08X}-{c:08X}"

8. Reduced keygen

The standalone keygen runs the reduced VM itself and checks that register 3 finishes at 0x7F3A9C10 before printing so a bad modulo operation cant quietly produce a serial that looks right but fails inside the crackme.