The universal antidebugger, x64 revamped

A single step for a debugger a giant leap for the obfuscator.

When a debugger hits a breakpoint, it can perform single-stepping into the subsequent instructions by halting itself each time. To do so, it uses a specially crafted flag called Trap Flag (TF) residing at 0x8th bit position inside the EFLAGS x86 register.

If the Trap Flag is enabled, the processor then triggers an interrupt after each instruction has been executed. In our case, the referenced interrupt is 0x1, the second one from the beginning of the IDT index.

lkd> !idt
Dumping IDT: fffff8007725b000

00:	fffff80074966100 nt!KiDivideErrorFaultShadow
01:	fffff80074966180 nt!KiDebugTrapOrFaultShadow

During the past decades folks have been taking advantage of this debugger/interrupt relationship to thwart analysis and steer program behavior away from debugging eyeballs. This method, widely documented in the Eildad Eilam book has been awarded the universal status due to its effectiveness in detecting both user and kernel debuggers. Before jumping into the antidebugging section, I thought would be helpful to provide a crude but effective way of setting up a VS2019 x64 project that can integrate both C and MASM (assembly) languages in single piece.

Preparing a VS2019 C/MASM project.

Since MSVC has indefinitely banned x64 inline assembly, if we really want to stick with Visual Studio 2019 our only option is to include a separate .asm file into our C/C++ code.

Here a quick tutorial on how to setup a VS2019 project that will let C/C++ and MASM coexist together.

1. Create a classic C++ Console app project.
2. Change project to target Debug/x64
3. Right click on the Project > Build Dependencies > Build customization > check the ‘MASM’ flag
4. Right click on the source folder and add the .asm file
5. On the newly created .asm file, right click on it > “Properties” > Change ‘Item Type’ to ‘Microsoft Macro Assembler’
6. In case we want to call external function via MASM, we need to add some dependencies: right-click on the solution > “Properties” > “Linker” > “Input” and add the following ‘Additional Dependencies’ legacy_stdio_definitions.lib and legacy_stdio_wide_specifiers.lib

That’s about it. The only remaining bit is to include all the MAMS references in our C code, but we’ll get there soon.

It’s a x64 trap!

The well-known universal way to detect if a debugger is currently attached to our program is to enable the trap flag and then raise en exception: if a debugger is attached, it will ‘ingest’ the raised exception and skip the real deal of the program. After poking the internet for a while, but not beyond page #2 of your-fav-search-engine, I couldn’t find a simple way to integrate this old trick inside a regular VS project, so I came up with the following solution:

#include <SDKDDKVer.h>
#include <stdio.h>
#include <Windows.h>
#include <iostream>

// external .asm reference
extern "C"
{
    void trap_64();
};


int main(int argc, char* argv[])
{
    system("PAUSE"); // you can attach the debugger here
    BOOL bExceptionHit = FALSE;
    __try
    {
        trap_64(); // actual call to .asm file
    }
    __except (EXCEPTION_EXECUTE_HANDLER)
    {
        bExceptionHit = TRUE;

    }

    if (bExceptionHit == FALSE)
        printf("A debugger is attached!\n");
    else
        printf("No debugger detected\n"); // no debugger present:
                                          // we can continue with
                                          // the regular program flow
 
    return 0;
}

The reason behind invoking the System Exception Handler it will be soon said: if a process enables the TF without a debugger attached, it will simply crash without an exception handling mechanism taking care of the exception itself (read - the Kernel reacting to the interrupt.)

More precisely, from Windows Internals 5th Edition, Chapter 3:

If a debugger is attached, it is alerted about the single step exception. If no debugger is attached or the exception is not handled by the debugger, the Operating System’s Exception Handling mechanism is invoked. If the Exception Handling mechanism still does not handle the exception, the debugger is given a second chance to handle the exception. If no debugger is available, the process is killed.

So, in order to keep the process alive we must involve the system SEH.

Now, bringing us to the meat of the matter, AKA the referenced .asm file, that is responsible for enabling the trap flag.

_TEXT	SEGMENT
 
PUBLIC trap_64
 
trap_64 PROC
	 pushfq
     or qword ptr[rsp], 100h
     popfq
     ret
trap_64 ENDP
 
_TEXT	ENDS
END 

The trap_64 procedure first saves the entire qword values of the EFLAGS¹ register into the stack (pushfq), then it ORes against the dereferenced stack pointer to the target value needed, 0x100, matching the 0x8th bit (that is 1 0000 0000 in binary ). It then loads the new qword values into the EFLAGS via a popfq. The reason we have to place at the end a ret instruction is two folded: after setting the flag we need to execute any kind instruction to trigger the SEH, furthermore, we have to return to main in order to adhere to the /GS compiler option (which could also be disabled entirely as an alternative solution).

Now, we can RELEASE our VS2019 project and give it a shot from the command line. Once we hit the system PAUSE, we can promptly attach WinDBG to the target process and after resuming execution we can happily spot the ‘Single step exception’ being triggered, which confirms that our plan worked out as expected.

0:004> g
(1b74.27bc): Single step exception - code 80000004 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.

Continuing the execution flows, will we get the final proof from the program itself, confirming it has correctly detected the debugger presence.

That’s all it takes to thwart debuggers on Windows x64. Now it’s time to warm up the engine of the real obfuscators >:)

All the referenced code can be found here