Debugging 64-bit Windows Binaries
Saturday, October 04, 2014 | Author: Deep Flash
I wanted to write about debugging Windows 64-bit executables. Olly Debugger is the most commonly used user mode debugger for 32-bit Windows Binaries. However, the 64-bit version of Olly Debugger is not yet developed and available. You could either use IDA Pro to perform static analysis of 64-bit Windows Binaries or use the Debugging Tools provided by Windows (windbg). However, the interface provided by Olly Debugger is much more convenient to use.

Fortunately, now we have a debugger which supports 64-bit Windows Binaries as well. It is called x64_dbg and is available for download at http://sourceforge.net/projects/x64dbg/

The official site of the debugger is: http://x64dbg.com and the project is maintained at bitbucket.org

The interface provided by this debugger is similar to Olly Debugger which is great. It has support for plugins similar to Olly Debugger as well.

There is a good description of the components (for instance, Debugging Core engine, Disassembler Engine) used to develop x64_dbg here: http://x64dbg.com/#credits

Now, let us use x64_dbg to debug a binary and understand how the arguments are passed to a function in 64-bit Binaries. We know that in 32-bit Windows Binaries, the arguments are passed through the stack.

Let us consider a function called sum() which takes 4 arguments as an input and calculates the sum of these numbers. For a 32-bit binary, the arguments would be passed on the stack as shown below:

The C program which implements the sum() function:



And the corresponding disassembly:

push d
push c
push b
push a
call sum
mov ebx, eax

Inside the function sum, the arguments would be accessed as shown below:

push ebp
mov ebp, esp
sub esp,
mov eax, dword ptr ds:[ebp+0x8]
mov ebx, dword ptr ds:[ebp+0xc]
mov ecx, dword ptr ds:[ebp+0x10]
mov edx, dword ptr ds:[ebp+0x14]

As you can see above, the function, sum accesses the arguments from the stack using ebp (function prolog will set ebp to esp). Both the local variables and the arguments are referenced using ebp.

Now, let us consider the same function in a 64-bit binary and see how the arguments are passed to the function as shown below.


First, we locate the main subroutine. You can locate the main subroutine similar to the way we locate it for 32-bit Windows Binaries. Below we can see the call to main() subroutine with the two arguments, argc and argv passed to it.


Inside the main subroutine, let us look at how the parameters are passed to the function, sum() and how they are accessed within the function.




As you can see, the 4 arguments are passed to the function sum using the registers, rcx, rdx, r8 and r9. There is still a similarity with the calling convention used in 32-bit Windows Binaries and that is, the return value will be present in the rax register.

Now, let us look at the disassembly of function, sum():


Here, we can see that the 4 arguments are accessed from the registers and saved on the stack before the space for local variables is created on the stack.

So, now we know how the arguments are passed in the 64-bit Windows Calling convention and this will be helpful while reversing 64-bit windows binaries. Please note that there is a similar calling convention used for 64-bit binaries on Linux as well.