Angr, qira & PIN vs. ptrace anti-debug & anti-symbolic-execution

I stumbled across a very interesting anti-debugging technique using ptrace that also trips up Angr unless special ptrace behavior is explained to it. The executable under test produces a double ptrace call:

The first call is familiar. It's just a standard call with a 0/-1 check to see if the executable is being traced. But the second call return value is used in a calculation, and then again from a third call later. From Ghidra:

  lVar2 = ptrace(PTRACE_TRACEME,0,0,0);

  local_c = (int)lVar2;

  if (local_c == -1) {

    FUN_00400786();

  }

  lVar2 = ptrace(PTRACE_TRACEME,0,0);

 [...SNIP...]

  lVar2 = ptrace(PTRACE_TRACEME,0,0,0);

  local_50 = (uint)lVar2;

  local_c = local_50 + local_c + local_10 + local_14;

  if (local_c == 0) {

    puts("You win!");

  }

So what does a repeated ptrace(TRACEME,....) return in a non-debugged application? I did not know. And I could not find any documentation about this either. It could be an error, because we are already attached. Or it could be smart enough to see that it's the same PID attaching and simply succeed a second time. And not knowing, we can not just patch out any of the ptrace calls, because we need to retain the behavior. The first call may affect the second, and there is a third call later in the program as well. Interesting. We can invert or completely patch the jnz instruction after the -1 check, of course. To analyze a program's behavior, I like to use qira, and the standard qira uses QEMU in user mode. Apparently, this utilizes some sort of ptrace interface, and that implies:

• that the ptrace check for -1 fails and
• if the first check is patched, we don't know if the subsequent ptrace calls return the same value they would if not run under QEMU

Now, we could just write a small program that doubles up an ptrace calls and prints out the return values and then use that information to patch the executable. However, there is a provision to use a PIN tracing tool in qira, and that avoids the problem:

$ qira --pin ./executable

Tracing the program yields the very interesting information that a repeated call to ptrace(TRACEME,....) in a program that is not being debugged returns 0 for the first call and -1 for all subsequent calls.

Nice. Knowing that it was just the usual work flow of using qira-IDA-Ghidra-retdec-radare2-etc. to understand the program logic. The program applies some complicated formula to the input and requires some sort of constraint solver to get at the solution. And this brings us to the anti-symbolic-execution effect of the repeated ptrace calls.

This is a known issue; see this pull request discussion:

https://github.com/angr/simuvex/pull/78

and here:

https://github.com/angr/simuvex/pull/78/commits/90af0227e0c5d61a0756625a5d0e6c638363652e

For now, ptrace simply returns an unconstrained value.

Testing using the following hooks (more on hooks later):

def hook_ptrace1_before_call(state):

        print('rax before ptrace call 1: ' + str(state.regs.rax))

def hook_ptrace1_after_call(state):

        print('rax after ptrace call 1: ' + str(state.regs.rax))

results in:

rax before ptrace call 1: <SAO <BV64 0x0>>

rax after ptrace call 1: <SAO <BV64 unconstrained_ret_ptrace_15_64{UNINITIALIZED}>>

However, the right return value of ptrace is critical to yielding a properly constrained system. Without it we get an explosion of the solution space.

Do do this, what we can do is 'hook' the execution of the ptrace calls. A hook is a mechanism allowing the change the behavior of the program flow. We need 3 pieces of information:

• the address where we want to 'hook'
• what we want to do at that address
• the length of the instruction(s) we want to skip in bytes

Lets look at it. From Ghidra:

        00400943 b8  00  00       MOV        EAX ,0x0

                 00  00

        00400948 e8  03  fd       CALL       ptrace                                           long ptrace(__ptrace_request __r

                 ff  ff

        0040094d 89  45  fc       MOV        dword ptr [RBP  + local_c ],EAX

So the first call to ptrace is at address 0x00400948. and the instruction is 5 bytes long. Lets take the test hook from above as an example:

p = angr.Project('./executable',auto_load_libs=False)

def hook_ptrace1_before_call(state):

        print('rax before ptrace call 1: ' + str(state.regs.rax))

p.hook(0x400948, hook_ptrace1_before_call,0)

The first piece of information, the address, is the first parameter to the hook() call. The second piece of information, what we want to do, is the function hook_ptrace1_before_call(), which is the second parameter to the hook() call. The third piece of information, the length of instructions to skip, is zero in this case, because we don't want to skip any instructions. We are just printing out the state of rax.

However, to solve this challenge, we do want to skip the actual ptrace call and furthermore simulate it's effect on rax. The X86 instuction set has variable length instructions, but we can see from the Ghidra snippet above that 5 bytes are used for the ptrace call. So,

def hook_ptrace1(state):

        print("ptrace1 hooked")

        state.regs.rax = 0

p.hook(0x400948, hook_ptrace1, length=5)

Will do the following:

• Whenever address 0x400948 is executed, "ptrace1 hooked" is printed and rax is set to 0.
• The actual instruction, CALL ptrace, is skipped

We can then combine that with further hooks for the other ptrace calls to force the behavior we learned from the PIN tracer:

def hook_ptrace1(state):

        print("ptrace1 hooked")

        state.regs.rax = 0

def hook_ptrace2(state):

        print("ptrace2 hooked")

        state.regs.rax = -1

def hook_ptrace3(state):

        print("ptrace3 hooked")

        state.regs.rax = -1

p.hook(0x400948, hook_ptrace1, length=5)

p.hook(0x400979, hook_ptrace2, length=5)

p.hook(0x400BEB, hook_ptrace3, length=5)

And after that the rest is just standard angr solving for the flag.....