gdb vs. ptrace ...... fight!

ptrace is very commonly used in software protections schemes, because it can be very effective. I see 3 levels of ptrace based protection, 2 of which can be addressed with gdb in some way.

1 - a single ptrace call.

This is an almost trivial anti-debugging method. There is a single ptrace() call in the executable. Consider this detect_ptrace.c

#include <stdio.h>
#include <sys/ptrace.h>

int main() {
    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) {
        printf("Debugger\n");
    } else {
        printf("Normal\n");
    }
    return 0;
}

If the application is being debugged, ptrace returns -1 and is detected in that way by the above code.

$ gcc detect_ptrace.c 
$ chmod +x a.out 
$ ./a.out 
Normal

$ gdb a.out
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
[ ... ]
(gdb) r
Starting program: /home/jan/Downloads/a.out 
Debugger
[Inferior 1 (process 16137) exited normally]
(gdb) quit
$

This is easy to circumvent in gdb:

$ gdb a.out
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
[ ... ]
(gdb) catch syscall ptrace
Catchpoint 1 (syscall 'ptrace' [101])
(gdb) commands 1
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>set $eax = 0
>continue
>end
(gdb) r
Starting program: /home/jan/Downloads/a.out 

Catchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45  ../sysdeps/unix/sysv/linux/ptrace.c: No such file or directory.

Catchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45 in ../sysdeps/unix/sysv/linux/ptrace.c
Normal
[Inferior 1 (process 16262) exited normally]
(gdb)

What happens here is that catch syscall ptrace sets a breakpoint on ptrace. gdb allows for execution of a list of commands upon entering a breakpoint. Since the catch command created the first catchpoint, we can enter these commands for catchpoint 1 with commands 1. set $eax = 0 sets the return value. The ptrace function call would set $eax to 0 if successful and -1 if it fails. continue means just that - continue program execution. end tells gdb that we are done entering commands.

Note that we actually break twice, once on entering the ptrace syscall and once on exit.

2 - multiple ptrace calls in single thread

This is tougher, especially if the code is obfuscated. Some binaries execute ptrace more than once, checking that the first time the return code is 0 and that future calls are -1. Often this check is obfuscated to various degrees. Consider this detect_ptrace.c

#include <stdio.h>
#include <sys/ptrace.h>

int main() 
{
    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) // first call
        {
        printf("Debugger (first check)\n");
        } 
    else 
        {
        if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) // second call
            {
            printf("Normal\n");
            } 
        else 
            {
            printf("Debugger (second check)\n");
            }
        }
return 0;
}



$ gcc detect_ptrace2.c 
jan@jan-HP-ENVY-17-Notebook-PC:~/Downloads$ ./a.out 
Normal
jan@jan-HP-ENVY-17-Notebook-PC:~/Downloads$ gdb a.out
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
[ ... ]
(gdb) r
Starting program: /home/jan/Downloads/a.out 
Debugger (first check)
[Inferior 1 (process 16818) exited normally]
(gdb) quit

$ gdb a.out
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
[ ... ]
(gdb) catch syscall ptrace
Catchpoint 1 (syscall 'ptrace' [101])
(gdb) commands 1
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>set $eax = 0
>continue
>end
(gdb) r
Starting program: /home/jan/Downloads/a.out 

Catchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45  ../sysdeps/unix/sysv/linux/ptrace.c: No such file or directory.

Catchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45 in ../sysdeps/unix/sysv/linux/ptrace.c

Catchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45 in ../sysdeps/unix/sysv/linux/ptrace.c

Catchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:45
45 in ../sysdeps/unix/sysv/linux/ptrace.c
Debugger (second check)
[Inferior 1 (process 16856) exited normally]
(gdb)

This can be addressed by first determining the address of the first ptrace call, and then writing a script that checks for that address. Here an example for an executable that has the first ptrace call at 0x4d55dc which then proceeds to print out the addresses of all the other ptrace calls:

catch syscall ptrace
commands 1
 if ($rip) == 0x4d55dc
  print "yep"
  print $rip
  set $rax = 0
  continue
 else
  print "nope"
  print $rip
  set $rax = -1
  continue
 end
end

In that particular example, I was able to take a list of the addresses where ptrace was called and tell ghidra that that was code:

3 - multi-threaded ptrace - self-debugging & nanomites.

Some advanced protection systems fork a thread which then tries to attach to the main thread as a debugger. This will fail if gdb is already attached as a debugger. And gdb will not be able to attach to a thread that has done this already. nanomites protection goes one step further, replacing some of the code with instructions that trip the debugger. This debugger then performs some work before returning control to the child. This type of protection needs a more advanced approach, manual unpacking, PANDA tracing or some such thing. But that is a topic for another time.