PANDA - Whole System Tracing

PANDA is a platform that allows the complete tracing of execution of a virtual machine. This is incredibly useful. The system trace can be operated on after the fact using a rich API. With this API a so-called plugin is written, and this plugin then operates on the trace. There are many sample plugins available already ranging from capturing the GUI output to a video to extracting binary coverage information for IDA pro. Setting it up can be tricky, especially with newer kernels, and so I am documenting my process here. I like to use live DVDs as the client VMs, and so this example covers setup of PANDA for the usage of that case.

1: Install PANDA on host

I am drawing from the install process is described at : https://github.com/panda-re/panda/blob/master/panda/docs/build_ubuntu.md

On a clean Ubuntu 18.04.03

$ sudo apt install git

# clone and build PANDA. This does download PANDA twice, but it's easy this way. 

$ git clone https://github.com/panda-re/panda
$ cd panda/panda/scripts/
$ ./install_ubuntu.sh
$ cd panda/build/x86_64-softmmu
$ ./panda-system-x86_64 --monitor stdio -m 4096 -cdrom '/home/ubuntu/Downloads/ubuntu-18.04.4-desktop-amd64.iso'

This will launch a virtual machine which can be traced, in this case a Ubuntu 18.04.04 live DVD

2: Trace execution


The --monitor stdio option kept the qemu command prompt open to receive commands and this is where we issue the command to start recording

(qemu) begin_record therecording
(qemu) writing snapshot:  ./therecording-rr-snp
opening nondet log for write: ./therecording-rr-nondet.log
end_record
(qemu) Time taken was: 7 seconds.
Checksum of guest memory: 0x4e5c394a

And at some point we end the recording

(qemu) end_record

This record can now be operated on. One possibility is to generate a movie showing what the GUI displayed during the recording session.

$ sudo apt install ffmpeg
$ ./panda-system-x86_64 -m 4096 -replay therecording -panda replaymovie
$ ../../../../plugins/replaymovie/movie.sh
$ ffplay replay.mp4

And this can then be turned into an animated GIF, if you have need for that.

$ ffmpeg -y -i replay.mp4 -vf palettegen palette.png
$ ffmpeg -i replay.mp4 -i palette.png -lavfi paletteuse  out.gif

The animation below is the outcome of this recording.

3: Generate osi data for client

Next, in order for PANDA to be able to dissect the client system, we have to collect some information inside it. So from within the virtual machine,

$ sudo apt install git build-essential linux-headers-`uname -r`
$ git clone https://github.com/panda-re/panda
$ cd panda/panda/plugins/osi_linux/utils/kernelinfo
$ make
$ sudo insmod kernelinfo.ko

The above command will fail with an error message, but it will provide the required information through dmesg

$ dmesg

However: Newer Linux kernels randomize the address space so these numbers change every boot.


This can be turned off at boot time by passing the kernel boot parameter nokaslr

I have collected this information for the boot DVDs for Ubuntu 18.04.3 and 18.04.4 with KASLR disabled:

Update: These kernel parameters have been merged to PANDA: https://github.com/panda-re/panda/pull/559

So you don't have to collect it if you are using those two kernels.

for 18.04.3:

name = 5.0.0-23-generic|#24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019|x86_64
version.a = 5
version.b = 0
version.c = 15
task.per_cpu_offsets_addr = 18446744071599519776
task.per_cpu_offset_0_addr = 18446612687365341184
task.current_task_addr = 89088
task.init_addr = 18446744071602009920
#task.per_cpu_offsets_addr = FFFFFFFF823B7820
#task.per_cpu_offset_0_addr = 0xFFFF88813BA00000
#task.current_task_addr = 0x00015C00
#task.init_addr = 0xFFFFFFFF82617740
task.size = 9152
task.tasks_offset = 1960
task.pid_offset = 2216
task.tgid_offset = 2220
task.group_leader_offset = 2280
task.thread_group_offset = 2392
task.real_parent_offset = 2232
task.parent_offset = 2240
task.mm_offset = 2040
task.stack_offset = 24
task.real_cred_offset = 2624
task.cred_offset = 2632
task.comm_offset = 2640
task.comm_size = 16
task.files_offset = 2712
cred.uid_offset = 4
cred.gid_offset = 8
cred.euid_offset = 20
cred.egid_offset = 24
mm.size = 1032
mm.mmap_offset = 0
mm.pgd_offset = 80
mm.arg_start_offset = 304
mm.start_brk_offset = 280
mm.brk_offset = 288
mm.start_stack_offset = 296
vma.size = 208
vma.vm_mm_offset = 64
vma.vm_start_offset = 0
vma.vm_end_offset = 8
vma.vm_next_offset = 16
vma.vm_flags_offset = 80
vma.vm_file_offset = 160
fs.f_path_dentry_offset = 24
fs.f_path_mnt_offset = 16
fs.f_pos_offset = 104
fs.fdt_offset = 32
fs.fdtab_offset = 40
fs.fd_offset = 8
qstr.size = 16
qstr.name_offset = 8
path.d_name_offset = 32
path.d_iname_offset = 56
path.d_parent_offset = 24
path.d_op_offset = 96
path.d_dname_offset = 72
path.mnt_root_offset = 0
path.mnt_parent_offset = -16
path.mnt_mountpoint_offset = -8

and for 18.04.4:

name = 5.3.0-28-generic|#30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020|x86_64
version.a = 5
version.b = 3
version.c = 13
task.per_cpu_offsets_addr = 18446744071599864096
task.per_cpu_offset_0_addr = 18446612687365341184
task.current_task_addr = 93120
task.init_addr = 18446744071601993600
#task.per_cpu_offsets_addr = FFFFFFFF8240B920
#task.per_cpu_offset_0_addr = 0xFFFF88813BA00000
#task.current_task_addr = 0x00016BC0
#task.init_addr = 0xFFFFFFFF82613780
task.size = 9152
task.tasks_offset = 1984
task.pid_offset = 2240
task.tgid_offset = 2244
task.group_leader_offset = 2304
task.thread_group_offset = 2416
task.real_parent_offset = 2256
task.parent_offset = 2264
task.mm_offset = 2064
task.stack_offset = 24
task.real_cred_offset = 2648
task.cred_offset = 2656
task.comm_offset = 2672
task.comm_size = 16
task.files_offset = 2744
cred.uid_offset = 4
cred.gid_offset = 8
cred.euid_offset = 20
cred.egid_offset = 24
mm.size = 1032
mm.mmap_offset = 0
mm.pgd_offset = 80
mm.arg_start_offset = 304
mm.start_brk_offset = 280
mm.brk_offset = 288
mm.start_stack_offset = 296
vma.size = 208
vma.vm_mm_offset = 64
vma.vm_start_offset = 0
vma.vm_end_offset = 8
vma.vm_next_offset = 16
vma.vm_flags_offset = 80
vma.vm_file_offset = 160
fs.f_path_dentry_offset = 24
fs.f_path_mnt_offset = 16
fs.f_pos_offset = 104
fs.fdt_offset = 32
fs.fdtab_offset = 40
fs.fd_offset = 8
qstr.size = 16
qstr.name_offset = 8
path.d_name_offset = 32
path.d_iname_offset = 56
path.d_parent_offset = 24
path.d_op_offset = 96
path.d_dname_offset = 72
path.mnt_root_offset = 0
path.mnt_parent_offset = -16
path.mnt_mountpoint_offset = -8

This information should be in panda/plugins/kernelinfo.conf

Now the client VM system is basically set up for tracing. However, user mode binaries also use ASLR by default, and it is helpful in many cases to disable that too. This can be done by running the following command in the VM:

more /proc/sys/kernel/randomize_va_space
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

4: Test plugins

We can now test the PANDA installation with some of its' other built-in plugins. There is a long list of existing plugins here:

First, lets test the OSI data we collected above:

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ ./panda-system-x86_64 -m 4096 -replay therecording -panda osi -panda osi_linux:kconf_group=ubuntu:5.3.0-28-generic:64 -os linux-64-ubuntu -panda osi_test > ositest.txt
PANDA[osi_linux]:adding argument kconf_group=ubuntu:5.3.0-28-generic:64.
PANDA[core]:os_familyno=2 bits=64 os_details=ubuntu
PANDA[core]:initializing osi
PANDA[core]:loading required plugin osi_linux
PANDA[core]:initializing osi_linux
PANDA[osi_linux]:W> kernelinfo bytes [20-23] not read
PANDA[osi_linux]:W> kernelinfo bytes [124-127] not read
PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi_linux.so already loaded
PANDA[core]:initializing osi_test
PANDA[core]:loading required plugin osi
PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi.so already loaded
ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ tail ositest.txt 
RR_INTERRUPT_REQUEST number = 2620, size = 36680 bytes
RR_EXIT_REQUEST number = 0, size = 0 bytes
RR_SKIPPED_CALL number = 0, size = 0 bytes
RR_END_OF_LOG number = 1, size = 10 bytes
RR_PENDING_INTERRUPTS number = 0, size = 0 bytes
RR_EXCEPTION number = 0, size = 0 bytes
max_queue_len = 397
Checksum of guest memory: 0x4e5c394a
Replay completed successfully
Exiting cpu_handle_execption loop
ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$

Exellent! Lets try to collect process coverage data:

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ ./panda-system-x86_64 -m 4096 -replay therecording -panda osi -panda osi_linux:kconf_group=ubuntu:5.3.0-28-generic:64 -os linux-64-ubuntu -panda coverage:filename=test_coverage.csv,mode=process
PANDA[osi_linux]:adding argument kconf_group=ubuntu:5.3.0-28-generic:64.
PANDA[core]:os_familyno=2 bits=64 os_details=ubuntu
PANDA[coverage]:adding argument filename=test_coverage.csv.
PANDA[coverage]:adding argument mode=process.
PANDA[core]:initializing osi
PANDA[core]:loading required plugin osi_linux
PANDA[core]:initializing osi_linux
PANDA[osi_linux]:W> kernelinfo bytes [20-23] not read
PANDA[osi_linux]:W> kernelinfo bytes [124-127] not read
PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi_linux.so already loaded
PANDA[core]:initializing coverage
PANDA[coverage]:output file name test_coverage.csv
PANDA[coverage]:file buffer_size 8192
PANDA[coverage]:log all records DISABLED
PANDA[coverage]:mode process
PANDA[core]:loading required plugin osi
PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi.so already loaded
PANDA[coverage]:start disabled DISABLED
loading snapshot
... done.
opening nondet log for read :  ./therecording-rr-nondet.log
./therecording-rr-nondet.log:   139046873 instrs total.
therecording:     1390473 (  1.00%) instrs.    0.59 sec.  4.04 GB ram.
therecording:     2780941 (  2.00%) instrs.    0.81 sec.  4.04 GB ram.
therecording:     4171411 (  3.00%) instrs.    1.06 sec.  4.05 GB ram.
therecording:     5561875 (  4.00%) instrs.    1.30 sec.  4.05 GB ram.
therecording:     6952345 (  5.00%) instrs.    1.52 sec.  4.06 GB ram.

[ ... ]

therecording:   134875470 ( 97.00%) instrs.   14.07 sec.  4.13 GB ram.
therecording:   136265941 ( 98.00%) instrs.   14.23 sec.  4.13 GB ram.
therecording:   137656406 ( 99.00%) instrs.   14.37 sec.  4.13 GB ram.
./therecording-rr-nondet.log:  log is empty.
./therecording-rr-nondet.log:  log is empty.
Time taken was: 14 seconds.
Stats:
RR_INPUT_1 number = 0, size = 0 bytes
RR_INPUT_2 number = 0, size = 0 bytes
RR_INPUT_4 number = 1413, size = 19782 bytes
RR_INPUT_8 number = 18855, size = 339390 bytes
RR_INTERRUPT_REQUEST number = 2620, size = 36680 bytes
RR_EXIT_REQUEST number = 0, size = 0 bytes
RR_SKIPPED_CALL number = 0, size = 0 bytes
RR_END_OF_LOG number = 1, size = 10 bytes
RR_PENDING_INTERRUPTS number = 0, size = 0 bytes
RR_EXCEPTION number = 0, size = 0 bytes
max_queue_len = 397
Checksum of guest memory: 0x4e5c394a
Replay completed successfully
Exiting cpu_handle_execption loop
ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$

Great! We now have PANDA set up to record coverage data for processes launched straight from a live DVD. This coverage data can then be imported into IDA pro. But that is a topic for another day...

Appendix: file exchange

Bridged networking is a pain. I prefer to make a drive image 

$ dd if=/dev/zero of=diskimg.ext3 bs=100M count=1
$ mkfs.ext3 diskimg.ext3
$ sudo ./panda-system-x86_64 --enable-kvm --monitor stdio -m 4096 -cdrom 'ubuntu-18.04.4-desktop-amd64.iso' -hda diskimg.ext3

To mount the file in the host after exiting PANDA

$ mkdir /home/ubuntu/vmshare
$ sudo mount -o loop diskimg.ext3 /home/ubuntu/vmshare/

and to unmount the file

$ umount /home/ubuntu/vmshare

Appendix: Building without the install_ubuntu.sh script

The instructions above are the simplest way to get going. However, the script - install_ubuntu.sh - downloads another copy of PANDA using git. And sometimes one wants to recompile after making some code changes. So, if PANDA is already compiled on the computer, and compiled and the prerequisites are installed, one can compile manually like this:

Make sure to adjust the prefix directoy to the location on your computer..

$ ./configure --target-list=x86_64-softmmu,i386-softmmu,arm-softmmu,ppc-softmmu --prefix=/home/jan/Downloads/panda/panda/scripts/panda/build/install --python=/usr/bin/python2 --disable-vhost-net --extra-cflags=-DXC_WANT_COMPAT_DEVICEMODEL_API --extra-cflags=-DOSI_PROC_EVENTS --extra-cflags=-DOSI_MAX_PROC=256 --extra-cflags=-DOSI_LINUX_PSDEBUG --extra-cflags=-Wformat-truncation=0 --disable-werror --cc=gcc-9 --cxx=g++-9 --host-cc=gcc-9
$ make