PANDA - Whole System Tracing

PANDA is a platform that allows the complete tracing of execution of a virtual machine. This is incredibly useful. The system trace can be operated on after the fact using a rich API. With this API a so-called plugin is written, and this plugin then operates on the trace. There are many sample plugins available already ranging from capturing the GUI output to a video to extracting binary coverage information for IDA pro. Setting it up can be tricky, especially with newer kernels, and so I am documenting my process here. I like to use live DVDs as the client VMs, and so this example covers setup of PANDA for the usage of that case.

1: Install PANDA on host

I am drawing from the install process is described at : https://github.com/panda-re/panda/blob/master/panda/docs/build_ubuntu.md

On a clean Ubuntu 18.04.03

$ sudo apt install git

# clone and build PANDA. This does download PANDA twice, but it's easy this way.

$ git clone https://github.com/panda-re/panda

$ cd panda/panda/scripts/

$ ./install_ubuntu.sh

$ cd panda/build/x86_64-softmmu

$ ./panda-system-x86_64 --monitor stdio -m 4096 -cdrom '/home/ubuntu/Downloads/ubuntu-18.04.4-desktop-amd64.iso'

This will launch a virtual machine which can be traced, in this case a Ubuntu 18.04.04 live DVD

2: Trace execution

The --monitor stdio option kept the qemu command prompt open to receive commands and this is where we issue the command to start recording

(qemu) begin_record therecording

(qemu) writing snapshot:  ./therecording-rr-snp

opening nondet log for write: ./therecording-rr-nondet.log

end_record

(qemu) Time taken was: 7 seconds.

Checksum of guest memory: 0x4e5c394a

And at some point we end the recording

(qemu) end_record

This record can now be operated on. One possibility is to generate a movie showing what the GUI displayed during the recording session.

$ sudo apt install ffmpeg

$ ./panda-system-x86_64 -m 4096 -replay therecording -panda replaymovie

$ ../../../../plugins/replaymovie/movie.sh

$ ffplay replay.mp4

And this can then be turned into an animated GIF, if you have need for that.

$ ffmpeg -y -i replay.mp4 -vf palettegen palette.png

$ ffmpeg -i replay.mp4 -i palette.png -lavfi paletteuse  out.gif

The animation below is the outcome of this recording.

3: Generate osi data for client

Next, in order for PANDA to be able to dissect the client system, we have to collect some information inside it. So from within the virtual machine,

$ sudo apt install git build-essential linux-headers-`uname -r`

$ git clone https://github.com/panda-re/panda

$ cd panda/panda/plugins/osi_linux/utils/kernelinfo

$ make

$ sudo insmod kernelinfo.ko

The above command will fail with an error message, but it will provide the required information through dmesg

$ dmesg

However: Newer Linux kernels randomize the address space so these numbers change every boot.

This can be turned off at boot time by passing the kernel boot parameter nokaslr

I have collected this information for the boot DVDs for Ubuntu 18.04.3 and 18.04.4 with KASLR disabled:

Update: These kernel parameters have been merged to PANDA: https://github.com/panda-re/panda/pull/559

So you don't have to collect it if you are using those two kernels.

for 18.04.3:

name = 5.0.0-23-generic|#24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019|x86_64

version.a = 5

version.b = 0

version.c = 15

task.per_cpu_offsets_addr = 18446744071599519776

task.per_cpu_offset_0_addr = 18446612687365341184

task.current_task_addr = 89088

task.init_addr = 18446744071602009920

#task.per_cpu_offsets_addr = FFFFFFFF823B7820

#task.per_cpu_offset_0_addr = 0xFFFF88813BA00000

#task.current_task_addr = 0x00015C00

#task.init_addr = 0xFFFFFFFF82617740

task.size = 9152

task.tasks_offset = 1960

task.pid_offset = 2216

task.tgid_offset = 2220

task.group_leader_offset = 2280

task.thread_group_offset = 2392

task.real_parent_offset = 2232

task.parent_offset = 2240

task.mm_offset = 2040

task.stack_offset = 24

task.real_cred_offset = 2624

task.cred_offset = 2632

task.comm_offset = 2640

task.comm_size = 16

task.files_offset = 2712

cred.uid_offset = 4

cred.gid_offset = 8

cred.euid_offset = 20

cred.egid_offset = 24

mm.size = 1032

mm.mmap_offset = 0

mm.pgd_offset = 80

mm.arg_start_offset = 304

mm.start_brk_offset = 280

mm.brk_offset = 288

mm.start_stack_offset = 296

vma.size = 208

vma.vm_mm_offset = 64

vma.vm_start_offset = 0

vma.vm_end_offset = 8

vma.vm_next_offset = 16

vma.vm_flags_offset = 80

vma.vm_file_offset = 160

fs.f_path_dentry_offset = 24

fs.f_path_mnt_offset = 16

fs.f_pos_offset = 104

fs.fdt_offset = 32

fs.fdtab_offset = 40

fs.fd_offset = 8

qstr.size = 16

qstr.name_offset = 8

path.d_name_offset = 32

path.d_iname_offset = 56

path.d_parent_offset = 24

path.d_op_offset = 96

path.d_dname_offset = 72

path.mnt_root_offset = 0

path.mnt_parent_offset = -16

path.mnt_mountpoint_offset = -8

and for 18.04.4:

name = 5.3.0-28-generic|#30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020|x86_64

version.a = 5

version.b = 3

version.c = 13

task.per_cpu_offsets_addr = 18446744071599864096

task.per_cpu_offset_0_addr = 18446612687365341184

task.current_task_addr = 93120

task.init_addr = 18446744071601993600

#task.per_cpu_offsets_addr = FFFFFFFF8240B920

#task.per_cpu_offset_0_addr = 0xFFFF88813BA00000

#task.current_task_addr = 0x00016BC0

#task.init_addr = 0xFFFFFFFF82613780

task.size = 9152

task.tasks_offset = 1984

task.pid_offset = 2240

task.tgid_offset = 2244

task.group_leader_offset = 2304

task.thread_group_offset = 2416

task.real_parent_offset = 2256

task.parent_offset = 2264

task.mm_offset = 2064

task.stack_offset = 24

task.real_cred_offset = 2648

task.cred_offset = 2656

task.comm_offset = 2672

task.comm_size = 16

task.files_offset = 2744

cred.uid_offset = 4

cred.gid_offset = 8

cred.euid_offset = 20

cred.egid_offset = 24

mm.size = 1032

mm.mmap_offset = 0

mm.pgd_offset = 80

mm.arg_start_offset = 304

mm.start_brk_offset = 280

mm.brk_offset = 288

mm.start_stack_offset = 296

vma.size = 208

vma.vm_mm_offset = 64

vma.vm_start_offset = 0

vma.vm_end_offset = 8

vma.vm_next_offset = 16

vma.vm_flags_offset = 80

vma.vm_file_offset = 160

fs.f_path_dentry_offset = 24

fs.f_path_mnt_offset = 16

fs.f_pos_offset = 104

fs.fdt_offset = 32

fs.fdtab_offset = 40

fs.fd_offset = 8

qstr.size = 16

qstr.name_offset = 8

path.d_name_offset = 32

path.d_iname_offset = 56

path.d_parent_offset = 24

path.d_op_offset = 96

path.d_dname_offset = 72

path.mnt_root_offset = 0

path.mnt_parent_offset = -16

path.mnt_mountpoint_offset = -8

This information should be in panda/plugins/kernelinfo.conf

Now the client VM system is basically set up for tracing. However, user mode binaries also use ASLR by default, and it is helpful in many cases to disable that too. This can be done by running the following command in the VM:

more /proc/sys/kernel/randomize_va_space

echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

4: Test plugins

We can now test the PANDA installation with some of its' other built-in plugins. There is a long list of existing plugins here:

https://github.com/panda-re/panda/tree/master/panda/plugins

First, lets test the OSI data we collected above:

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ ./panda-system-x86_64 -m 4096 -replay therecording -panda osi -panda osi_linux:kconf_group=ubuntu:5.3.0-28-generic:64 -os linux-64-ubuntu -panda osi_test > ositest.txt

PANDA[osi_linux]:adding argument kconf_group=ubuntu:5.3.0-28-generic:64.

PANDA[core]:os_familyno=2 bits=64 os_details=ubuntu

PANDA[core]:initializing osi

PANDA[core]:loading required plugin osi_linux

PANDA[core]:initializing osi_linux

PANDA[osi_linux]:W> kernelinfo bytes [20-23] not read

PANDA[osi_linux]:W> kernelinfo bytes [124-127] not read

PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi_linux.so already loaded

PANDA[core]:initializing osi_test

PANDA[core]:loading required plugin osi

PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi.so already loaded

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ tail ositest.txt

RR_INTERRUPT_REQUEST number = 2620, size = 36680 bytes

RR_EXIT_REQUEST number = 0, size = 0 bytes

RR_SKIPPED_CALL number = 0, size = 0 bytes

RR_END_OF_LOG number = 1, size = 10 bytes

RR_PENDING_INTERRUPTS number = 0, size = 0 bytes

RR_EXCEPTION number = 0, size = 0 bytes

max_queue_len = 397

Checksum of guest memory: 0x4e5c394a

Replay completed successfully

Exiting cpu_handle_execption loop

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$

Exellent! Lets try to collect process coverage data:

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$ ./panda-system-x86_64 -m 4096 -replay therecording -panda osi -panda osi_linux:kconf_group=ubuntu:5.3.0-28-generic:64 -os linux-64-ubuntu -panda coverage:filename=test_coverage.csv,mode=process

PANDA[osi_linux]:adding argument kconf_group=ubuntu:5.3.0-28-generic:64.

PANDA[core]:os_familyno=2 bits=64 os_details=ubuntu

PANDA[coverage]:adding argument filename=test_coverage.csv.

PANDA[coverage]:adding argument mode=process.

PANDA[core]:initializing osi

PANDA[core]:loading required plugin osi_linux

PANDA[core]:initializing osi_linux

PANDA[osi_linux]:W> kernelinfo bytes [20-23] not read

PANDA[osi_linux]:W> kernelinfo bytes [124-127] not read

PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi_linux.so already loaded

PANDA[core]:initializing coverage

PANDA[coverage]:output file name test_coverage.csv

PANDA[coverage]:file buffer_size 8192

PANDA[coverage]:log all records DISABLED

PANDA[coverage]:mode process

PANDA[core]:loading required plugin osi

PANDA[core]:/home/ubuntu/panda/panda/scripts/panda/build/x86_64-softmmu/panda/plugins/panda_osi.so already loaded

PANDA[coverage]:start disabled DISABLED

loading snapshot

... done.

opening nondet log for read :  ./therecording-rr-nondet.log

./therecording-rr-nondet.log:   139046873 instrs total.

therecording:     1390473 (  1.00%) instrs.    0.59 sec.  4.04 GB ram.

therecording:     2780941 (  2.00%) instrs.    0.81 sec.  4.04 GB ram.

therecording:     4171411 (  3.00%) instrs.    1.06 sec.  4.05 GB ram.

therecording:     5561875 (  4.00%) instrs.    1.30 sec.  4.05 GB ram.

therecording:     6952345 (  5.00%) instrs.    1.52 sec.  4.06 GB ram.

[ ... ]

therecording:   134875470 ( 97.00%) instrs.   14.07 sec.  4.13 GB ram.

therecording:   136265941 ( 98.00%) instrs.   14.23 sec.  4.13 GB ram.

therecording:   137656406 ( 99.00%) instrs.   14.37 sec.  4.13 GB ram.

./therecording-rr-nondet.log:  log is empty.

./therecording-rr-nondet.log:  log is empty.

Time taken was: 14 seconds.

Stats:

RR_INPUT_1 number = 0, size = 0 bytes

RR_INPUT_2 number = 0, size = 0 bytes

RR_INPUT_4 number = 1413, size = 19782 bytes

RR_INPUT_8 number = 18855, size = 339390 bytes

RR_INTERRUPT_REQUEST number = 2620, size = 36680 bytes

RR_EXIT_REQUEST number = 0, size = 0 bytes

RR_SKIPPED_CALL number = 0, size = 0 bytes

RR_END_OF_LOG number = 1, size = 10 bytes

RR_PENDING_INTERRUPTS number = 0, size = 0 bytes

RR_EXCEPTION number = 0, size = 0 bytes

max_queue_len = 397

Checksum of guest memory: 0x4e5c394a

Replay completed successfully

Exiting cpu_handle_execption loop

ubuntu@ubuntu:~/panda/panda/scripts/panda/build/x86_64-softmmu$

Great! We now have PANDA set up to record coverage data for processes launched straight from a live DVD. This coverage data can then be imported into IDA pro. But that is a topic for another day...

Appendix: file exchange

Bridged networking is a pain. I prefer to make a drive image

$ dd if=/dev/zero of=diskimg.ext3 bs=100M count=1

$ mkfs.ext3 diskimg.ext3

$ sudo ./panda-system-x86_64 --enable-kvm --monitor stdio -m 4096 -cdrom 'ubuntu-18.04.4-desktop-amd64.iso' -hda diskimg.ext3

To mount the file in the host after exiting PANDA

$ mkdir /home/ubuntu/vmshare

$ sudo mount -o loop diskimg.ext3 /home/ubuntu/vmshare/

and to unmount the file

$ umount /home/ubuntu/vmshare

Appendix: Building without the install_ubuntu.sh script

The instructions above are the simplest way to get going. However, the script - install_ubuntu.sh - downloads another copy of PANDA using git. And sometimes one wants to recompile after making some code changes. So, if PANDA is already compiled on the computer, and compiled and the prerequisites are installed, one can compile manually like this:

Make sure to adjust the prefix directoy to the location on your computer..

$ ./configure --target-list=x86_64-softmmu,i386-softmmu,arm-softmmu,ppc-softmmu --prefix=/home/jan/Downloads/panda/panda/scripts/panda/build/install --python=/usr/bin/python2 --disable-vhost-net --extra-cflags=-DXC_WANT_COMPAT_DEVICEMODEL_API --extra-cflags=-DOSI_PROC_EVENTS --extra-cflags=-DOSI_MAX_PROC=256 --extra-cflags=-DOSI_LINUX_PSDEBUG --extra-cflags=-Wformat-truncation=0 --disable-werror --cc=gcc-9 --cxx=g++-9 --host-cc=gcc-9

$ make

References:

PANDA github page https://github.com/panda-re/panda
Building a custom malware sandbox with PANDA https://adalogics.com/blog/Building-a-custom-malware-sandbox-with-PANDA-Part-1
Packer-Complexity Analysis in PANDA https://webthesis.biblio.polito.it/7519/1/tesi.pdf
Generating linux osi data https://github.com/panda-re/panda/blob/master/panda/plugins/osi_linux/README.md
Qemu transfer files between host and guest via the floppy drive - http://wiki.linuxquestions.org/wiki/Qemu_transfer_files_between_host_and_guest_via_the_floppy_drive

Google Sites

Report abuse