KVM2KFX

KVM2KFX

Transplanting a KVM VM for fuzzing on Xen. The following instructions detail how to install a KVM host kernel + QEMU that will pause on the execution of a magic CPUID instruction executed in a guest VM. Using the magic instruction we can precisely pause the VM as part of a fuzzing harness at the start point. Once the VM is paused, the VM state (memory + CPU) can be exported and then loaded up on Xen for fuzzing using KF/x.

KVM setup

0. Grab git repo

cd ~
git clone https://github.com/intel/kernel-fuzzer-for-xen-project

1. Install build dependencies

sudo apt-get install build-essential bc fakeroot flex bison libelf-dev libssl-dev ncurses-dev libpixman-1-dev pkg-config zlib1g-dev libglib2.0-dev dh-autoreconf libpci-dev libudev-dev autoconf gawk git elfutils libvirt-clients virt-manager ninja-build

2. Patch Linux

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.tar.xz
tar xvf linux-5.15.tar.xz
cd linux-5.15
patch -p1 < ~/kernel-fuzzer-for-xen-project/patches/0001-linux-kfx-cpuid-vmexit.patch
make x86_64_defconfig
make deb-pkg
sudo dpkg -i ../linux-image-*
cd ~

3. Patch QEMU

wget https://download.qemu.org/qemu-6.0.1.tar.xz
tar xvf qemu-6.0.1.tar.xz
cd qemu-6.0.1
patch -p1 < ~/kernel-fuzzer-for-xen-project/patches/0001-qemu-*.patch
patch -p1 < ~/kernel-fuzzer-for-xen-project/pathces/0002-qemu-*.patch
./configure --target-list=x86_64-softmmu --prefix=/opt/qemu-kfx
make
sudo make install
echo "/opt/qemu-kfx/bin/qemu-system-x86_64 PUx," | sudo tee /etc/apparmor.d/local/usr.sbin.libvirtd
sudo /etc/init.d/apparmor restart
cd ..

4. Reboot into new kernel

sudo reboot

5. Create VM with new QEMU

When using virsh in your VM's config, change emulator line of VM in XML config:

   <emulator>/opt/qemu-kfx/bin/qemu-system-x86_64</emulator>

For example in virt-manager you can edit the VM's configuration XML after you enable Edit -> Preferences -> Enable XML editing.

Also make sure your VM has only a single CPU! Multi-CPU VMs are not currently supported.

After you change the config make sure to save it, and then start your VM.

Note: If you are launching the VM with QEMU directly just use /opt/qemu-kfx/bin/qemu-system-x86_64 as your QEMU and add -qmp tcp:127.0.0.1:44444,server,nowait -monitor telnet:127.0.0.1:55555,server,nowait so that you can issue the required monitor and qmp commands to QEMU directly. Adjust the scripts and commands described here accordingly.

6. Execute magic CPUID in VM

Log into the VM and execute:

cpuid -l 0x13371337

The VM state should show as paused in virsh list

7. Save VM

sudo ~/kernel-fuzzer-for-xen-project/scripts/kvm2kfx.sh <vmname>

Several files should now be in your folder: <vmname>-{regmap, memmap, vmcore};

8. Change CPUID that the VM will pause on:

For example to use 0x13371338 instead of the default 0x13371337:

Convert to decimal:

echo $((16#13371338))

Note output: 332376504

Run:

sudo virsh qemu-monitor-command <vmname> '{ "execute": "kfx", "arguments": { "cpuid": 332376504 } }'

Unpause VM

sudo virsh resume <vmname>

Execute new CPUID in VM:

cpuid -l 0x13371338

Create new save file as necessary.

If you are not using virsh you can pass the required qmp command to QEMU by piping

{ "execute": "qmp_capabilities" }
{ "execute": "kfx", "arguments": { "cpuid": 332376504 } }

to a tool like netcat:

cat change-cpuid.txt | nc 127.0.0.1 44444

Xen setup

0. Transfer saved files from KVM host

scp <vmname>-* [email protected]:~

1. Create VM shell for transplant

Use the following as transplant.cfg (change memory to be larger then it was on KVM by at least 1Gb):

arch = 'x86_64'
name = "transplant"
memory = 9000
vcpus = 1
type = "hvm"
hap = 1
vga = "none"
vnc = 0
vmtrace_buf_kb = 65536

Run:

xl create -p -e transplant.cfg
xl list

Note the transplant VMs domainid.

2. Transplant the VM

~/kernel-fuzzer-for-xen-project/xen-transplant <transplant_domainid> <vmname>-regmap <vmname>-memmap <vmname>-vmcore