[Linux Kernel Exploitation 0x0] Debugging the Kernel with QEMU

Hi folks, in this post I'm going to walk through how to setup the linux kernel for debugging. I will also demonstrate that the setup works by setting a break-point to a test driver I wrote myself. All the code will be available from my gitlab, all the links to my gitlab will be re-posted at the end.

The setup I describe here re-uses some parts of the syzkaller setup, and for good reason later on in the post series I will break into a tutorial for the syzkaller tool as well. So lets get on with it.

Screenshot of a successful debug session with full debug symbols for the kernel! We can even see the call to start_kernel and a frame before that as well!

The Process

Okay so we want to study kernel exploitation but given that the kernel isn't something totally accessible in userspace, its not as convenient to debug as userpace stuff, we need a bit of a run up before we can actually poke and prod the kernel to figure out how to write our exploits. So there's a number of important steps to how we get this done, here's what we're going to do:

Build a kernel
Build an image
Launch the virtual machine
Attach and setup the debugger
Building, loading and debugging a test module

We also need to be able to build our kernel because there may be build options that are important to configure in order to control exploit protection or include modules and functionality to the kernel when needed.

Building a Kernel

Okay so before we get going with launching our Qemu instances and debugging modules we need an environment. For convenience sake I'm working off of a fresh Ubuntu 18.04.5 LTS machine. I'll document the processes from fresh install to first successful kernel build.

To start we need to make sure we have everything we need to build a kernel:

$sudo apt-get update

$sudo apt-get upgrade

$sudo apt-get install git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison qemu-system-x86

Next we obviously need a kernel so lets download a brand new kernel:

$wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.9.7.tar.xz
--2020-11-10 23:00:26-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.9.7.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.225.176, 2a04:4e42:35::432
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.225.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 115538096 (110M) [application/x-xz]
Saving to: ‘linux-5.9.7.tar.xz’

linux-5.9.7.tar.xz 42%[=============> ] 46.79M 3.08MB/s eta 23s

...

$tar -xf linux-5.9.7.tar.xz

We're just a couple steps from sending the final build commands, before we get to that lets make sure the kernel config is ready to rock. Because we're working on a Linux host we can simply swipe the .config for the virtual machine's Ubuntu kernel like so:

$cp /boot/config-5.4.0-52-generic .config

We then need to select some options that make debugging and exploit dev a little easier. First thing we need is to merge some options for making the kernel easier to run in a virtual machine:

$make kvmconfig

Using .config as base
Merging ./kernel/configs/kvm_guest.config
#
# merged configuration written to .config (needs make)
#

...

Great, now we need to enable some options for debug symbols, kaslr and other awesome things. So open the .config somewhere in a text editor and make sure you either add or modify the file so these options are set:

CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y
# CONFIG_RANDOMIZE_BASE is not set

Cool now we need to make sure the config is ready to go for a build:

$make savedefconfig

$make -j4

...

Now you should grab some coffee, play a startcraft2 game because this may take a while. Okay so if your build worked you should have an object file in the following location:

[kernel_dir]/arch/x86_64/boot/bzImage

Build an image

We're going to build an image for this kernel so we might as well plop a "image" directory in this folder:

$mkdir [kernel_dir]/image/

Once you're kernel is build we need to start thinking about how to build a file system for this. Here I'm going to cheat and steal some tips from the syzkaller folks. We need to first download syzkaller, as follows:

$git clone https://github.com/google/syzkaller.git

Cloning into 'syzkaller'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
...

Move back to the kernel build and setup an image:

$cd [kernel_dir]/image/

$cp [syzkaller_dir]/tools/create_image.sh .

Okay so we can now create an image, all we need to do is simply invoke create_image.sh:

$./create_image.sh

+ DIR=chroot
+ PREINSTALL_PKGS=openssh-server,curl,tar,gcc,libc6-dev,time,strace,sudo,less,psmisc,selinux-utils,policycoreutils,checkpolicy,selinux-policy-default,firmware-atheros,python,xrdp,g++,make,libtool,autoconf,nasm
+ '[' -z ']'
+ ADD_PACKAGE=make,sysbench,git,vim,tmux,usbutils,tcpdump

...

If that worked you should have the following in your folder:

$ls

chroot/

create-image.sh

stretch.id_rsa

stretch.id_rsa.pub

stretch.img

Launch the virtual machine

Now we can launch qemu with all the goodies in place:

qemu-system-x86_64 \
-kernel ../arch/boot/x86_64/bzImage \
-append "console=ttyS0 root=/dev/sda earlyprintk=serial nokaslr"\
-hda ./stretch.img \
-net user,hostfwd=tcp::10021-:22 -net nic \
-enable-kvm \
-nographic \
-m 2G \
-s \
-S \
-smp 2 \
-pidfile vm.pid \
2>&1 | tee vm.log

...

The -s is a shorthand for -gdb tcp::1234, which means the gdbserver will be hosted at port 1234. -S tells qemu not to start the cpu automatically, this gives us a chance to set a breakpoint before the kernel starts executing.

So that's the image running smoothly, lets setup our debugging environment.

Attach and setup the debugger

We can then attach a gdb debugger to the qemu instance as follows. On another terminal, separate from the one running your qemu instance, start up gdb and issue the following commands:

$cd [kernel_dir]/image/

$gdb ../vmlinux

Reading symbols from ../vmlinux...

(gdb) target remote :1234

Remote debugging using :1234
0x000000000000fff0 in exception_stacks ()

(gdb) c

We give the "c" command to continue execution. We can now set some of our own breakpoints. As part of the tutorial I've included a custom IOCTL driver and app code (code that invokes the ioctl from userspace), i thought this would be nifty since it shows full ability to develope and debug a driver, something crucial to hunting down modern bugs and exploit development. Anyway lets code and build our own module.

Building, Loading and debugging a test module

Okay so we need to make a test ioctl driver, so lets head over the to kernel source directory and make a new folder in the /driver/ subfolder:

$cd [kernel_dir]/drivers/

$mkdir debug_driver/

$cd debug_driver/

$touch debug_driver.c

$touch debug_driver_app.c

$touch Makefile

The code for debug_driver.c and debug_driver_app.c as we well as the Makefile are available at this repo https://gitlab.com/k3170makan/linux-kernel-exploit-development. All you need to do is download the repo and stick this in its own folder under [kernel_dir]/drivers/. To build the module the we need to set the "M" variable in the kernel make script:

$cd [kernel_dir]; make -C . M=drivers/debug_driver/

make: Entering directory '/home/kh3m/Research/Kernel/debug_image/linux-5.5.3'
AR drivers/debug_driver//built-in.a
CC [M] drivers/debug_driver//debug_driver.o

...

Now we need to get this module on our qemu host somehow, I do this the hard way, I'm sure there's all sorts of nifty ways to scp files onto the qemu host but I actually just re-create the image after copying the drivers to a folder to be baked into the start up filesystem. First we need to edit create-image.sh so it includes everything in a folder we specify, that way we can just dump stuff in the folder and run create-image.sh whenever we want those files on a live instance.

So before create-image.sh builds the disk image on line 129, stick this in there:

++ sudo cp -r ./add/* $DIR/home/.

now we make a "add" folder and stick the kernel module and app code in there:

$ cd [kernel_dir]/image/

$ mkdir add/

$ cd add/

$ cp ../../drivers/debug_driver/debug_driver.ko .

$ cp ../../drivers/debug_driver/debug_driver_app.c .

$ ./create-image.sh

Okay so we have a module, we have a symbol file debug_driver.ko, with stuff we need to set breakpoints. Lets load the module into the kernel, then check where it gets loaded before we actually set the breakpoint:

root@syzkaller:$ cd /home/

root@syzkaller:$ insmod debug_driver.ko

[   32.792570] audit: type=1400 audit(1605058227.605:7): avc: denied { module_load } for pid=249 comm="insmod" path="/home/debug_driver.ko" dev="sda" ino=21253 scontext=system_u:system_r:kernel_t:s0 1
[   32.793766] debug_driver: loading out-of-tree module taints kernel.
[   32.800394] [debug_driver] loaded!
[   32.800826] [debug_driver] device registered successfully
[   32.802298] [debug_driver] device has been successfully created

Before we can debug it properly we need to know where it is loaded in kernel memory:

root@syzkaller:/home# cat /proc/modules
debug_driver 16384 0 - Live 0xffffffffa0000000 (O)

Okay lets now set our breakpoint and load the symbol file using the base address of the module:

(gdb) add-symbol-file ../drivers/debug_driver/debug_driver.ko 0xffffffffa0000000
add symbol table from file "../drivers/debug_driver/debug_driver.ko" at
.text_addr = 0xffffffffa0000000
(y or n) y
Reading symbols from ../drivers/debug_driver/debug_driver.ko...
(gdb) break dev_read
Breakpoint 1 at 0xffffffffa0000010: file drivers/debug_driver//debug_driver.c, line 81.
(gdb) c

Cool lets execute the driver program so we can trigger the code we want:

root@syzkaller:$ gcc -o debug_driver_app.elf debug_driver_app.c

root@syzkaller:/home# ./debug_driver_app.elf
Usage: ./debug_driver_app.elf [message to write] [read length]

root@syzkaller:$ ./debug_driver_app.elf "hello" 10

[ 160.083320] [debug_driver] message successfully copied message => [hello]
[ 160.083326] [debug_driver] buffer copied to message holder
[debug_driver] r[ 160.086175] [debug_driver] device released

This should trigger the dev_read function; and as we can see in the attached debugger:

Thread 2 hit Breakpoint 1, dev_read (filep=0xffff888067c29dc0, buffer=0xffff888067c29dc0 "",
len=16, offset=0xffffc900002c7eb8) at drivers/debug_driver//debug_driver.c:81
81 error_count = copy_to_user(buffer,message,len); //copy out of message into buffer

So thats the breakpoint hit! We achived our goal for this post, if you'd like to explore more try setting more breakpoints and before moving on to the next post make sure to get your gdb foo up. Next post is going to look at exploitation of stack vulnerabilities.

k3170

Search This Blog