Remote Debugging the running OpenBSD kernel

A way to understand the OpenBSD internals

The Problem

A few month ago, I tried porting the FreeBSD kdb along with it's gdb stub implementations to OpenBSD as a practice of learning the internals of an BSD operating system. The ddb code in both FreeBSD and OpenBSD looks pretty much the same and the GDB Remote Serial Protocol looks very minimal.

But sadly I got very busy and the work is stalled but I'm planning on resuming the attempt as soon as I get the chance, But there is an alternative way to Debugging the OpenBSD kernel via QEMU. What I did below is basically the same with a few minor changes which I hope to describe it as best.

Installing OpenBSD on Qemu

For debugging the kernel, we need a working OpenBSD system running on Qemu. I choosed to create a raw disk file to be able to easily mount it later via the host and copy the custom kernel onto it.

$ qemu-img create -f raw disk.raw 5G
$ qemu-system-x86_64 -m 256M \
    -drive format=raw,file=install63.fs \
    -drive format=raw,file=disk.raw

Custom Kernel

To debug the kernel, we need a version of the kernel with debugging symbols and for that we have to recompile it first. The process is documented at Building the System from Sourc:

$ cd /usr/src
$ cd sys/arch/amd64/conf
$ config GENERIC
$ cd ../compile/GENERIC/
$ make -j 8
$ ls -la obj/bsd{,.gdb}
-rwxrwx---  1 root  bijan  13110110 Apr 14 10:34 obj/bsd
-rwxrwx---  1 root  bijan  55392459 Apr 14 10:34 obj/bsd.gdb

Then we can copy the bsd kernel to the guest machine and keep the bsd.gdb on the host to start the remote debugging via gdb.

$ vnconfig vnd0 disk.raw
$ disklabel vnd0
# /dev/rvnd0c:
type: ESDI
disk: ESDI/IDE disk
label: QEMU HARDDISK   
duid: 3e97835c8cfa04a4
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 652
total sectors: 10485760
boundstart: 64
boundend: 10474380
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  a:         10474304               64  4.2BSD   2048 16384 12958 
  c:         10485760                0  unused                    

$ mount /dev/vnd0a /mnt
$ cp obj/bsd /mnt/bsd
$ umount /mnt
$ vnconfig -u vnd0

Remote debugging kernel

Now it's to time to boot the guest with the new custom kernel. Remember that the -s argument enables the gdb server on qemu on localhost port 1234 by default:

$ qemu-system-x86_64 -m 256M -s \
    -net nic -net user \
    -drive format=raw,file=install63.fs \

Now to finally attach to the running kernel:

$ gdb obj/bsd.gdb
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd6.3"...
(gdb) target remote :1234
Remote debugging using :1234
[New Thread 1]
0xffffffff8123bcf5 in broxton_set_cdclk (dev=0x100, frequency=-2119613816)
    at /home/bijan/projects/github.com/openbsd/src/sys/dev/pci/drm/i915/intel_display.c:5569
5569                    val |= divider;
Current language:  auto; currently minimal
(gdb)