A short note on setting up a Linux kernel debugging environment

This is a relatively short post on how to set up a Linux environment for kernel debugging using Buildroot.

This note overlaps a lot with this blog post by Nick Desaulniers, and another post from Guanzhou Hu. Relative to both of them I’ve added a little more information around Buildroot configuration, but they are nonetheless good resources.

Setup

To start off, you’ll want to clone the Buildroot sources:

git clone https://git.buildroot.net/buildroot

You’ll need to make an initial .config file for Buildroot. For the rest of this note, I’m assuming that we’ll want to build our kernel as x86-64 and run it under QEMU, so we’ll use the following command:

make qemu_x86_64_defconfig

Now start the Buildroot TUI with

make menuconfig

If you’ve built the Linux kernel from source before, the Buildroot UI should feel familiar. Instead of being presented with kernel build parameters to tweak, you’ll have various options for how you want to configure the operating system, such as

what version of the Linux kernel you want to use;
what packages should be installed by default;
whether login with the root user is permitted (and the root user’s default password);

and so on.

Once you’ve finished selecting the options that you want for Buildroot, you should save your config file (by default, to .config) and exit. For the purposes of this note, I used the following configuration options:

# Some packages require C++ support in order to be installed
# -> Toolchain -> Enable C++ support
BR2_TOOLCHAIN_BUILDROOT_CXX=y

# I like having OpenSSH installed by default as another means of getting access
# to the environment, but it isn't strictly necessary.
# -> Target packages -> Networking applications -> openssh
BR2_PACKAGE_OPENSSH=y
BR2_PACKAGE_OPENSSH_CLIENT=y
BR2_PACKAGE_OPENSSH_SERVER=y
BR2_PACKAGE_OPENSSH_KEY_UTILS=y
BR2_PACKAGE_OPENSSH_SANDBOX=y

# Add a post-build script for some minor SSH configuration
# -> System configuration -> Custom scripts to run before creating filesystem images
BR2_ROOTFS_POST_BUILD_SCRIPT="board/qemu/x86_64/post-build.sh ./scripts/config.sh"

# Use Ext4 for the filesystem image
# -> Filesystem images -> ext2/3/4 root filesystem
BR2_TARGET_ROOTFS_EXT2_4=y
BR2_TARGET_ROOTFS_EXT2_GEN=4

The ./scripts/config.sh shell script just contains the following:

#!/bin/sh

PUBKEY="..."  # Replace with your public key
OUTDIR=output/target

addline() {
  grep -qxF "$1" "$2" || echo "$1" >> "$2"
}

mkdir -p ${OUTDIR}/root/.ssh
addline "PermitRootLogin    yes"    ${OUTDIR}/etc/ssh/sshd_config
addline "${PUBKEY}"                 ${OUTDIR}/root/.ssh/authorized_keys
chmod 600 ${OUTDIR}/root/.ssh/authorized_keys

Once you’ve set up your configuration, you can start building the Linux environment with

make -j$(nproc)

Using a custom kernel

If you just want to build the kernel with custom configuration parameters, you can run

make linux-menuconfig

to create a kernel build config file. Then you can add the following to your Buildroot config:

BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=./path/to/config/file

If you’re experimenting with some changes to the kernel sources, however, you’ll need to do one of the following:

Configure some custom kernel patches via BR2_GLOBAL_PATCH_DIR or BR2_LINUX_KERNEL_PATCH.
Point to a tarball with the modified kernel source using

BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION="file:///path/to/sources.tar.gz"

Booting the kernel with QEMU

If everything has gone smoothly up to this point, your newly-built Linux system should now be in the output/images directory (relative to the root of the Buildroot repository). You can now run your environment under QEMU with (for instance)

qemu-system-x86_64 \
    -kernel ./output/images/bzImage \
    -drive file=./output/images/rootfs.ext4,if=virtio,format=raw \
    -nographic \
    -enable-kvm \
    -cpu host \
    -m 512M \
    -append "root=/dev/vda console=tty1 console=ttyS0"

You should see your system booting and reach the Buildroot login prompt (by default the only user you can login as is root, without any password):

Welcome to Buildroot
buildroot login: root
#

Press Ctrl-a x to exit.

If you installed OpenSSH, you should also add

-net nic,model=virtio -net user,hostfwd=tcp::2222-:22

to your command. This will forward port 22 from the virtual machine to port 2222 on your host, making the VM accessible over SSH.

Debugging the kernel with GDB

Now that we have our VM running, we want to start using GDB to debug it. First, we’ll need to build the kernel to support debugging using a custom build config. You should run make linux-menuconfig and set the following options:

Kernel hacking -> Kernel debugging
Kernel hacking -> Compile-time checks and compiler options -> Debug information
Kernel hacking -> Compile-time checks and compiler options -> Provide GDB scripts for kernel debugging

Alternatively, put the following in your kernel build config:

CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_GDB_SCRIPTS=y

Run make again to build your environment. When we run QEMU now, we’ll add the following to our command:

-gdb tcp::1234 will open up a gdbserver on TCP port 1234.
The -S flag will tell the kernel to wait before booting. This will allow us to attach GDB to the kernel before the boot process starts.
We’ll add nokaslr to the kernel command line parameters to disable address space layout randomization in the kernel.

Your full command should now look something like this:

qemu-system-x86_64 \
    -kernel ./output/images/bzImage \
    -drive file=./output/images/rootfs.ext4,if=virtio,format=raw \
    -nographic \
    -append "nokaslr root=/dev/vda console=tty1 console=ttyS0" \
    -enable-kvm \
    -cpu host \
    -m 512M \
    -gdb tcp::1234 \
    -S

Before starting GDB, you’ll want to ensure that you’ll be able to load the vmlinux-gdb.py script by adding it to your auto-load safe-path:

echo "add-auto-load-safe-path $(realpath output/images/linux/vmlinux-gdb.py)" \
    >> ~/.gdbinit

Then run

gdb -x ./kerneldebug.gdb ./output/images/linux/vmlinux

where kerneldebug.gdb contains the following:

# Attach to QEMU
target remote :1234

# Add any breakpoints you already know you want to set here, e.g.
#
#   hbreak function_1
#   hbreak function_2
#   ...

hbreak start_kernel

If all went well, you should be able to run

(gdb) c

in GDB, and then see Linux boot up in the terminal where you ran QEMU.

Additional references

As mentioned earlier, Nick Desaulnier’s post and Guanzhou Hu’s post were both helpful references for me.

The Buildroot user manual is the most complete source of information on how to use Buildroot.

The kernel sources have some documentation on how to debug the kernel with GDB.