Shweta, back from hospital, was relaxing in the library, reading various books. Ever since she learned of the
ioctl way of debugging, she was impatient to find out more about debugging in kernel-space. She was curious about how and where to run the kernel-space debugger, if there was any. This was in contrast with application/user-space debugging, where we have the OS running underneath, and a shell or a GUI over it to run the debugger (like
gdb, and the data display debugger,
ddd). Then she came across this interesting kernel-space debugging mechanism using
kgdb, provided as part of the kernel itself, since kernel 2.6.26.
The debugger challenge in kernel-space
As we need some interface to be up to run a debugger to debug anything, a kernel debugger could be visualised in two possible ways:
- Put the debugger into the kernel itself, accessible via the usual console. For example, in the case of
kdb, which was not official until kernel 2.6.35, one had to download source code (two sets of patches — one architecture-dependent, one architecture-independent) from this FTP address and then patch these into the kernel source. However, since kernel 2.6.35, the majority of it is in the officially released kernel source. In either case,
kdbsupport needs to be enabled in kernel source, with the kernel compiled, installed and booted with. The boot screen itself would give the
- Put a minimal debugging server into the kernel; a client would connect to it from a remote host or local user-space over some interface (say serial or Ethernet). This is
kgdb, the kernel’s
gdbserver, to be used with
gdbas its client. Since kernel 2.6.26, its serial interface is part of the official kernel release. However, if you’re interested in a network interface, you still need to patch with one of the releases from the kgdb project page. In either case, you need to enable
kgdbsupport in the kernel, recompile, install and boot the new kernel.
Please note that in both the above cases, the complete kernel source for the kernel to be debugged is needed, unlike for building modules, where just headers are sufficient. Here is how to play around with
kgdb over the serial interface.
Setting up the Linux kernel with kgdb
Here are the prerequisites: Either the kernel source package for the running kernel should be installed on your system, or a corresponding kernel source release should have been downloaded from kernel.org.
First of all, the kernel to be debugged needs to have
kgdb enabled and built into it. To achieve that, the kernel source has to be configured with
CONFIG_KGDB=y. Additionally, for
kgdb over serial,
CONFIG_KGDB_SERIAL_CONSOLE=y needs to be configured. And
CONFIG_DEBUG_INFO is preferred for symbolic data to be built into the kernel, to make debugging with
gdb more meaningful.
CONFIG_FRAME_POINTER=y enables frame pointers in the kernel, allowing
gdb to construct more accurate stack back-traces. All these options are available under “Kernel hacking” in the menu obtained in the kernel source directory (preferably as root, or using
sudo), by issuing the following command:
$ make mrproper # To clean up properly $ make oldconfig # Configure the kernel same as the current running one $ make menuconfig # Start the ncurses based menu for further configuration
See the highlighted selections in Figure 1, for how and where these options would be:
- “KGDB: kernel debugging with remote gdb” –>
- “KGDB: use kgdb over the serial console” –>
- “Compile the kernel with debug info” –>
- “Compile the kernel with frame pointers” –>
Once configuration is saved, build the kernel (run
make), and then a
make install to install it, along with adding an entry for the installed kernel in the GRUB configuration file. Depending on the distribution, the GRUB configuration file may be
/etc/grub.cfg, or something similar. Once installed, the
kgdb-related kernel boot parameters need to be added to this new entry, as shown in the highlighted text in Figure 2.
kgdboc is for
gdb connecting over the console, and the basic format is
kgdboc= <serial_device>, <baud-rate> where:
<serial_device>is the serial device file (port) on the system running the kernel to be debugged
<baud-rate>is the baud rate of this serial port
kgdbwait tells the kernel to delay booting till a
gdb client connects to it; this parameter should be given only after
With this, we’re ready to begin. Make a copy of the
vmlinux kernel image for use on the
gdb client system. Reboot, and at the GRUB menu, choose the new kernel, and then it will wait for
gdb to connect over the serial port.
All the above snapshots are with kernel version 220.127.116.11. The same should work for any 2.6.3x release of the kernel source. Also, the snapshots for
kgdb are captured over the serial device file
/dev/ttyS0, i.e., the first serial port.
Setting up gdb on another system
Following are the prerequisites:
- Serial ports of the system to be debugged, and the other system to run
gdb, should be connected using a null modem (i.e., a cross-over serial) cable.
- The vmlinux kernel image built, with
kgdbenabled, needs to be copied from the system to be debugged, into the working directory on the system where
gdbis going to be run.
gdb to connect to the waiting kernel, launch
gdb from the shell and run these commands:
(gdb) file vmlinux (gdb) set remote interrupt-sequence Ctrl-C (gdb) set remotebaud 115200 (gdb) target remote /dev/ttyS0 (gdb) continue
In the above commands,
vmlinux is the kernel image copied from the system to be debugged.
Debugging using gdb with kgdb
After this, it is all like debugging an application from
gdb. One may stop execution using
Ctrl+C, add break points using
b[reak], stop execution using
n[ext] … — the usual
gdb way. There are enough GDB tutorials available online, if you need them. In fact, if you are not comfortable with text-based GDB, use any of the standard GUI tools over
ddd, Eclipse, etc.
By now, Shweta was excited about wanting to try out
kgdb. Since she needed two systems to try it out, she went to the Linux device drivers’ lab. There, she set up the systems and ran
gdb as described above.