Kernel Debugging Using Kprobe and Jprobe

First, we must pay homage to the best-known kernel-debugging technique/tool, the printk. It is a universal tool, customisable enough to provide you the desired output — and usually when all else fails, this can save the day! It’s very easy to use; it simply prints whatever you like, to track the execution of a module by printing its status after each step, and thus pin-pointing bugs and defects.

Every good thing comes at a price, however — and printk‘s price is its static nature, which may become a bit of a problem when frequent changes are required in the information needed to be printed while debugging. Also, in order to implement printk, you need to recompile the kernel with printk statements added to the specific module you’re debugging. This is usually a time-consuming process of building, installing and rebooting the kernel.

These problems can be resolved by using the dynamic tools Kprobe and Jprobe. The great thing about Linux is that kernel 2.6.x already contains Kprobe; we don’t need installation or patching to try it. So it’s time for some hands-on fun!

For the record, my system has an AMD triple-core, and runs Fedora 13 with the 2.6.34 vanilla kernel. If you don’t have one, you can follow these steps to install a fresh vanilla kernel on your system:

• Download the kernel source from kernel.org.
• Extract the tar ball with tar -jxvf linux-2.6.34.tar.bz2.
• Go to the extracted directory and run the following commands:

  # make menuconfig
  # make
  # make modules_install
  # make install

Finally, reboot your system, and choose the newly compiled kernel on the bootloader screen.

About Kprobe

Kprobe is a very simple method to probe the running kernel. At a fundamental level, it requires the address of a kernel function that needs to be debugged. Then, you create pre- and post-handlers that will print a debugging message when the target kernel function is called. (Actually, a handler performs any action specified in its code; in this case, it happens to be printing.) Thus, every time that function is called, you can track it.

An example

To keep things simple, I have created a small and easy-to-understand example. The target kernel function is ip_rcv(). The Kprobe example kernel module is as follows:

#include<linux/module.h> 
#include<linux/version.h> 
#include<linux/kernel.h> 
#include<linux/init.h> 
#include<linux/kprobes.h> 

static unsigned int counter = 0;
int Pre_Handler(struct kprobe *p, struct pt_regs *regs){ 
    printk("Pre_Handler: counter=%u\n",counter++);
    return 0; 
} 

void Post_Handler(struct kprobe *p, struct pt_regs *regs, unsigned long flags){ 
    printk("Post_Handler: counter=%u\n",counter++);
} 

static struct kprobe kp; 

int myinit(void) 
{ 
    printk("module inserted\n "); 
    kp.pre_handler = Pre_Handler; 
    kp.post_handler = Post_Handler; 
    kp.addr = (kprobe_opcode_t *)0xc071c9a9; 
    register_kprobe(&kp); 
    return 0; 
} 

void myexit(void) 
{ 
    unregister_kprobe(&kp); 
    printk("module removed\n "); 
} 

module_init(myinit); 
module_exit(myexit); 
MODULE_AUTHOR("Manoj"); 
MODULE_DESCRIPTION("KPROBE MODULE"); 
MODULE_LICENSE("GPL");

The makefile required to build the kernel module object file that you need to insert into the kernel is as follows:

obj-m +=mod1.o mod2.o 
KDIR= /lib/modules/$(shell uname -r)/build 
all: 
    $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules 
clean: 
       rm -rf *.o *.ko *.mod.* .c* .t*

Code walk-through

Here’s an explanation for the less obvious sections of the code.

struct kprobe kp;

To make use of Kprobe functionality, you must declare a variable of the structure struct kprobe, which is declared in include/linux/kprobes.h. Here’s a little extract:

struct kprobe {
    .
    .
    kprobe_opcode_t *addr; 
    kprobe_pre_handler_t pre_handler; 
    kprobe_post_handler_t post_handler; 
}

The three members listed above are of interest to us. You need to assign the kernel address of the target function to the addr member; you can retrieve the address from the /proc/kallsyms file, as follows:

# cat /proc/kallsyms | grep ip_rcv
c071c3e0 t ip_rcv_finish
c071c9a9 T ip_rcv

Once you’ve found the address, use it in the myinit() function, as follows:

kp.addr = (kprobe_opcode_t *)0xc071c9a9;

Kprobe executes handler functions before and after the target kernel function is called, and we created the Pre_Handler() and Post_Handler() functions for this purpose. Assign these to their respective pointer members in the Kprobe struct — pre_handler and post_handler — in myinit(), as you can see. Finally, register your Kprobe with the kernel, with register_kprobe(&kp);.

Then compile the module by running make:

# make
make -C /lib/modules/2.6.34/build SUBDIRS=/root/kprobe modules
make[1]: Entering directory '/root/linux-2.6.34'
  CC [M]  /root/kprobe/mod1.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /root/kprobe/mod1.mod.o
  LD [M]  /root/kprobe/mod1.ko
make[1]: Leaving directory '/root/linux-2.6.34'

When done, you are ready to test your example module by inserting it into the kernel:

# insmod mod1.ko

Confirm that the module is successfully inserted:

# lsmod | head -n 5 
Module                  Size  Used by 
mod1                     904  0 
fuse                   46627  2 
sunrpc                158985  1 
xt_physdev              1355  1

Now, since you have used ip_rcv() as your target function, you need to invoke it with a simple ping:

# ping localhost

Run dmesg and find your module’s messages:

module inserted
Pre_Handler: counter=0
Post_Handler: counter=1
Pre_Handler: counter=2
Post_Handler: counter=3

As you see, you can probe a kernel address and do instrumentation without recompiling the kernel, as was required by the simple printk. When you are done with your debugging, don’t forget to remove the module:

# rmmod mod1

In the exit function, myexit(), Kprobe is unregistered by calling unregister_kprobe(&kp);.

However, Kprobe has limits to what you can do with it. In the above example, you have just printed some messages in the handlers; you cannot access the function’s arguments with Kprobe. Let’s move on to something better.

Probing with Jprobe

For those who like bonus features, Jprobe is another kind of probing technique, which can be used to access the target function’s arguments, and thus display what was passed to the function. The basics are the same as that of Kprobe, but this additional feature makes Jprobe an interesting tool.

To get the Jprobe structure details, look in the file include/linux/kprobes.h:

struct jprobe {
        struct kprobe kp; 
        void *entry;    /* probe handling code to jump to */ 
};

As you see, it contains a struct kprobe member, plus a pointer to store the address of a handler function to jump to.

A Jprobe example

#include<linux/module.h> 
#include<linux/version.h> 
#include<linux/kernel.h> 
#include<linux/init.h> 
#include<linux/kprobes.h> 
#include<net/ip.h> 

int my_handler (struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev){ 

    struct iphdr *my_iph; 
    u32 S_ip,D_ip; 
    my_iph = ip_hdr(skb); 
    S_ip = my_iph->saddr; 
    D_ip = my_iph->daddr; 
    printk("Source IP: \n"NIPQUAD_FMT,NIPQUAD(S_ip)); 
     jprobe_return(); 
} 

static struct jprobe my_probe; 

int myinit(void) 
{ 
    my_probe.kp.addr = (kprobe_opcode_t *)0xc071c9a9; 
    my_probe.entry = (kprobe_opcode_t *)my_handler; 
    register_jprobe(&my_probe); 
    return 0; 
} 

void myexit(void) 
{ 
    unregister_jprobe(&my_probe); 
    printk("module removed\n "); 
} 

module_init(myinit); 
module_exit(myexit); 

/*Kernel module Comments*/ 
MODULE_AUTHOR("Manoj"); 
MODULE_DESCRIPTION("SIMPLE MODULE"); 
MODULE_LICENSE("GPL"); 
//MODULE_LICENSE("GPL v2");

Code walk-through

The example is simple to understand, but let me explain things a bit. Here, in the myinit() function, you assigned the target function address to the addr member of the Kprobe member struct kp, just like for the earlier module. The main difference is that you’ve now assigned a single handler function, my_handler, to the entry member:

my_probe.entry = (kprobe_opcode_t *)my_handler;

You’ve probably already noted that the signature of the single handler function here is quite different from the Kprobe handlers. The reason is, the handler must have the same arguments as that of the kernel function you’re probing, which is once again ip_rcv():

int my_handler (struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev);
extern int   ip_rcv(struct sk_buff *skb, struct net_device *dev,  struct packet_type *pt, struct net_device *orig_dev);

Jprobe lets us access the arguments of a function by calling your handler with the same arguments passed to the target function. This means that when ip_rcv is called, its arguments can be accessed from your probe handler as it is able to refer to the function’s address space plus the components within that function stack.

The line my_iph = ip_hdr(skb); will extract the IP header from sk_buff. Then extract the source and destination IP addresses in dot notation form, using the NIPQUAD and NIPQUAD_FMT macros declared in include/linux/kernel.h, and print the addresses.

Now, compile your module, insert it, and check that the module has been inserted successfully, just as before. Again, to invoke ip_rcv(), run a ping and then run dmesg to check the output:

# ping www.google.com
# dmesg
Source IP: 192.168.1.1
Destination IP: 192.168.1.3
Source IP: 209.85.231.104
Destination IP: 192.168.1.3

The output shows that Jprobe lets you get the function’s argument values, which can be very handy when debugging data-dependent bugs.