How to Cross Compile the Linux Kernel with Device Tree Support

You may have heard of the many embedded target boards available today, like the BeagleBoard, Raspberry Pi, BeagleBone, PandaBoard, Cubieboard, Wandboard, etc. But once you decide to start development for them, the right hardware with all the peripherals may not be available. The solution to starting development on embedded Linux for ARM is by emulating hardware with QEMU, which can be done easily without the need for any hardware. There are no risks involved, too.
QEMU is an open source emulator that can emulate the execution of a whole machine with a full-fledged OS running. QEMU supports various architectures, CPUs and target boards. To start with, let’s emulate the Versatile Express Board as a reference, since it is simple and well supported by recent kernel versions. This board comes with the Cortex-A9 (ARMv7) based CPU.
In this article, I would like to mention the process of cross compiling the Linux kernel for ARM architecture with device tree support. It is focused on covering the entire process of working—from boot loader to file system with SD card support. As this process is almost similar to working with most target boards, you can apply these techniques on other boards too.

Device tree
Flattened Device Tree (FDT) is a data structure that describes hardware initiatives from open firmware. The device tree perspective kernel no longer contains the hardware description, which is located in a separate binary called the device tree blob (dtb) file. So, one compiled kernel can support various hardware configurations within a wider architecture family. For example, the same kernel built for the OMAP family can work with various targets like the BeagleBoard, BeagleBone, PandaBoard, etc, with dtb files. The boot loader should be customised to support this as two binaries-kernel image and the dtb file – are to be loaded in memory. The boot loader passes hardware descriptions to the kernel in the form of dtb files. Recent kernel versions come with a built-in device tree compiler, which can generate all dtb files related to the selected architecture family from device tree source (dts) files. Using the device tree for ARM has become mandatory for all new SOCs, with support from recent kernel versions.

Building QEMU from sources
You may obtain pre-built QEMU binaries from your distro repositories or build QEMU from sources, as follows. Download the recent stable version of QEMU, say qemu-2.0.tar.bz2, extract and build it:

tar -zxvf qemu-2.0.tar.bz2
cd qemu-2.0
./configure --target-list=arm-softmmu, arm-linux-user --prefix=/opt/qemu-arm
make
make install

You will observe commands like qemu-arm, qemu-system-arm, qemu-img under /opt/qemu-arm/bin.
Among these, qemu-system-arm is useful to emulate the whole system with OS support.

Preparing an image for the SD card
QEMU can emulate an image file as storage media in the form of the SD card, flash memory, hard disk or CD drive. Let’s create an image file using qemu-img in raw format and create a FAT file system in that, as follows. This image file acts like a physical SD card for the actual target board:

qemu-img create -f raw sdcard.img 128M
#optionally you may create partition table in this image #using tools like sfdisk, parted 
mkfs.vfat sdcard.img
#mount this image under some directory and copy required files
mkdir /mnt/sdcard 
mount -o loop,rw,sync sdcard.img /mnt/sdcard

Setting up the toolchain
We need a toolchain, which is a collection of various cross development tools to build components for the target platform. Getting a toolchain for your Linux kernel is always tricky, so until you are comfortable with the process please use tested versions only. I have tested with pre-built toolchains from the Linaro organisation, which can be got from the following link http://releases.linaro.org/14.0.4/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux.tar.xz or any latest stable version. Next, set the path for cross tools under this toolchain, as follows:

tar -xvf gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux.tar.xz -C /opt
export PATH=/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin:$PATH

You will notice various tools like gcc, ld, etc, under /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin with the prefix arm-linux-gnueabihf-

Building mkimage
The mkimage command is used to create images for use with the u-boot boot loader.
Here, we’ll use this tool to transform the kernel image to be used with u-boot. Since this tool is available only through u-boot, we need to go for a quick build of this boot loader to generate mkimage. Download a recent stable version of u-boot (tested on u-boot-2014.04.tar.bz2) from ftp.denx.de/pub/u-boot:

tar -jxvf u-boot-2014.04.tar.bz2
cd u-boot-2014.04
make tools-only

Now, copy mkimage from the tools directory to any directory under the standard path (like /usr/local/bin) as a super user, or set the path to the tools directory each time, before the kernel build.

Building the Linux kernel
Download the most recent stable version of the kernel source from kernel.org (tested with linux-3.14.10.tar.xz):

tar -xvf linux-3.14.10.tar.gz
cd linux-3.14.10 
make mrproper #clean all built files and configuration files
make ARCH=arm vexpress_defconfig #default configuration for given board
make ARCH=arm menuconfig #customize the configuration

Then, to customise kernel configuration (Figure 1), follow the steps listed below:

1) Set a personalised string, say ‘-osfy-fdt’, as the local version of the kernel under general setup.
2) Ensure that ARM EABI and old ABI compatibility are enabled under kernel features.
3) Under device drivers–> block devices, enable RAM disk support for initrd usage as static module, and increase default size to 65536 (64MB).
You can use arrow keys to navigate between various options and space bar to select among various states (blank, m or *)
4) Make sure devtmpfs is enabled under the Device Drivers and Generic Driver options.
Now, let’s go ahead with building the kernel, as follows:

#generate kernel image as zImage and necessary dtb files
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage dtbs
#transform zImage to use with u-boot 
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- uImage \ LOADADDR=0x60008000 
#copy necessary files to sdcard
cp arch/arm/boot/zImage /mnt/sdcard
cp arch/arm/boot/uImage /mnt/sdcard
cp arch/arm/boot/dts/*.dtb /mnt/sdcard
#Build dynamic modules and copy to suitable destination
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules_install \ INSTALL_MODPATH=<mount point of rootfs>

You may skip the last two steps for the moment, as the given configuration steps avoid dynamic modules. All the necessary modules are configured as static.

Getting rootfs
We require a file system to work with the kernel we’ve built. Download the pre-built rootfs image to test with QEMU from the following link: http://downloads.yoctoproject.org/releases/yocto/yocto-1.5.2/machines/qemu/qemuarm/core-image-minimal-qemuarm.ext3 and copy it to the SD card (/mnt/image) by renaming it as rootfs.img for easy usage. You may obtain the rootfs image from some other repository or build it from sources using Busybox.
Your first try
Let’s boot this kernel image (zImage) directly without u-boot, as follows:

export PATH=/opt/qemu-arm/bin:$PATH
qemu-system-arm -M vexpress-a9 -m 1024 -serial stdio \
-kernel /mnt/sdcard/zImage \
-dtb /mnt/sdcard/vexpress-v2p-ca9.dtb \
-initrd /mnt/sdcard/rootfs.img -append “root=/dev/ram0 console=ttyAMA0”

In the above command, we are treating rootfs as ‘initrd image’, which is fine when rootfs is of a small size. You can connect larger file systems in the form of a hard disk or SD card. Let’s try out rootfs through an SD card:

qemu-system-arm -M vexpress-a9 -m 1024 -serial stdio \
-kernel /mnt/sdcard/zImage \
-dtb /mnt/sdcard/vexpress-v2p-ca9.dtb \
-sd /mnt/sdcard/rootfs.img -append “root=/dev/mmcblk0 console=ttyAMA0”

In case the sdcard/image file holds a valid partition table, we need to refer to the individual partitions like /dev/mmcblk0p1, /dev/mmcblk0p2, etc. Since the current image file is not partitioned, we can refer to it by the device file name /dev/mmcblk0.
Building u-boot
Switch back to the u-boot directory (u-boot-2014.04), build u-boot as follows and copy it to the SD card:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- vexpress_ca9x4_config 
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- 
cp u-boot /mnt/image
# you can go for a quick test of generated u-boot as follows
qemu-system-arm -M vexpress-a9 -kernel /mnt/sdcard/u-boot -serial stdio

Let’s ignore errors such as ‘u-boot couldn’t locate kernel image’ or any other suitable files.

The final steps
Let’s boot the system with u-boot using an image file such as SD card, and make sure the QEMU PATH is not disturbed.
Unmount the SD card image and then boot using QEMU.

umount /mnt/sdcard
qemu-system-arm -M vexpress-a9 -sd sdcard.img -m 1024 -serial stdio -kernel u-boot

You can stop autoboot by hitting any key within the time limit and enter the following commands at the u-boot prompt to load rootfs.img, uimage, dtb files from the SD card to suitable memory locations without overlapping. Also, set the kernel boot parameters using setenv as shown below (here, 0x82000000 stands for the location of the loaded rootfs image and 8388608 is the size of the rootfs image).
Note: The following commands are internal to u-boot and must be entered within the u-boot prompt.

fatls mmc 0:0 #list out partition contents
fatload mmc 0:0 0x82000000 rootfs.img # note down the size of image being loaded
fatload mmc 0:0 0x80200000 uImage
fatload mmc 0:0 0x80100000 vexpress-v2p-ca9.dtb
setenv bootargs 'console=ttyAMA0 root=/dev/ram0 rw initrd=0x82000000,8388608'
bootm 0x80200000 - 0x80100000

Ensure a space before and after the ‘–’‘–’ symbol in the above command.

Log in using ‘root’ as the username and a blank password to play around with the system.
I hope this article proves useful for bootstrapping with embedded Linux and for teaching the concepts when there is no hardware available.

Acknowledgements
I thank Babu Krishnamurthy, a freelance trainer for his valuable inputs on embedded Linux and omap hardware during the course of my embedded journey. I am also grateful to C-DAC for the good support I’ve received.

References
[1] elinux.org/Qemu
[2] Device Tree for Dummies by Thomas Petazzoni (free-electrons.com)
[3] Few inputs taken from en.wikipedia.org/wiki/Device_tree
[4] mkimage man page from u-boot documentation