Saving Private GRUB: Remotely Restoring MBR on an Enterprise Server

Saving Private GRUB

Corporate hardware systems differ from their conventional desktop counterparts primarily because they have a number of interesting features. For example, they provide an ability to connect a virtual CD-ROM drive, and use a virtual monitor, just like a remote KVM. While this is not so important for desktop systems, it could be essential for large server farms.

For example, let’s imagine that you have accidentally damaged GRUB, and you’re far away from your office. What should you do — ask someone to enter the server room and recover it, or can you try and fix it remotely? The answer is that you can easily recover GRUB from your current location!

Let’s first understand the scenario, so that you know what I’m talking about. We have a couple of different projects hosted on a single branded (HP ProLiant 580) server whose configuration includes two RAID controllers, two physical Xeon processors and 24 GB of RAM — enough power for a dozen virtual machines. During the initial setup of Debian, the server was configured to boot from and use only the first RAID controller, while the second controller was configured as RAID 5 and left intact.

After a while, some more projects were added on the system, and it became clear that more space was needed; so it was decided to occupy the second RAID array as well. However, it was highly desirable to have it as RAID1+0 space; so a reconfiguration was done. However, after the first reboot, not a single ping was received from the server. What went wrong? Global loss of data, or something trickier?

My first response was to check whether GRUB was still good, or had been destroyed. If the latter, what would be my next step — a full Debian reinstallation, together with all hosted virtual machines, and associated user data? From a rational point of view, this would be a huge loss of time. Maybe I ought to try and restore GRUB first — but how? Fortunately, Hewlett-Packard hardware comes with the iLO service on-board.

iLO — integrated Lights-Out

For years, OEMs of complex computer systems have tried to expand the functionality of their enterprise-level product lines, for example, by utilising out-of-band management (a system feature using which working traffic goes through a casual network card, while service traffic comes via a dedicated channel). In fact, a server has an additional Ethernet controller that’s intended for service functions only.

For instance, through this dedicated channel, access to the server is organised — it can be seen as a secured and absolutely independent connection to a server. Don’t misuse it with a VLAN subnet, because it’s a completely separate network. Thus far, the traffic that goes through this dedicated channel is truly for management purposes. So, you can establish a virtual TTY connection to a UNIX server, or organise a fully featured graphical access to the server by means of iLO — the integrated Lights-Out service — and not be afraid of overloading other channels.

Basically, iLO is a special portion of firmware (i.e., BIOS) developed by Hewlett-Packard that transforms TCP/IP packets into serial line signals. There are competitors to iLO implemented by other vendors. In particular, Sun Microsystems had developed a feature pack called Advanced Lights Out Management (ALOM). Fujitsu has Integrated Remote Management Controller (iRMC), and Dell has Dell Remote Access Controller, or DRAC. All of them have one goal — to offer a virtual KVM.

The HP iLO implementation is available in two options: you can either connect to the server via SSH and perform all tasks in a text-console session, or you can launch your Web browser and have a full-featured graphical desktop environment in order to manipulate the hardware. In the latter case, you get a far more functional solution, because you can use the local screen and peripherals by your side as virtual devices (the real ones are left in the server room). We’ll be using only the advanced features of iLO.

Let’s get started with Mozilla Firefox, and locate the URL to the server’s iLO home page. You need to have a Web browser with a Java plugin to log in to iLO. After you’ve been authorised via the SSL connection, you will see the iLO main page (Figure 1).

iLO main page after login

Figure 1: iLO main page after login

Here, you will find the following tabs: System Status, Remote Console, Virtual Media, Power Management and Administration. Basically, all information about the HP server is available at a glance. However, our primary interest is hidden under the Remote Console and Virtual Media tabs.

From the first tab, you can access a graphical display of the server console, as seen in Figure 2. The Virtual Media tab allows you to connect floppy, USB stick or CD-ROM images as if they were real hardware on the server — see Figure 3. What’s interesting about all this is the fact that those images are on your side — on the same host on which you’re running the Web browser. Handy, isn’t it?

Virtual screen remotely displaying server console

Figure 2: Virtual screen remotely displaying server console

Virtual media is connected in a blink of an eye

Figure 3: Virtual media is connected in a blink of an eye

So we have access to the console as if we ourselves have a real keyboard and a screen. We can see the configuration of the RAID controller, and the server’s boot priority (the sequence of devices that the server checks in order to boot off — CD ROM, Ethernet card, or directly from HDD). In order to change it, you should press the F9 key right after the initialisation of both RAID controllers (they’re named HP Smart Array P400 Controller). After you press the F9 key, you’re in the Setup Utility (Figure 4).

HP ROM-based Setup Utility (virtual screen)

Figure 4: HP ROM-based Setup Utility (virtual screen)

Here, you need to pay attention to the option Standard Boot Order (IPL). Entering that option gives you a listing of devices, as shown in Figure 5.

Boot sequence, under “Standard Boot Order (IPL)”

Figure 5: Boot sequence, under “Standard Boot Order (IPL)”

The first device to be polled is CD-ROM, which would be a real CD inserted into the CD bay slot, or an image mounted via the network from your PC. If neither is available, the next device to be checked is a hard drive, seen as IPL:2, and so on. Now, you need to check what the boot sequence is for the composite IPL:2 device.

Go to the (main BIOS set-up screen) menu option Boot Controller Order. The screen is shown in Figure 6.

Boot order within RAID controllers

Figure 6: Boot order within RAID controllers

It is clear that the boot priority is as follows: first the PCI Embedded HP Smart Array P400 Controller (also known as Slot 0), then the PCI Embedded HP Integrated PCI IDE Controller, and finally the second RAID controller, in Slot 11.

In order to see the layout of the relevant RAID array, you need to do a warm reboot, and press F8 immediately after the initialisation of the respective controller. Figure 7 shows the main setup menu for the RAID controller in Slot 11.

Main menu for setting up the RAID controller's options

Figure 7: Main menu for setting up the RAID controller's options

Pay attention to the last option, Select as Boot Controller. This option is responsible for setting the boot priority among RAID disks. If you selected this option, then the loader will begin using this particular disk for its purposes. Therefore, even if you’ve correctly selected all of the previous options in the ROM-based Setup Utility, and forgotten about this tricky option, then expect your system to not be bootable. But don’t panic. We know how to restore the MBR, and we’ll do just that. Let’s save private GRUB!

Get my data back!

To rescue all user data as well as all the Linux systems configuration, we could grab ourselves any Live CD, boot off it, and copy the files to SAN storage. After that, we could reinstall the entire system from scratch.

However, I think this approach is wrong. Instead of copying several thousand gigabytes from RAID arrays via the local network, we can save our time. The first thing we can avoid is the need to integrate network drivers into the Live CD. This particular HP model has 2 Gigabit NICs, and once these drivers are put into the initramfs file, all we need is to find and use them. Otherwise, you’ll have to change the network topology — find a person who will switch cables from one Ethernet port to another. That’s not easy to explain to a stranger who is miles away from you, in the middle of the night, especially when the server room hosts dozens of communication racks and cabinets.

Let’s make sure that the drives on the first RAID controller (where we store the data and the system) are consistent. In order to do this, continuously press the F8 button right after the controller initialisation of Slot 0 to get into the RAID configuration.

Useful payload and systems data are obviously left intact

Figure 8: Useful payload and systems data are obviously left intact

According to the layout visible in Figure 8, the data we need has remained untouched on the disks. This is great; full speed ahead! Our task is to boot the system, then transfer the /boot/ contents onto our PC. Only after that will we create a bootable ISO image with the kernel and initramfs files rescued from the system. This ISO image will load the whole system as it was before the crash. Once the system is loaded fully, we will try to rerun the last command, grub-install. Easier said than done, right? Let’s see.

I used the kernel option init=/bin/sh, when I started a usual installation of Debian Lenny. This allows me to have a console in runlevel 1, after all kernel initialisation and system configuration takes place. Let’s see exactly what virtual devices are connected to this HP server:

# dmesg | grep HP
usb 5-1: Manufacturer: HP
HP CISS Driver (v 3.6.20)
usb 5-2: Manufacturer: HP
usb 5-2.2: Manufacturer: HP
input: HP Virtual Keyboard as /devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb5/5-1/5-1:1.0/input/input1
generic-usb 0003:03F0:1027.0001: input,hidraw0: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1/input0
input: HP Virtual Keyboard as /devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb5/5-1/5-1:1.1/input/input2
generic-usb 0003:03F0:1027.0002: input,hidraw1: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1/input1
scsi 2:0:0:0: CD-ROM    HP    Virtual DVD-ROM    PQ: 0 ANSI: 0 CCS

In the above code-snippet, the HP RAID driver is registered as “CISS”. Let’s repeat the grep, but with ciss as the filter:

# dmesg | grep ciss
cciss 0000:25:00.0: PCI INT A-> GSI 34 (level, low) -> IRQ 34
cciss 0000:25:00.0: irq 69 for MSI/MSI-X
cciss 0000:25:00.0: irq 70 for MSI/MSI-X
cciss 0000:25:00.0: irq 71 for MSI/MSI-X
cciss 0000:25:00.0: irq 72 for MSI/MSI-X
IRQ 71/cciss0: IRQF_DISABLED is not guaranteed on shared IRQs
cciss0: <0x3230> at PCI 0000:25:00.0 IRQ 71 using DAC
cciss/c0d0: unknown partition table
cciss 0000:02:00.0: PCI INT A-> GSI 16 (level, low) -> IRQ 16
cciss 0000:02:00.0: irq 73 for MSI/MSI-X
cciss 0000:02:00.0: irq 74 for MSI/MSI-X
cciss 0000:02:00.0: irq 75 for MSI/MSI-X
cciss 0000:02:00.0: irq 76 for MSI/MSI-X
IRQ 75/cciss1: IRQF_DISABLED is not guaranteed on shared IRQs
cciss1: <0x3230> at PCI 0000:02:00.0 IRQ 75 using DAC
cciss/c1d0: p1 p2
cciss/cldl: unknown partition table

As you see, this gives us two partitions found on one RAID controller (cciss/c1d0), and an empty partition table for the other. There’s obviously a destroyed MBR there. We can try to mount the first partition now:

# mount /dev/cciss/c1d0p1 /tmp/1 -t ext2
EXT2-fs warning (device cciss/c1d0p1): ext2_fill_super: mounting ext3 filesystem as ext2
# ls -l /tmp/1>

rw-r--r--     1 0 0    1724047     Oct 21 07:45   System.map-2.6.32-4-pve
rw-r--r--     1 0 0    106083      Oct 21 07:45   config-2.6.32-4-pve
drwxr-xr-x    2 0 0    4096        Dec 8 06:54    grub
rw-r--r--     1 0 0    10404494    Nov 13 07:17   initrd.img-2.6.32-4-pve
rw-r--r--     1 0 0    10402231    Nov 8 13:42    initrd.img-2.6.32-4-pve.bak
drwx------    2 0 0    16384       Nov 8 12:38    lost+found
rw-r--r--     1 0 0    124152      Oct 8 2008     memtest86+.bin
rw-r--r--     1 0 0    2507616     Oct 21 07:45   vmlinuz-2.6.32-4-pve

The partition that we’ve mounted is actually the original /boot directory. Now we need to transfer the necessary files onto our local Linux machine, and from these, create a mini boot ISO image, specifically designed for our server. How do we do this? We still have an unused virtual USB drive slot. Let’s first prepare a loopback device file on our local system:

# dd if=/dev/zero bs=1M count=40 of=blank.iso
# /sbin/mkfs.ext3 -F blank.iso

Next, we connect the blank.iso image with the ext3 filesystem to the server USB slot via the Java applet (the Virtual Floppy/USBKey option seen in Figure 3). After this, tailing dmesg on the server console shows up something like the following:

usb 5-2.1: new full speed USB device using uhci_hcd and address 10
usb 5-2.1: New USB device found, idVendor=03f0, idProduct=1427
usb 5-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2.1: Product: Virtual Floppy
usb 5-2.1: Manufacturer: HP
usb 5-2.1: configuration #1 chosen from 1 choice
scsi8 : SCSI emulation for USB Mass Storage devices

# scsi 8:0:0:0: Direct-Access        HP    Virtual ReyDriue 0.01 PQ: 0 ANSI: 2
sd 8:0:0:0: Attached scsi generic sg2 type 0
sd 8:0:0:0: [sda] 81920 512-byte logical blocks: (41.9 MB/40.0 MiB)
sd 8:0:0:0: [sda] Write Protect is off
sd 8:0:0:0: [sda] Assuming drive cache: write through
sd 8:0:0:0: [sda] Assuming drive cache: write through
sda: unknown partition table
sd 8:0:0:0: [sda] Assuming drive cache: write through
sd 8:0:0:0: [sda] Attached SCSI removable disk

Now we mount the newly detected (virtual) USB drive:

# mount /dev/sda /tmp/2 -t ext3

We copy the kernel, the initramfs file with integrated drivers, and settings from the GRUB loader into the mounted image, sync, to ensure that buffers are flushed (data is written across the network onto our disk) and we then unmount our virtual USB drive:

# cp /tmp/1/initrd* /tmp/2
# cp /tmp/1/vmlinuz* /tmp/2
# cp /tmp/1/grub/menu* /tmp/2
# sync && umount /tmp/2

On the local machine, we disconnect the ISOs from the virtual USB and virtual CD using the Virtual Media tab of the Java applet. Now, we mount this image file locally, and with the help of the Syslinux utility, create a bootable ISO image.

First, we create a directory structure as follows:

./isolinux/isolinux.bin
./isolinux/isolinux.cfg
./vmlinuz-2.6.32-4-pve
./initrd.img-2.6.32-4-pve
./grub/menu.lst

The generated directory tree for the mini ISO image should conform to Syslinux rules. Next, we launch the mkisofs command with these options:

# mkisofs -o mini_iso.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table CD_root

Booting with the new mini-boot ISO

We connect the new mini_iso.isoas a virtual CD drive on the server, and in the iLO page’s Administration tab, click the Reset button. We switch to the virtual console and watch the system boot messages. As is visible in Figure 9, the system is intact, and boots as it should.

Boot messages

Figure 9: Boot messages

Cool! Now we need to point the boot loader to a new location — to be precise, to the beginning of the /dev/cciss/c1d0 device, where it was logically expected, but it wasn’t present. But first, let’s verify what we can see at the beginning of the second disk:

# hexdump -C /dev/cciss/c0d0 | less
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This tells us that the MBR table for the second RAID array is completely blank. Well, we’ve proved our hypothesis that the system fault was because of a blank MBR. It’s time to run GRUB, to generate an MBR for the new location:

# /usr/sbin/grub-install /dev/cciss/c1d0

Let’s make sure that the MBR is generated and written correctly for the first disk:

# hexdump -v /dev/cciss/c1d0 | less

0000000 48eb 0090 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0000 0000 0000 0000 0000 0203
0000040 00ff 2000 0001 0000 0200 90fa f690 80c2
0000050 0275 80b2 59ea 007c 3100 8ec0 8ed8 bcd0
0000060 2000 a0fb 7c40 ff3c 0274 c288 be52 7d7f
0000070 34e8 f601 80c2 5474 41b4 aabb cd55 5a13
0000080 7252 8149 55fb 75aa a043 7c41 c084 0575
0000090 e183 7401 6637 4c8b be10 7c05 44c6 01ff
00000a0 8b66 441e c77c 1004 c700 0244 0001 8966
00000b0 085c 44c7 0006 6670 c031 4489 6604 4489
00000c0 b40c cd42 7213 bb05 7000 7deb 08b4 13cd
00000d0 0a73 c2f6 0f80 ea84 e900 008d 05be c67c
00000e0 ff44 6600 c031 f088 6640 4489 3104 88d2
00000f0 c1ca 02e2 e888 f488 8940 0844 c031 d088
0000100 e8c0 6602 0489 a166 7c44 3166 66d2 34f7
0000110 5488 660a d231 f766 0474 5488 890b 0c44
0000120 443b 7d08 8a3c 0d54 e2c0 8a06 0a4c c1fe
0000130 d108 6c8a 5a0c 748a bb0b 7000 c38e db31
0000140 01b8 cd02 7213 8c2a 8ec3 4806 607c b91e
0000150 0100 db8e f631 ff31 f3fc 1fa5 ff61 4226
0000160 be7c 7d85 40e8 eb00 be0e 7d8a 38e8 eb00
0000170 be06 7d94 30e8 be00 7d99 2ae8 eb00 47fe
0000180 5552 2042 4700 6f65 006d 6148 6472 4420
0000190 7369 006b 6552 6461 2000 7245 6f72 0072
00001a0 01bb b400 cd0e ac10 003c f475 00c3 0000
00001b0 0000 0000 0000 0000 0000 0000 0000 0280
00001c0 0001 8183 8020 0040 0000 0000 0010 8200
00001d0 8001 fe8e ffe0 0040 0010 ac80 087a 0000
00001e0 0000 0000 0000 0000 0000 0000 0000 0000
00001f0 0000 0000 0000 0000 0000 0000 0000 aa55

Yes, the GRUB loader is generated on the first RAID disk. Now we flush any disk caches and do a warm reboot of the system, meanwhile disconnecting our mini-boot ISO from the virtual CD slot:

# sync && reboot

After reboot, we can enjoy the restored and, more importantly, the completely working system.

Here, I’ve explained in detail how to recover a Linux system with the help of the integrated iLO software from Hewlett-Packard. It no longer matters if you’re far away from office, so long as you’re able to log in to the company LAN via VPN, and able to access the server’s iLO home page. From here, you can quite easily perform non-trivial tasks like a full system reconfiguration of a working system, complete installation, or restoring after a crash.

Resources

All published articles are released under Creative Commons Attribution-NonCommercial 3.0 Unported License, unless otherwise noted.
Open Source For You is powered by WordPress, which gladly sits on top of a CentOS-based LEMP stack.

Creative Commons License.