Linux on POWER

I guess that almost all desktop systems in the office or home are derived from one prototype, the IBM PC (personal computer), which was built around the concept of an open architecture that IBM introduced 30 years ago. It’s true that the company itself didn’t quite understand how revolutionary open hardware architecture would be a few years later, in the mid-80s. Whatever the case, the fact remains that the PC became popular because many independent manufacturers could produce a wide range of compatible peripherals. This, in turn, allowed a competitive market to develop, eventually making desktop PCs available to everyone.

The open development model of Linux first embraced the desktop market of x86 computers, and then proceeded to “capture” other architectures as well (now more than 20), ranging from x86, ARM and MIPS, to Alpha, SPARC, PowerPC, IA64 and S390.

While “desktop” platforms (x86 and ARM) have many different Linux distributions and other operating systems that can run on them, the “pure” server architectures like IA64, S390 and Power cannot boast of this — the number of distributions is much, much smaller. This fact is understandable. Quality and the cost of server hardware for industrial applications is immeasurably higher than for the usual PC. Moreover, not every Linux distribution vendor can find the resources to build and provide long-term support for major server hardware architecture — even if they are willing to.

Some time ago, I got my hands on an IBM IntelliStation POWER 285 workstation, considered the closest model to large industrial servers. The main brain of the station is a dual-core POWER 5+ processor.

Although announced back in 2005, the system is serious stuff even now. Judge for yourself: 4 GB RAM (max capacity 32 GB), with a disk subsystem that has 2×146 GB SCSI drives (max disk storage 1.2 TB). It has a self-diagnostic system and memory “self-healing” technology known as ChipKill, a remote HMC (Hardware Management Console) and a pair of NICs working at 1 Gbit. This system is unbeatable for an entry-level enterprise server and simply forces you to take another respectful look at IBM  — Big Blue creates innovative and striking products.

Well, that’s enough of an introduction. Let’s learn Linux installation and operation on IBM hardware. And later decide how it is possible to effectively exploit Linux on Power.

Remote access

IBM produces equipment that has OpenFirmware (OF) inside. This is analogous to the traditional PC BIOS used on x86 systems. With OF’s help, you can assign which source should be used to start an OS, perform a number of diagnostic procedures, etc. OF itself uses a ServiceProcessor — a special unit embedded into the big tower. When you plug in the power cord, first the ServiceProcessor starts. That initially sets up a remote access option, and then awaits the user’s response.

The user has to press the hardware Start button on the front panel. So, when plugged in, IntelliStation is never completely turned off — even with the main CPUs off, the ServiceProcessor doesn’t sleep. The scheme is reminiscent of a similar product — iLO, developed by Compaq/HP. iLO also lets one start and control a server remotely.

It’s worth noting that IBM also has additional functions run by HMC (Hardware Management Console), among which are planning and allocation of physical resources to logical partitions (LPAR/DLPAR), splitting main CPU performance onto logical CPUs (Capacity On Demand) and related actions that provide virtualisation management on POWER-based processors. Unfortunately, I can only mention these capabilities, and not go any deeper into the subject, because that would be beyond the scope of this review.

So, in order to establish an access link to the ServiceProcessor and later install Linux, you will need an RS232(F)<->RS232(F) cable and a dedicated workstation, from which you can directly control your IntelliStation 285. As a console, any PC with an installed command-line tool like minicom or surecom (a GUI alternative) will suit. Set the communication speed to 19,200 bit/s, the standard default rate for IBM hardware, plug in the power cord and wait for the login prompt after running minicom -D /dev/ttyS0 -s 19200 -o. See Figure 1.

To start the whole system and gain access to OF, from where you can select an OS source,  choose: 1. Power/Restart Control -> 1. Power On/Off System -> 8. Power on. After that, in approximately two minutes, you will hear the noise of the fans — the big system has started initialisation and a minute later the hardware will be completely initialised, signalled via two speaker beeps.

At the same time, on the console, you should see a message from SMS (System Management Services) as in Figure 2, where you choose 1 to enter the system OF menu and then select 1. Select Install/Boot Device to choose the boot device (which you will run the installer from).

You should see something like Figure 3, where I chose 4. IDE, because the built-in CD-ROM is connected to the IDE connector. In your version, it might be 3. CD/DVD.

In the subsequent menu (Figure 4), I chose 2. Normal Mode Boot, which is suitable for installation, and confirmed exit from SMS. My SLES 11’s Linux kernel correctly identified the IntelliStation 285.

The installation process will proceed as follows:

Welcome to SuSE Linux Enterprise 11!

Type  "install"  to start the YaST installer on this CD/DVD
Type  "slp"      to start the YaST install via network
Type  "rescue"   to start the rescue system on this CD/DVD

Welcome to yaboot version r22.8-r1190.SuSE
booted from '/pci@800000020000003/pci@2,3/ide@1/disk@0:1,\suseboot\yaboot.ibm'
running with firmware 'IBM,SF240_403' on model 'IBM,9111-285', serial 'IBM,0206ABB30', partition '0'
Enter "help" to get some basic usage information
boot:
Please wait, loading kernel...
Allocated 01800000 bytes for executable @ 03000000
Elf32 kernel loaded...

SuSE Linux zImage starting: loaded at 03000000-0469cd30 (4000000/0/02039a68; sp: 02c9fd50)
uncompressing ELF header done. (00000100 bytes)
Allocated 0164c090 bytes for kernel @ 04800000
Leave 0122a5ba bytes for initrd @ 034668bb
uncompressing kernel done. (00f5a920 bytes)
entering kernel at 04810000(34668bb/122a5ba/02039a68)

The installation procedure reminds me of what happens on the x86, with just one difference — in our case, control is via a serial line in text VT102 mode, so some characters may not be displayed properly as you can see in Figure 5.

Several menu items, such as the network address, may require you to return to re-edit these fields. We can use Tab/Alt+Tab to move between text interface fields. These keyboard shortcuts are functional even in this very minimal mode.

Embedded Linux on ServiceProcessor

In addition to control via a serial line, IntelliStation (and in general, all IBM products) offers management via an ordinary Web browser. This Web control mechanism (Advanced System Management — see Figure 6) is provided by a tiny computer within the IntelliStation, with Linux and BusyBox utilities.

This ServiceProcessor also uses Power technology — a PowerPC 405 CPU from IBM [Datasheet PDF] but with a much smaller form factor. This uses a low-power processor FSP (Flexible Service Processor) running at about 200 MHz (see Figure 7).

ASM provides the same controls as the serial-line SMS but in a more usable manner (Figure 8). It lets you manage the system remotely — start/stop it, view logs, allocate resources (like Capacity on Demand parameters), etc.

Let’s briefly see how this subsystem is launched. When you connect the mains power to the box, the ServiceProcessor automatically starts the embedded Linux. After initially testing  some components of a larger block, ServiceProcessor displays on a front panel the readiness status for full system start. When you hit the “Power On” button, OpenFirmware loads and ultimately passes control to your Linux boot loader, yaboot.

Like with the large system, this small subsystem has two independent Ethernet controllers, referred to as HMC 1 and HMC 2. It makes sense to connect them to separate LANs/VLANs for a special management network. First, it helps access restriction and security. Second, you can avoid a network traffic overload on a single LAN.

IP addresses for these ports can be supplied by a DHCP server on a management network. If one isn’t found, default addresses are automatically assigned: 192.168.2.147 (port C7-T1, on HMC1) and 192.168.3.147 (port C7-T2, HMC2). You can log in via telnet, with the default credentials admin/admin.

$ mount
/dev/mtdblock/0/3 on / type cramfs (rw)
none on /dev type devfs (rw)
proc on /proc type proc (rw)
/dev/mtdblock/0/1 on /opt/extucode type jffs2 (rw)
/dev/mtdblock/0/1 on /etc type jffs2 (rw)
/dev/mtdblock/tmpram/tmpram on /tmp type jffs2 (rw)
none on /var type tmpfs (rw)
/dev/mtdblock/pcscore/pcscore on /core type jffs2 (rw)
/dev/mtdblock/pcsram/pcsram on /opt/p1 type jffs2 (rw)
shm on /var/shm type shm (rw)
/dev/mtdblock/1/1 on /alt/extucode type jffs2 (rw)
/dev/mtdblock/2/3 on /opt/fips type cramfs (rw)
/dev/mtdblock/nvram/nvram on /opt/p2 type zofs (rw,sync)
$ uptime
8:49am  up 1 day, 14:53, load average: 0.52, 0.20, 0.07
$ uname -a
Linux fsp 2.4.18-rc4 #1 Wed Mar 25 15:01:22 UTC 2009 ppc unknown
$ cat /proc/version
Linux version 2.4.18-rc4 (root@mcpbuild6) (gcc version 3.2.3 20030401 (prerelease)) #1 Wed Mar 25 15:01:22 UTC 2009

As we see, Embedded Linux uses JFFS/CRAMFS filesystems, which are common in tiny products. An interesting feature is NVRAM memory mounted as storage (/dev/mtdblock/nvram/nvram). Reading data from its files gives the status of different subsystems. You can also change some values using commands from /opt/fips/bin — particularly, you can even try to launch several subsystems (or maybe the full system box) with /opt/fips/bin/powr_fsp_spcn, which checks the current system status, and accordingly launches /bin/powr_L4 or /bin/powr_sqh. The ASM Web server (named webs), when you click the button Save settings and power on does the same.

Available distros for POWER

Originally, I planned to use Red Hat Enterprise Linux as the main OS on this system. However, neither RHEL 5.7 nor 6.2 Beta, nor even 6.2 started correctly. RHEL 6.2 Beta and a more recent version 6.2 spews a kernel trap error during initial startup. Yes, the IntelliStation 285 wasn’t in the supported hardware list, but neither was it for SLES 10/11. Still, SLES 10 and 11 support it unofficially. Let us see how the installation process goes, and whether we can bring the system to production mode.

In addition to RHEL and SLES, numerous systems based on the POWER processor even supported Ubuntu 10.04 (LTS edition). However, my experience proved that this model wasn’t suitable for Ubuntu — it cannot start properly. Moreover, newer versions of Ubuntu come without any POWER/PowerPC architecture support at all, unfortunately.

I checked if the most recent release (6.0.3) of Ubuntu’s parent distribution, Debian, could boot. It wasn’t able to start — it drops directly into OpenFirmware. Another failure! Well, let’s not despair. Quite possibly, the Debian community will fix it.

Oddly enough, a completely amateurish project called Crux PPC has a fully working distribution — Crux PPC 2.7a, which not only loads on Power hardware, but also provides video output via correctly-set framebuffer mode. None of the other distributions mentioned can boast of that! All provided only a serial-line text mode. In general, Crux PPC runs as a LiveCD and has no installer, so I advise professionals to use it. Again, this topic is out of the scope of this article, so I will leave it for later.

Performance

As I mentioned a bit earlier, IBM products are one level, or a “head above” their competitors. Comparing the very different Power and x86 architectures is hard, because there are no objective criteria for such a comparison. However, from an ordinary user’s perspective, we can try the 7Zip archiver in its benchmark mode. This program is completely open source, so anyone can compile it.

Let us compare IntelliStation based on the POWER5+ processor with 2 cores and 4 GB RAM, manufactured in 2005, with an Intel Core 2 Duo (2 cores with hyper-threading) and 4 GB RAM, but released three years later (2008). First, the IntelliStation:

linux:~/src/p7zip_9.20.1/bin # ./7za b

7-Zip (A) 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

RAM size:    3632 MB,  # CPU hardware threads:   2
RAM usage:    425 MB,  # Benchmark threads:      2

Dict        Compressing          |        Decompressing
      Speed Usage    R/U Rating  |    Speed Usage    R/U Rating
       KB/s     %   MIPS   MIPS  |     KB/s     %   MIPS   MIPS

22:    1931    99   1891   1878  |    24309   100   2198   2194
23:    1871   100   1916   1907  |    24132   100   2212   2209
24:    1816    99   1964   1953  |    23917   100   2221   2219
25:    1780    99   2043   2032  |    23525   100   2213   2212
----------------------------------------------------------------
Avr:           99   1954   1942               100   2211   2209
Tot:          100   2082   2076

Next, let us test the Core 2 Duo system:

Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz
7-Zip 9.04 beta  Copyright (c) 1999-2009 Igor Pavlov  2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

RAM size:    4006 MB,  # CPU hardware threads:   2
RAM usage:    425 MB,  # Benchmark threads:      2

Dict        Compressing          |        Decompressing
      Speed Usage    R/U Rating  |    Speed Usage    R/U Rating
       KB/s     %   MIPS   MIPS  |     KB/s     %   MIPS   MIPS

22:    2588   138   1831   2518  |    33730   165   1850   3045
23:    2532   136   1903   2580  |    32265   159   1853   2954
24:    2588   142   1958   2783  |    32828   165   1843   3046
25:    2540   144   2011   2900  |    30299   155   1838   2849
----------------------------------------------------------------
Avr:          140   1925   2695               161   1846   2974
Tot:          150   1886   2834

As you can see, the performance doesn’t differ much. Even a slightly outdated IBM station can compete with a relatively new x86 product.

Despite the fact that equipment based on the Power processor is marketed by IBM as a corporate power-horse for commercial operating systems like AIX and IBM i (produced by IBM itself), we have seen that it can also be used with Linux. Thus, virtually the entire software stack that successfully works on x86 can also be run on Power — Apache, Java, WebSphere, MySQL, etc.

Taking into account the fact that IBM hardware usually comes with unique technologies like the hardware management console and the hardware virtualisation hypervisor, you can easily create an effective solution by combining the best features from both the hardware and software sides — the corporate and the public world.