[Beowulf] Sil 3124 controller, boot from disk, UUID issues

[David Mathog]

The Arima HDAMAI's onboard Sil 3114 controller wasn't able to deliver
maximum disk performance even after much tweaking.  So a Syba
SY-PCX40009 (Sil 3124) PCI-X controller was installed (firmware 6.3.18).
 When the system was booted from a small disk still on the 3114
controller linux had no problem seeing a large disk on the 3124
controller, and it could mount the large disks partitions and use them
normally.  Performance testing for the disk on the 3124 was much better
than for the 3114, with a sustained write of 4GB at 108.4MB/s, and
bonnie++ results of:

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
newsaf.bio.calte 8G 49331  94 116716  45 51925  18 53114  92 136625  21
315.3   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
newsaf.bio.caltech.edu,8G,49331,94,116716,45,51925,18,53114,92,136625,21,315.3,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Here's the problem - in order to boot from a Sil 3124 controller the
disk must be configured as a single disk concatenation set.  If it
isn't, the disk does not show up in the BIOS boot menu.  This is
different from the 3114 controller, where any disks plugged in, but not
part of a raid set, show up on the boot list.  After doing that, the
concatenation set is listed as a disk in the boot menu in the bios (not
the disk itself).  Booting from this "concatenation set" works, mostly,
but /dev/sda7 was not mounting.  However, it could be mounted later with

mount -t ext4 /dev/sda7 /home

This shows why the original mount fails:

# blkid
/dev/sda6: UUID="80d7830d-53f5-4b69-8b4b-e67ee1d47c9c" TYPE="ext4" 
/dev/sda1: UUID="ce8b6ea5-8bca-45d3-b9f0-a265e611b929" TYPE="ext4" 
/dev/sdb1: UUID="caf1e9ea-4eae-41f8-98d1-c4e4cbd6b102" TYPE="ext4" 
/dev/sdb6: UUID="2b0ef732-9526-49c1-9cb9-f0b73c86874c" TYPE="ext4" 
/dev/sdb5: UUID="485b0c52-e133-4359-88fa-c01d7d4ae3d1" TYPE="swap" 
/dev/sda5: UUID="eb79e4d0-66a2-473d-a8fa-5c444e38bb87" TYPE="swap" 
/dev/sda7: TYPE="silicon_medley_raid_member" 

and /etc/fstab had this to get /dev/sda7 mounted:

UUID=2c87348d-a79e-46ff-8693-1df40409a805 /home ext4 relatime 1 2

where that was the UUID seen by the small disk before the disk was made
into a concatenation set.  Earlier I had found that plugging disks into
the 3114 caused them to have a higher priority than those on the 3124,
even when the boot device was on the 3124, so using "/dev/sda7"
type entries wasn't reliable if the system was being reconfigured - it
could come up as sdb or sdc.

What I'm seeking is a way to configure this system so that it will boot
properly from the disk on the 3124 controller whether or not there are
disk(s) also plugged into the 3114.  Possible?  The only thing I could
come up with that was sure to work was to NOT make the boot disk into a
concatenation set, and boot the kernel (with sata_sil24) from a USB key,
then have it pick up its partitions.  Is there a way to do this without
resorting to arcane boot mechanisms???

I foresee problems at reboot too, because:

# fsck -t ext4 /dev/sda7
fsck from util-linux-ng 2.16.1
fsck: fsck.silicon_medley_raid_member: not found
fsck: Error 2 while executing fsck.silicon_medley_raid_member for /dev/sda7

Besides this oddness with the UUID on the 7th partition, what else might
the controller have done when it set this disk up as single disk
concatenation set?  Would it be prudent to boot from another OS
(network), remake the filesystems, and restore them from backups?

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech