[Beowulf] Re: Any industry-standards that allow automated BIOS modifications and dumping? IPMI cannot do it, can it?

[Don Holmgren]

On Wed, 21 Oct 2009, Rahul Nabar wrote:

> On Wed, Oct 21, 2009 at 9:18 PM, Rahul Nabar <rpnabar at gmail.com> wrote:
>> I did find the "bootdev" option. So I can change the boot seq. but not
>> the other BIOS settings.
>>
>> Any "industry standard" to change BIOS settings unattended. Preferably over LAN?
>
> I guess another way to pose the question is:
>
> What's the converse command (if at all exists) for "dmidecode".
>
> dmidecode (or biosdecode) pops the BIOS settings to console. What can
> push settings into the BIOS?
>




A long answer, sorry - read at your own risk.  Maybe there will be some useful 
information.  Since you have Dell equipment maybe their OpenManage stuff will do 
exactly what you want.

The short answer, unfortunately, is no.  Check with your system vendor to see
if they can provide a utility.

There is an industry standard - the "ISA System Architecture" - that defined
among many other things the required interfaces to the real time clock chip
(Motorola MC146818) that included a small amount of NVRAM - usually called the
"CMOS memory" - that was kept non-volatile by means of a battery on your
motherboard.  The ISA standard includes the specifications for how some of those
memory locations are to be used to store, for example, information about your
hard disk drive and your floppy drives (that is, the BIOS settings of that
time).  Pretty much every x86 PC that you buy today, AFAIK (I've never seen an
exception) still includes this RTC and NVRAM area.  Motherboards still have a
lithium battery that is used to keep this alive, although for a time in
competing RTC chips these batteries were integrated into the integrated circuit
(a very dumb idea, since systems are often used beyond the lifetime of
the batteries).

In Linux, /dev/nvram lets you access this area, and in 2.6 kernels (maybe in 2.4
also??) you can generally
      `cat /proc/driver/nvram`
to see a dump of the ISA-specified data stored there (you may have to `modprobe
nvram` first).

On some motherboards I've used over the years, the BIOS settings were all stored 
in this nvram area, and so to replicate the BIOS settings from one system to a 
second system with the same motherboard, you could readout all of the data and 
write it back to the second system.  There is a checksum kept in the nvram that 
has to be updated appropriately, otherwise at the next power on self test the 
system would complain about corrupted CMOS and would restore the default 
settings.  The Linux nvram driver handles the checksum updating for you.  (Not
really - keep reading).

The interface to access the RTC nvram area is very crude if you are writing
your own code (as opposed to using the Linux nvram driver): use the "out"
instruction to write the byte offset (the address) to I/O port 0x70, and then
either read ('in') or write ('out') a byte (or word) from I/O port 0x71.  See
     http://wiki.osdev.org/CMOS
     http://www.bioscentral.com/misc/cmosmap.htm
     http://bochs.sourceforge.net/techspec/CMOS-reference.txt
and
     http://www.pixelbeat.org/docs/bios/
for discussions.  In the ISA standard, the offset is limited to the range
0-0x80 (128 bytes).

The bad news is that motherboards typically have larger nvram areas than this 
128-byte ISA region, and so some of the BIOS settings you care about may be 
stored elsewhere (maybe even in EEPROM??).  Worse, they may protect the settings 
(in the sense of detecting corruption) with additional checksums or CRCs, and 
unless you know which bytes or even bit ranges are used (and in which order) to 
compute the CRC, you won't be able to change individual settings w/out causing 
the POST to declare corruption, even if you could figure out which bits and/or 
bytes corresponded to a given BIOS setting.

On either the Intel "Torrey Pines" or "Buckner" motherboards (can't remember
now), there was a second 128-byte region that was accessed via I/O ports 0x72
and 0x73 in the same way that 0x70 and 0x71 are used for the ISA nvram region.
All of the BIOS settings were in the two 128-byte regions, and we were able to
replicate BIOS settings in Linux by copying all the bytes from both regions from
one motherboard to another.  Sweet - saved tons of "crash cart" work on a very
big cluster.

Unfortunately, I've never gotten this same thing to work on motherboards since 
those Intel boards.  Often the POST detects corruption (although this can be 
useful - more below).  So, perhaps larger or other nvram resources are probably 
being used, and who knows (without reading the BIOS source code, which AMI, 
Pheonix, etc. will probably never give you) what the interface to that memory 
might be on a given computer.

On a large cluster of Intel Lancewood motherboards many years ago, we found a
brute force solution for forcing BIOS settings (these motherboards had the
annonying habit of deciding that the 2nd processor was dead, and you could only
resurrect SMP operations by going into the BIOS and telling the system to reset
the processor timer test; our cluster was remote and we needed to do this using
serial redirect, and so we needed to force the BIOS into a state where the
redirect worked).  We used a PCI bus analyzer (made by VMetro) to record all I/O
bus activity following executing the "Save Settings and Reboot" menu option in
the BIOS.  All activity to ports like 0x70 and 0x71 above was captured.  Through
trial and error we parsed out of this huge trace the raw I/O commands that
were sufficient to put the BIOS settings into a known good state.  When a
Lancewood wiped the CMOS settings, in Linux we "replayed" this magic set of I/O
instructions.

On request, various manufacturers (Intel, Asus, Supermicro) have provided us
utilities (always for DOS) for reading out and writing BIOS settings - you can
always ask your vendor to see if such a utility is available for your hardware.
It's a bit of a pain to have to go into DOS to do this, but it's better than
nothing (we usually include a DOS partition on our system disks that can be used
for flashing the BIOS, BMC, gigE, and other firmware).  For one cluster, the
motherboard manufacturer provided us with a tool from the BIOS vendor (AMI?)
that allowed us, in a Windows application, to alter the default BIOS settings.
These then get loaded when during the power on self test CMOS corruption is
detected; to corrupt our CMOS, in Linux we just scribbled in the RTC nvram
region.

Hope this was useful, or at least maybe a mildly interesting read.  It would be
absolutely terrific if BIOS vendors would always provide the software to
manipulate, or at least replicate, BIOS settings.  Somehow it seems like well
into the 2nd decade of Linux clusters this particular issue should have been
solved once and for all.


Don Holmgren
Fermilab