[Beowulf] PXE boot with X7DWT-INF IB onboard card

[Beat Rubischon]

Hi Duke!

On 12.11.12 12:33, Duke Nguyen wrote:
> Unfortunately after reading, I found out that our built-card is too old 
> for Mellanox FlexBoot

"PXE Boot" is a nice container (buzzword? :-) for a hand full of simple
steps. First the PXE Boot ROM asks for a DHCP address, second it will
load PXElinux over tftp. PXElinux will ask again for a configuration
file and finally loads the kernel and an initial ramdisk. Again tftp is
used.

DHCP and tftp are damn simple protocols on top of IPv4 UDP to avoid a
full featured TCP/IP stack in the boot ROMs. They are also pretty slow.
Loading a file over TFTP isn't limited by the network, it's usually
limited by the slow implementation of the client side stack in the boot
ROM. Improvements by switching from GigE to IB won't give you any
performance improvements. It's usually also not "big data". We are
talking about ~100MBytes for a kernel and an initrd.

Your task is now to add all IB relevant stuff into the initrd to make
sure your adapters are brought up and you are able to mount your central
storage over NFS using IPoIB, NFSoRDMA, iSCSI using IPoIB, SRP or
whatever you like to use. BTW: There is no difference in this initrd if
you will load it over IB or GigE. It's absolutely the same work.

The only advantage of booting over IB using a tool like FlexBoot from
Mellanox would be the possibility to eliminate the separate GigE
connection to the nodes. But wait, we don't have IPMI over IB. I was
told that the Mellanox chips have a dedicated foot providing IPMI, but
no mainboard vendor ever connected this one to the BMC. At the end
you'll need the GigE cabling for IPMI and I see no point to fiddle
around with the firmwares and BIOSses to make the first steps going over IB.

Beat

-- 
     \|/                           Beat Rubischon <beat at 0x1b.ch>
   ( 0-0 )                             http://www.0x1b.ch/~beat/
oOO--(_)--OOo---------------------------------------------------
Meine Erlebnisse, Gedanken und Traeume: http://www.0x1b.ch/blog/