[Beowulf] PXE boot with X7DWT-INF IB onboard card

[Duke Nguyen]

On 11/13/12 3:32 PM, Beat Rubischon wrote:
> "PXE Boot" is a nice container (buzzword? :-) for a hand full of simple
> steps. First the PXE Boot ROM asks for a DHCP address, second it will
> load PXElinux over tftp. PXElinux will ask again for a configuration
> file and finally loads the kernel and an initial ramdisk. Again tftp is
> used.
>
> DHCP and tftp are damn simple protocols on top of IPv4 UDP to avoid a
> full featured TCP/IP stack in the boot ROMs. They are also pretty slow.
> Loading a file over TFTP isn't limited by the network, it's usually
> limited by the slow implementation of the client side stack in the boot
> ROM. Improvements by switching from GigE to IB won't give you any
> performance improvements. It's usually also not "big data". We are
> talking about ~100MBytes for a kernel and an initrd.

Yes, I am reading (and practising) how to boot over IP using DHCP and 
tftp now. Still struggling to get it successfully booted.

>
> Your task is now to add all IB relevant stuff into the initrd to make
> sure your adapters are brought up and you are able to mount your central
> storage over NFS using IPoIB, NFSoRDMA, iSCSI using IPoIB, SRP or
> whatever you like to use. BTW: There is no difference in this initrd if
> you will load it over IB or GigE. It's absolutely the same work.
>
> The only advantage of booting over IB using a tool like FlexBoot from
> Mellanox would be the possibility to eliminate the separate GigE
> connection to the nodes. But wait, we don't have IPMI over IB. I was
> told that the Mellanox chips have a dedicated foot providing IPMI, but
> no mainboard vendor ever connected this one to the BMC. At the end
> you'll need the GigE cabling for IPMI and I see no point to fiddle
> around with the firmwares and BIOSses to make the first steps going over IB.

Hum... looks like I still need a lot of readings :).

On 11/13/12 4:20 PM, Jon Tegner wrote:
> How are you planning to boot your nodes?
>
> I have used perceus (http://www.perceus.org/) and was happy with it.
> There is also Warewulf (http://warewulf.lbl.gov  or
> http://hpc.admin-magazine.com/Articles/Warewulf-Cluster-Manager-Master-and-Compute-Nodes)
> which I haven't used.
>
> Anyone who has compared the two?

I dont know :(. All of this is new to me. I am reading and testing with 
tftp/dhpc/nfs now. Will try any other recommendation when I am able to 
boot up the nodes.

On 11/13/12 4:30 PM, Beat Rubischon wrote:
> I usually use the PXE mechanism to boot the standard installer image of
> my distribution of choice. Every recent distribution brings you the
> needed files on your installation media - have a look in images/pxeboot/
> of your RHEL / CentOS / Scientfic DVD, boot/*/loader of your SuSE DVD,
> the pxe subdirectory of the Ubuntu boot image directory... Once bootet,
> you deliver the packages using http, nfs etc. Just copy the content of
> the DVD or FTP directory to your headnode and make it available.

Yes, I did try this, and was able to boot Scientific Linux 6.3 LiveCD. 
But still struggling on next steps.

> To automate the installation by itself every distrbution accepts
> automatic installation instructions. RedHat calls it Kickstart, SuSE
> AutoYaST, Debian / Ubuntu Preseed. You'll find plenty of examples in the
> web. And you'll find plenty of tools automatic this stuff.
>
> No need to use CDs oder DVDs. No need to answer any questions. Only need
> to learn a little bit of systems management :-)
>
> Or to pay somebody delivering you support and / or software (-:

Sure, I prefer learning :).

On 11/13/12 5:19 PM, Eugen Leitl wrote:
> On Tue, Nov 13, 2012 at 10:45:36AM +0700, Duke Nguyen wrote:
>> Thanks for all suggestions and comments. Looks like we will go for a
>> gigabyte switch and boot nodes over Gb. Not to mention that this
> You probably mean GBit Ethernet. Gigabyte is a motherboard vendor.
>

Yes, that was what I meant.

On 11/13/12 5:29 PM, Hearns, John wrote:
> My honest advice to you is not to do any of this.
>
> There are lots of reliable, knowledgeable companies out there who will only
> Be too willing to partner with you and construct a cluster, plus expandable storage for you.
>
> I suggest that you start looking on various sites, eg Clustermonkey and Hpcwire,
> And invite some vendors in to talk to you.
>
> Many vendors have pre-packaged clusters which would suit you just fine.

Thanks for your very honest advice! Unfortunately it is not similar 
situation like in US, where we can find companies with appropriate 
services to help and assist us. In Vietnam, where we are now, cluster 
systems are already rare. In fact Intel is helping Vietnamese Goverment 
to build a first, big HPC cluster:

http://www.hpcwire.com/hpcwire/2012-10-01/hpc_comes_to_vietnam.html

Not to mention that the service's price, if available in Vietnam, might 
be far beyond our funding. Last, but not least, we are kind of building 
and learning for ourselves, so I prefer to learn (the hard way) :D.

D.