[Beowulf] Three questions on a new Beowulf Cluster

[Joe Landman]

John P. Kosky, PhD wrote:
> My company is taking it's first foray into the world of HPC with an 
> expandable architecture, 16 processor (comprised of quad core Opterons), 
> one header node cluster using Infiniband interconnects. OS has 
> tentatively been selected as SUSE 64-bit Linux. The principal purpose of 
> the cluster is as a tool for spacecraft and propulsion design support. 
> The cluster will therefore be running the most recent versions of 
> commercially available software - initially for FEA and CFD using COMSOL 
> Multiphysics and associated packages, NASTRAN, MatLab modules, as well 
> as an internally modified and expanded commercial code for materials 
> properties prediction,with emphasis on polymer modeling (Accelrys 
> Materials Studio). Since we will be repetitively running standard 
> modeling codes on this system, we are trying to make the system as user 
> friendly as possible... most of our scientists and engineers want to use 

Could you elaborate on this a little?  Do you want your users not to use 
command lines to submit jobs, but web interfaces instead?  Or are 
command lines ok?  This is usally cited as what people mean by "user 
friendly".

> this as a tool, and not have to become cluster experts. The company WILL 
> be hiring an IT Sys Admin with good cluster experience to support the 
> system, however...
> 
> Question 1:
> 1) Does anyone here know of any issues that have arisen running the 
> above named commercial packages on clusters using infiniband?

Not all of them use the exact same version of MPI stack.  We have 
customers running similar mixes (Dyna, NASTRAN, Accelrys, ...), and the 
stacks vary somewhat.  Try to use a similar MPI stack throughout (quite 
a few CAE codes will use HP MPI or Intel MPI).  This may save you some 
grief.

In both those cases, the MPI stack is pretty smart about linking to the 
Infiniband, though make sure that the MPI stack will talk to the correct 
library in your IB stack (DAPL or verbs or ...).

We normally use OFED for our customers IB efforts.  Some of the MPI 
stacks are compiled against vendor specific versions of IB stacks.


> Question 2:
> 2) As far as the MPI for the system is concerned, for the system and 
> application requirements described above, would OpenMPI or MvApich be 
> better for managing node usage?

The applications dictate what they are compiled against for MPI library 
usage.  MPI stacks are not ABI compatible, you cannot run mvapich 
binaries with an OpenMPI stack.

Moreover, MPI does not aide in the management of nodes.  There are other 
packages for that.  Some of the better ones for system management are 
Perceus, Rocks, and a few others.  Since you are using SuSE, Rocks is 
out.  We have built SuSE based clusters for quite a few customers, 
though most of the major cluster packages really don't support it that 
well.  Currently working on a diskless SuSE system.  Have it working, 
though still a bit more work to do on other elements.  The diskful 
system works quite well, and is effectively automatic at this point.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615