[Beowulf] Rocks Cluster and 2 Ethernet networks

[Reuti]

Hi,

> I'm considering installing the Rocks cluster distro on a cluster that uses
> only ethernet.  As I understand it, eth0 (or first network interface) is
> used for administration and also message passing if no other high speed
> interface is present (e.g. myrinet).  My question is, if each of my
> compute node have two ethernet interfaces, say eth0 and eth1, can the
> cluster be configured that message passing takes place only over eth1?  It
> would be nice to have an interface devoted to just message passing.  If it
> is possible, how would I go about setting it up?  If it's not possible, is
> there are a lot of performance loss due to the fact that other tasks (such
> as administration, etc.) are also taking place over eth0?  Thanks in
> advance for your help.

do you want to use the ch_p4 device of MPICH for communication? Then you simply 
have to set the machinefile for mpirun to include only the names of the second 
interface in all nodes. Maybe your queuingsystem can do this already for you. 
Furthermore, you have to change the setting in mpirun.args that way, that 
instead:

MPI_HOST=`hostname`

will be substituded with the name of the second interface. E.g.

MPI_HOST=`hostname | sed "s/^node/internal/"`

to change the name from node001 to internal001 or whatever names you use. 
Otherwise your machinefile will be scanned in a wrong way (wrong distribution 
of the processes to the nodes in the end), and the communication back from the 
slaves to the head node of the job will still use the wrong interface. You can 
simply include this at the beginning of the mpirun.arg file. If it's already 
set, it will no be set later in the script.

Cheers - Reuti