Myrinet setup (was: RE: [Beowulf])

[Reuti]

Hi Will,

Quoting William Burke <wburke999 at msn.com>:

<snip>
> I have but I do not see my name yet? How long is the verification process?

did you register as an observer - AFAIK you can post to the SGE list without 
being registered.

<snip>
> The mpi directory's mpich.template doesn't use mpirun.ch_gm so how does it
> know what version of mpirun to use? If I use the mpi what changes do I have
> to make?

You can have more than one MPI implementation installed in your cluster, and it 
may need some planning to set up the correct $PATH for each of the 
implementations you want to use (and the this way located mpirun must fit to 
your used version of MPI during compilation of your program). You may use a 
"which mpirun" to check it in your job script.

Also the supplied sge_mpirun will not use any Myrinet version on it's own - 
it's just a wrapper to the mpirun you set in the PE, so that you don't have to 
specify the usual options 'mpirun -machinefile $TMPDIR/machines -np $NSLOTS 
mypgm'.

I must admit: seems that the Myrinet stuff was more for 5.3 and not updated, as 
in 6.0 you can have more than one line for "start_proc_args" in your PE 
definition - so it just grabs the last \ in the first and only line beginning 
with "start_proc_args" line as mpirun command - will give the error message you 
got, that "\" is not existing.

As I said: we can use the default MPICH integration also for Myrinet and 
proceed this way.

> >> Can you please give more details of your queue and PE setup (qconf
> -sq/sp

Thx, I will keep the stuff. First one additional question (before I route you 
in the wrong direction): is it necessary for you to have a shared $TMPDIR for 
SGE? This is the one you set in your queue configuration (tmpdir 
/WEMS/grid/tmp) and seems for now to be on a file server.

More common and faster is to use the local /tmp on the nodes for this (you are 
right: SGE want to create there a directory for this job and some file for its 
own usage - but you are free to use this directory $TMPDIR also in your job 
script). It will be created for your job, and cleanly deleted after the job, so 
you won't have any leftover files.

Cheers - Reuti