<HTML>
<HEAD>
<TITLE>Re: [Beowulf] Please help to setup Beowulf</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
<BR>
On 2/20/09 10:23 AM, "Prentice Bisbal" <<a href="prentice@ias.edu">prentice@ias.edu</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Bogdan Costescu wrote:<BR>
> On Fri, 20 Feb 2009, Glen Beane wrote:<BR>
><BR>
>> I looked into SGE a long time ago, but I found the MPI support<BR>
>> terrible when compared to TORQUE/PBS Pro<BR>
><BR>
> Indeed and AFAIK is still in a similar state today. There was talk for a<BR>
> long time on the SGE devel list for a TM API to be added, but it seems<BR>
> like this is not considered a high priority feature. I've not only<BR>
> looked but actually used SGE for about 1 year (IIRC, about 5 years ago)<BR>
> during which I had to spend time fixing the interactions with LAM/MPI<BR>
> and many of the parallel applications that were used on that cluster -<BR>
> and finally gave up. On the plus side, during the time that SGE was<BR>
> used, I have never seen a process left behind from a job and the<BR>
> queueing system itself seemed very stable - something that I could not<BR>
> say for the OpenPBS/Torque that I've also tested at that time.<BR>
><BR>
<BR>
You need to take a fresh look at SGE and Open MPI. Open MPI seems to be<BR>
the new de facto standard MPI library, and you can compile it to be<BR>
fully integrated with both SGE and Torque. I just set up a cluster using<BR>
SGE and Open MPI (built with the --with-sge option), and I there's no<BR>
need to tinker with the SGE's MPI startup wrapper scripts like in the<BR>
past. Everything just works: SGE and OpenMPI communicate directly with<BR>
each other, and SGE has complete control over ALL the MPI processes.<BR>
<BR>
A couple of years ago I did setup SGE with MPICH, and had to tinker with<BR>
SGE's startup scripts to get everything to work correctly. Not that<BR>
difficult.<BR>
<BR>
I could be wrong but I think at that time, to use Torque you needed to<BR>
compile a separate mpiexec program developed by a 3rd party to get<BR>
"tight integration" between MPI and Torque.<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>It depended on the MPI implementation (and it still does). If it supported TM, then no 3rd party job launcher was necessary. For things like mpich 1.x that did not have TM support OSC’s mpiexec job launcher provided tight torque/PBS integration. LAM-MPI and OpenMPI have had TM support for a long time.<BR>
<BR>
<BR>
<BR>
-- <BR>
Glen L. Beane<BR>
Software Engineer<BR>
The Jackson Laboratory<BR>
Phone (207) 288-6153<BR>
<BR>
</SPAN></FONT>
</BODY>
</HTML>