[Beowulf] Open InfiniBand

[Greg M. Kurtzer]

I have been experimenting with OpenIB and OpenMPI on several clusters. I
will be building another one today (pending no fires) with cAos2 and
Warewulf. cAos-2 actually has the openIB gen2 stack in the standard
kernel (the 2.6.14.2 in the testing repository). I have seen several
other problems with that kernel on Dual core Opterons systems WRT to PCI
express (which is why that kernel hasn't made it out of the testing
repository yet). Luckily the system I want to run on is a EM64T ;)

cAos-2 also has current MPI's in its ports tree which also integrated
with environment modules. This means you can do:

   # cd /usr/src/ports/programs/openmpi/1.0
   # make install

And then all users can select the MPI that they want to use by:

   $ module load openmpi/1.0

OK, enough evangelizing... Back to the question at hand.

Our initial tests have been promising. Not perfect yet, but good enough
to start testing for production systems. (we would rather utilize a
community maintained solution then a proprietary stack that locks us to
a particular version/infrastructure (obviously)).

On Wed, Dec 21, 2005 at 09:55:55AM +0100, Daniel Pfenniger wrote:
> Hi,
> 
> We have a cluster with InfiniBand network that works flawlessly.  The problem is
> that the binary IB drivers are proprietary, and the company has been recently
> absorbed by a larger one, which means that the driver support for Mellanox
> cards is now frozen.
> The proprietary drivers prevents to upgrade or tweak the kernel, or to use
> a recent version of mpich2 (with MPI 2 and one-sided communications).
> 
> Fortunately OpenIB (openib.org) has published open source drivers which are
> included in the latest kernels >= 2.6.14.
> Before exploring this option I just want to ask the list if anyone has used the
> OpenIB drivers for production work, in particular with an open source MPI?
> 
> Thanks, Dan
> 
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Greg Kurtzer
Berkeley Lab, Linux guy