[Beowulf] mixing of Infiniband HCAs in cluster

[Gil Bloch]

> 
> In message from "Gil Bloch" <gil at mellanox.co.il> (Wed, 24 May 2006 
> 18:30:44 +0300):
> >>
> >> We have a pair of IB 4x (Mellanox MTLP23108) HCAs used currently 
> >>for  direct switchless connection of pair of nodes.  Now we plan to 
> >>buy 8-port (MTK37900) or 24-port IB switch w/HCAs,  which may be 
> >>also DDR IB. And we want to build this cluster including
> >> also old pair of nodes w/MTLP23108 HCAs. But is this possible - to
> >> have both HCA kinds (DDR and usual IB 4x HCAs) connected to the
same
> >> switch ? If the answer is "yes", is it necessary to setup all the
> >> DDR-HCA drivers manually for work w/"4x speed" or all the necessary
> >> things will be done via negotiations ?
> 
> >
> >InfiniBand defines auto-negotiation of link speed, ...
> >The only thing you should notice is that streams from DDR HCAs to SDR

> >HCAs should reduce data rate to avoid flooding the switches. This is 
> >done on a per-stream (QP) base, thus the DDR port can be fully 
> >utilized. Using CMA for connection settings will take care of it
> >>automatically. If
> >your application does not use CMA, you should take care of it somehow

> >(e.g. use CMA).
> 
> Sorry, what's CMA - Connection Manager ? In any case I don't know 
> about using of CMA if the application uses MVAPICH for parallelization

> ...
> 

Yes, I was referring to the Connection Manager. Since MVAPICH does not
use CMA for setting the connections, rate control will not be
configured. 

I wander what do you expect to get from this configuration? Running MPI
application with some links slower then the other might end up working
as fast as the slowest link (unless your application is not symmetric,
and some links are used more then the other).

If you need any help configuring the connections, please let me know.

Regards,
Gil Bloch

> Yours
> Mikhail
> 
> 
> 
> 
> 
> >> Yours
> >> Mikhail Kuzminsky
> >> Computer Assistance to Chemical Research Center,
> >> Zelinsky Institute of Organic Chemistry
> >> Moscow
> >> _______________________________________________
> >> Beowulf mailing list, Beowulf at beowulf.org
> >> To change your subscription (digest mode or unsubscribe) visit 
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> >>
> >
> >
> >_______________________________________________
> >Beowulf mailing list, Beowulf at beowulf.org
> >To change your subscription (digest mode or unsubscribe) visit 
> >http://www.beowulf.org/mailman/listinfo/beowulf