[Beowulf] How to configure a cluster network

[Nifty niftyompi Mitch]

On Thu, Jul 24, 2008 at 09:42:57AM -0700, Kilian CAVALOTTI wrote:
> On Thursday 24 July 2008 05:42:22 am andrew holway wrote:
> > To give a half bisectional bandwidth the best approach is to set up
> > two as core switches and the other 4 as edge switches.
> >
> > Each edge switch will have four connections to each core switch
> > leaving 16 node connections on each edge switch.
> >
> > Should provide a 64 port network.
> 
> I'm also curious to know if there's a general formula to determine the 
> required number of IB switches (given their ports count) to create a 
> full (or half) bisectional network capable of interconnecting say N 
> leaf nodes, and especially, if there's a way to deterministically infer 
> the manner to (inter)connect them.
> 
> I've seen numerous examples involving small amounts of nodes and 
> switches, but I can't figure a way to scale those examples to larger 
> networks.
> 
> Any pointers?

Pointers yes... clear answers not sure.

   http://www.infinibandta.org/home

Since 99% of all the IB switch silicon is from Mellanox today give the 
Mellanox web site a big look.   Lots of vendors build switches with
Mellanox silicon... Cisco and QLogic come to mind.

   http://www.cisco.com/en/US/prod/collateral/ps6418/ps6419/ps6421/prod_white_paper0900aecd8043ba1d.html

Your most cost effective solution will be a large port count switch.
Most are not 'ideal' but they are close to ideal and cost effective.

At the bottom of all this is cross-bar technology (KeyHint=cross-bar).
Some good research was done at Stanford on this.  Plug "Dan Lenoski
crossbar" into your favorite search engine.  Dan Lenowski  and others at
Stanford did some good work that resulted in the ccNUMA machines at SGI.
The SGI ccNUMA memory subsystem was built on cross-bar switches and
modest to large Orign systems had X-sectional bandwidth setup issues.
There is also a lot of telco research and work on this.

Next some attention needs to be given to the subnet manager as it sets up the 
maps that the devices use to build a fabric.

Expect to start in 2D space then to N space when building switched fabrics.
It pays to play with some hot glue bamboo skewers and yarn for the 2D, 3D and
4D(hypercube) space...

The 2D, 3D, 4D,.... ND meshes are in part why this can get hard.



-- 
	T o m  M i t c h e l l 
	Looking for a place to hang my hat.