[Beowulf] Switching

[Mark Hahn]

> Usually you can't connect more than one cable between two switches.
> The switches will not bundle those to gain higher bandwidth. However

eh?  LACP (802.3ad) is pretty common these days, and typically trunks
up to 8 links.

> on a managed switch you might be able to create separate networks,
> then you can connect one cable for each network to your existing
> switch.

yuck!  there is some correlation between LACP support and managed switches,
but I suspect that's mainly because LACP makes no sense on small switches,
and most larger switches are managed...

> >   I'll most likely be looking to expand my current cluster, and the
> > number of new nodes purchased will be greater the capacity of the
> > current switch (a Cisco 6509 with a Supervisor Engine 1A).

egads, not exactly a switch that pushes the cost-effectiveness envelope ;)

> >   My plan is take the spill-over nodes (about 32) and place them on a
> > separate switch. Just a simple, cheap 48 port GigE switch. These nodes

the cisco supports 802.3ad so should do trunks to many other switches,
such as procurve, smc, dlink, netgear.  I've used LACP among procurve's 
and it seems to work fairly well.

> > would only be for workup so a high-bandwith/low-latency connection to
> > the rest of the nodes is not crucial.

2-4x trunk is probably what you want.  I don't believe low-latency would 
be a serious problem - after all, you're probably already out in the 
30-50 us range with gigabit, so what's an extra 10?

> >   Connecting the switches together is where my knowledge starts to fall
> > down. I do want some amount of bandwith between the two switches. Is it
> > possible to do something like simply connect the two switches using a
> > handful of GigE cables?

yep.  it is interesting that LACP is aggregation, not striping.  that is, 
a packet will go over one link (the link choice depends on destination IP 
or MAC, I think.)  this means that you can probably choose a specific set 
of addresses which will go at just 1x over an 8x LACP...  in fact, this is 
why I'd certainly prefer a 10G link (your cisco can do it, as can recent 
switches from most other vendors).

has anyone done realistic performance measures of LACP?

regards, mark hahn.