[Beowulf] Questions about upgrading InfiniBand

[Prentice Bisbal]

On 04/18/2012 03:42 PM, Mark Hahn wrote:
>> Aggregation spine? Can you tell me more about that? Can you give me a
>> part/model number?
>
> spine is just the term for the trunk of a fat tree.  usually the
> per-rack switches are called leaves since if nothing else, they may
> not be at the top of the rack, or there may be more than one per rack...

Ahh... gotcha. In the previous e-mail, it sounded like a special line
card or something from the context. Tripped up by terminology.

>
> the good thing about the leaf/spine approach is that it's modular,
> and possibly less vendor-locked-in. (not that IB is really
> multi-vendor anyway).
>
> cable-wise, I'm not sure leaf-spine really wins, since you can think
> of a chassis switch as a leaf-spine with FR4 rather than CX4.  AFAIKT,
> the same radix-36 switch is used in each.
>
> spine/leave can be distributed so that there's no one place where you
> get too many cables.

My cluster is only 3 racks, with the head-node and IB switch in the
middle rack, so the cable don't have too far to go, so switching to
top-of-rack switches isn't that big of a deal for me. Of course, my
cluster might expand as part of this upgrade.

>
> "less than fully fat" fabrics seem to be pretty common when taking the
> modular approach.  for instance, if a 36x switch is split into 24 down
> (node) and 12 up, you can put two back-to-back, or have three going
> into a single spine switch, or more going into multiple
> spines.  you could even have some racks with more spineward links.
> and in your case, you could vary the number of links going to your
> existing chassis switch (though it probably shouldn't be the spine
> since all its links are slower...)
>
> -mark
>