[Beowulf] IB Cluster Configuration Options
hahn at physics.mcmaster.ca
Fri Jul 29 08:08:43 PDT 2005
> As a means to reduce our cost the suggestion was raised to cascade 4 9024
> switches, or reducing the node count, Opteron clock rate, and memory. Just
> wondering how severe the hop penalty might be in the case of cascading
IB switch vendors claim quite low per-hop latencies. of course, the real
problem is that the base of a non-fat switching tree is highly contended.
(the voltaire 9024 spec says 140 ns, for instance).
the question is: how much are you willing to pay for good
bisection-bandwidth? personally, I cannot imagine buying IB except for
extremely bandwidth-intensive apps, in which case, you probably want
a non-blocking (full bisection) switching network. the 9024 doesn't
seem to be offered with all 12x ports, which is kind of odd, since
a classic fat tree would be most natural with 8 12x ports at the top.
but a set of 5 9024-12 switches would get you a pretty well-connected
set of 60 nodes. a hypercube with express links would use 8 switches
and have great bandwidth and at most 3 switches on any path (I think!).
maybe you should consider alternatives (Myri 10G seems extremely attractive,
regards, mark hahn.
More information about the Beowulf