[Beowulf] torus versus (fat) tree topologies

[Chris Sideroff]

Mark Hahn wrote:

> what do you have against switches?  *someone* has to do the switching,

   Nothing personally ;-) There is the cost issue especially when the 
clusters get larger and you need multiple switches (node + spine) 
swtiches to complete the fat tree.  I guess I had blindly assumed that 
switches, in general, were the cause of some the latency - therefore by 
removing this component would also remove that part of latency.  Your 
succeeding comment addresses my false intuition.

> computation".  compare this to something like myrinet, where switching is
> done by 16 or 32x crossbar units.  not only fewer hops, but think also of 

   I didn't know that these switches operate like that.  Do Quadrics and 
Infiniband switches perform similar functions?

> practical factors, I believe: latency and packaging.  bisection bandwidth
> seems to appeal to a lot of people, but I'm not sure why.  n-dim tori

   I have had the luxury of testing an SCI 2D torus cluster and found 
latency performance to be exceptional.  In fact this has been most 
limiting factor for the major application (Fluent) running on our 
cluster.  Upon performing some performance profiling I found Fluent 
scales signficantly better on it than the gig/e network currently 
implemented.  I have not had the oportunity to evaluate any other 
interconnect hardware so I can not comment on their performance.

   From the documentation on vendors websites I believe it will give 
similar improvements to Fluent scalability.  This is why I was asking 
the question - torus vs. (fat) tree?  Looking for reasons other than 
performance.

   This topic follows an earlier post I submitted to the list about 
choosing a high-speed interconnect.  The consensus was that I should 
first determine whether I even need one, which I have done, and my 
conclusion was that it will benefit our hardware/software combination 
greatly.  I have some results which I will post soon.

Thanks, Chris Sideroff