[Beowulf] Cheap SDR IB

[Gilad Shainer]

Dear Mark, 

> > Just in case you've missed the announcements:
> > http://www.clustermonkey.net//content/view/222/1/
> 
> I'm always happy about new levels pricing agression, but I'm 
> a bit puzzled about for what kind of workloads this will matter.
> 
> whenever I ask about IB bandwidth, people always point 
> fingers at weather codes, which apparently are fond of doing 
> the transpose in multi-dimension FFT's using all-to-all.  
> while convenient, this seems a bit silly, since transpose is 
> O(N) communications, not O(N^2).
> 
> higher bandwidth/node also makes sense if you're configuring 
> fairly fat nodes (many cores, probably also lots of ram).  
> but if you do that, you also amortize the networking, so a 
> cheaper IB setup matters less.
> 
> perhaps there are some extremely file-IO intensive workloads 
> that can sustain ~1 GB/s, but I'd expect them to require some 
> hefty fileserving hardware, which would also hide the IB cost.

 
Bandwidth is one aspect of an interconnect. Other aspects are 
of course latency, CPU overhead etc etc. Some application will 
benefit from the latency, and some form bandwidth and some
from a combination of all. Weather codes will show great benefits,
and also CFD, rendering, bio codes (NAMD etc), Monte Carlo simulations
and even mathematica users. Just a partial list. 



> IB for gaming?  I have one ratio: 1e-1/3e-6.  that's human 
> reaction time versus IB latency.
> 

Oh yes... I guess you did not play for a long time. Did you? Talk
with someone who suffer from lagging and you will get the story, even
When he has a great video card. It's the network and the CPU overhead
that are the cause of this issue 


> also, I think it's a bit disingenous to use 10G Chelsio TOE 
> to compare, rather than 10G Myri which is cheaper and faster.  also:
> http://www.chelsio.com/sandia_benchmark_tech.html
> 

I really don't want to set fire here but funny that you point to such
data
as a proof point. If you want a serious discussion on Chelsio "testing"
we
can have it, probably in a different mail thread.



> finally, how the heck do you make Gb as slow as 120 us?
> 
> -Mark "not actually anti-IB" Hahn.
 
Good one. 


> PS: does anyone have first-hand experience with ConnectX performance?

I do, but you probably want to hear from Myricom ... :-)