[Beowulf] RDMA NICs

[Mark Hahn]

> Would anyone on this list have pointers to
> which network cards on market support 
> RDMA (Remote Direct Memory Access)?

ammasso seems to have real products.  afaikt, you link with their 
RDMA-enabled MPI library and get O(15) microsecond latencies.
to me, it's hard to see why this would be worth writing home...

> Would anyone have hands on experience 
> with performance, usability, and cost aspects
> of this new RDMA technology?

they work, but it's very unclear where their natural niche is.
if you want high bandwidth, you don't want gigabit.
if you want low latency, you don't want gigabit, even RDMA-gigabit.

the truth is that reducing networking overhead is always a bit of a 
hard sell.  consider the appeal of avoiding context switches - 
it sounds great, right?  how much of that appeal is based on the 
(mistaken) impression that context switches are expensive?  

lower CPU overhead also sounds great, but it made a lot more sense
5-10 years ago when systems had .3 GB/s memory bandwidth (<5% of today).

afaikt, gigabit-RDMA folks are really pinning their hopes on 10 Gb.