[Beowulf] TOE on Linux?

[Greg Lindahl]

On Mon, May 19, 2008 at 06:42:51PM -0400, Mark Hahn wrote:

> the RDMA meme always puzzles me.  easy to see why it's a seductive 
> idea, but I always run up against the problem of exactly how it's 
> supposed to be used - are there any moderately well-known applications
> that specifically use RDMA?

Yes and no. NWChem uses a library to give one-sided access global
memory, and has special code for many interconnects to use one-sided
hardware.  One of the InfiniPath whitepapers shows a pure MPI-1
implementation scaling better than a real one-sided implementation. Go
figure.

> afaik, RDMA is used by some MPI implementations;

... usually only for long messages; pretty much all non-Ethernet
hardware implements some kind of remote DMA for this purpose. For
short messages, it doesn't matter if you copy the data.

You do have to watch out for the terminology: some people use RDMA to
mean only one particular kind of remote DMA, and some people use RDMA
to refer to both messaging and RDMA. These are the same people who
breathlessly talk about the benefits of RDMA, and how any adaptor
which doesn't have their form of RDMA must be inferior.

-- greg