[Beowulf] Re: TOE on Linux

[Greg Lindahl]

Joe Landman wrote:

> Contrary to the detractors of the technologies comments, the
> TOE/RDMA card *did* provide fairly significant performance delta for
> real apps running MPI over gigabit ethernet.

As a detractor of TOEs, I should point out that one data point does
not prove that it's common that apps get a benefit.

I'd be willing to bet that this app was doing extremely large
transfers, and maybe even managed to get more concurrency with the
TOE... which could easily be a flaw in the MPI implementation's TCP
driver, a pretty common thing to be wrong. For example, LAM was always
much better than MPICH over TCP, and I wouldn't be surprised if
OpenMPI continues this superiority over MPICH-2.

The most interesting thing, to me, is that the various people selling
TOEs in the HPC arena publish almost no benchmarks. What's the message
rate and N1/2? The only N1/2 I've ever seen published was 100 kbytes.

(Obviously I'm not including Myricom in this bucket: they do publish
microbenchmarks.)

-- greg