[Beowulf] TOE on Linux?

[Mark Hahn]

> application environments.  Contrary to the detractors of the technologies 
> comments, the TOE/RDMA card *did* provide fairly significant performance 
> delta for real apps running MPI over gigabit ethernet.

well, let's not conflate these things: TOE and RDMA are not intrinsically
related...

> I won't talk on the business side of them.  We did see 4x better (wallclock) 
> time on customers StarCD calculations being run over the TOE/RDMA engine than 
> over the pure gigabit path.

do you have a sense for whether this was due to RDMA giving you lower latency
by bypassing the stack (either host or the nic-toe stack)?  I don't think
I've ever heard of TOE itself providing any latency benefit.

also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
rather something like GAMMA or OpenMX.  I don't suppose you had a chance 
to test those, did you?

> Customer was pleased until they realized that 
> the design of this card effectively killed their maximum memory size (3.1 GB 
> vs the 4 GB installed).  This was the iWarp bit.

I'm guessing that techniques from the IB world would ameliorate this...