[Beowulf] TOE on Linux?
hahn at mcmaster.ca
Wed May 14 14:20:25 PDT 2008
> application environments. Contrary to the detractors of the technologies
> comments, the TOE/RDMA card *did* provide fairly significant performance
> delta for real apps running MPI over gigabit ethernet.
well, let's not conflate these things: TOE and RDMA are not intrinsically
> I won't talk on the business side of them. We did see 4x better (wallclock)
> time on customers StarCD calculations being run over the TOE/RDMA engine than
> over the pure gigabit path.
do you have a sense for whether this was due to RDMA giving you lower latency
by bypassing the stack (either host or the nic-toe stack)? I don't think
I've ever heard of TOE itself providing any latency benefit.
also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
rather something like GAMMA or OpenMX. I don't suppose you had a chance
to test those, did you?
> Customer was pleased until they realized that
> the design of this card effectively killed their maximum memory size (3.1 GB
> vs the 4 GB installed). This was the iWarp bit.
I'm guessing that techniques from the IB world would ameliorate this...
More information about the Beowulf