[Beowulf] TOE on Linux?
landman at scalableinformatics.com
Wed May 14 15:32:55 PDT 2008
Mark Hahn wrote:
>> application environments. Contrary to the detractors of the
>> technologies comments, the TOE/RDMA card *did* provide fairly
>> significant performance delta for real apps running MPI over gigabit
> well, let's not conflate these things: TOE and RDMA are not intrinsically
Agreed, but you had both on this card, and you couldn't turn one off ...
>> I won't talk on the business side of them. We did see 4x better
>> (wallclock) time on customers StarCD calculations being run over the
>> TOE/RDMA engine than over the pure gigabit path.
> do you have a sense for whether this was due to RDMA giving you lower
> by bypassing the stack (either host or the nic-toe stack)? I don't think
> I've ever heard of TOE itself providing any latency benefit.
Usually a detriment to latency, but as I remember, they set up a fast
path for TCP offload via an iWarp/RDMA engine. So it bypassed the stack.
> also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
> rather something like GAMMA or OpenMX. I don't suppose you had a chance
> to test those, did you?
Sadly no. We are playing with OpenMX now though, for a different code.
And Ammasso is long gone.
>> Customer was pleased until they realized that the design of this card
>> effectively killed their maximum memory size (3.1 GB vs the 4 GB
>> installed). This was the iWarp bit.
> I'm guessing that techniques from the IB world would ameliorate this...
I personally don't see the value in RDMA for gigabit (10 GbE yes), when
IB is ~$180/port (HCA + cable) on the client side and ~$50/port on the
switch side. The gigabit switch you need for the low latency bits are
around $100/port, and then the offload/RDMA NIC is > $500/port ....
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Beowulf