[Beowulf] TOE on Linux?
landman at scalableinformatics.com
Wed May 14 13:48:43 PDT 2008
Prentice Bisbal wrote:
> Prentice Bisbal wrote:
>> Does anyone know of any network cards/drivers that support TOE (TCP
>> Offload Engine) for Linux? A hardware vendor just told me that Linux
>> does not support the TOE features of *any* network card.
>> Given Linux's strong presence in HPC and the value of having TOE in a
>> cluster, I find that hard to believe.
> I should have googled before asking that question. I googled for "Linux
> TCP Offload Engine" and got an eyeful. Apparently TOE = Evil to Linux
Ok... worth noting that there are opinions and many people have them.
We have had experience deploying and using Ammasso RDMA/TOE cards in
real world application environments. Contrary to the detractors of the
technologies comments, the TOE/RDMA card *did* provide fairly
significant performance delta for real apps running MPI over gigabit
I won't talk on the business side of them. We did see 4x better
(wallclock) time on customers StarCD calculations being run over the
TOE/RDMA engine than over the pure gigabit path. Customer was pleased
until they realized that the design of this card effectively killed
their maximum memory size (3.1 GB vs the 4 GB installed). This was the
Ok, now that I have said that (and expect quite a few comments about how
it really doesn't work :( and the Linux /usr/bin/time command must have
been broken that day ... ), Chelsio and a number of others do have
TOE/RDMA cards. Their drivers are in the OFED stack. Be prepared to
pay quite a bit for these cards. Before you do, have a good hard look
at Infiniband with RDMA capability. Does your app really need TCP
offload, or simply fast path MPI processing? Myricom, Mellanox, and
others can provide this through the respective stacks.
Basically I am not arguing against TOE, just I don't see the need in
many cases today. This is not to say it doesn't work. It does. It
just costs a bit to get it working, and you have to do the cost-benefit
analysis to determine whether or not you really need it, especially
compared to other lower cost technologies.
Put another way, what problem are you trying to solve where you believe
TOE to be the right answer? There may be alternative answers that have
less pain associated with them.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Beowulf