NIC/driver/channel bonding tests
Martin Siegert
siegert at sfu.ca
Fri Nov 17 18:51:31 PST 2000
I've tested a bunch of ethernet card/driver combinations. Here are the
results. I hope they may be useful.
Ethernet Card and Driver Test
=============================
The following combinations of ethernet cards and drivers were tested under
Linux (RedHat 6.2; kernel 2.2.16):
Realtek RTL8139 / rtl8139.o (version 1.07)
LinkSys LNE100Tx / tulip.o (version 0.91g-ppc)
3Com 905B / 3c59x.o (high-performance variant poll-2.2.17-pre18.c;
see http://www.uow.edu.au/~andrewm/linux/#3c59x-2.2)
3Com 905B / 3c90x.o (version 1.0.0i)
Intel EtherExpressPro 100 / eepro100.o (version 1.09j-t)
3 RTL8139 / rtl8139.o channel bonded
3 3Com 905B / 3c59x.o channel bonded
All tests were done using cross-over cables.
First test: two identically configured PIII/600MHz
Netperf benchmark:
tcp test: ./netperf -H p600 -i 10,2 -I 99,5 -l 60
udp test: ./netperf -H p600 -i 10,2 -I 99,5 -l 60 -t UDP_STREAM -- -m 8192 -s 32768 -S 32768
(for the udp test the throughput at the receiving end is reported)
All numbers in Mbit/s.
RTL8139 LNE100Tx 3c59x 3c90x EEPro100 3xRTL8139 3x3c59x
--------------------------------------------------------------------------
tcp | 85.67 93.72 94.09 94.10 93.38 228.37 279.55
udp | 87.93 95.75 95.82 83.85 95.75 127.47 266.15
Second test: Pentium 166MHz and dual PII/400MHz
Netperf benchmark:
tcp test: ./netperf -H p166 -i 10,2 -I 99,5 -l 60
udp test: ./netperf -H p166 -i 10,2 -I 99,5 -l 60 -t UDP_STREAM -- -m 8192 -s 32768 -S 32768
tcp test: ./netperf -H p400 -i 10,2 -I 99,5 -l 60
RTL8139 LNE100Tx 3c59x 3c90x EEPro100
----------------------------------------------------------------------
2xPII/400 -> P166, tcp | 62.08 65.24 84.62 54.73 61.23
P166 -> 2xPII/400, tcp | 66.36 68.19 92.91 63.76 85.35
P166 -> 2xPII/400, udp | 88.76 85.33 95.40 58.95 95.76
Remarks
=======
1. For channel bonding use the module bonding.c from the 2.2.17 kernel even
if your are using a 2.2.16 kernel. Otherwise bad things (kernel oops)
happen. I found out about this the hard way:
"/etc/rc.d/init.d/network restart" will crash your machine so badly that
it won't even reboot (it hangs on unmounting /proc with "device busy").
Only power cycling the box will get it back to life. The problem is
with bonding module: "ifconfig bond0 down; rmmod bonding" will yield
the same result (oops).
The 2.2.17 module seems to fix this.
2. The 3c59x.c from the 2.2.17 kernel produced basically the same results
as the high-performance variant used in the tests above. Somehow these
tests don't seem to use the performance enhancing "polling" modes of
new the driver. Does anybody have more information on this?
The 3c59x.c from kernels < 2.2.17 have bugs and should not be used.
3. The 3Com 3c90x.c driver has given me nothing but problems, particularly
in connection with NFS. Under heavy NFS load the interface would simply
lock up and only "/etc/rc.d/init.d/network restart" would solve the
problem. As the tests above show the performance of the 3c90x driver
is much worse than that of the 3c59x driver.
4. Channel bonding with the 3c59x driver works without any problems: just
create /etc/sysconfig/network-scripts/ifcfg-bond0 and setup the
/etc/sysconfig/network-scripts/ifcfg-eth1, etc. files. That's all
what is required. With the RealTek cards/driver this fails, because
the MAC addresses are not copied correctly. The only way out is to
actually change the real MAC addresses of the ethernet cards by making
them all equal. Laurent Itti described on this list how to do this.
Channel bonding with the 3c90x driver fails for the similar reasons
(in that case not even ifconfig reports identical MAC addresses).
Also with the RealTek cards the connection would occasionally die
and I had to run "/etc/rc.d/init.d/network restart". This did not
happen with the 3C905B/3c59x cards/driver.
I only had two tulip and Intel cards, so I could not test channel
bonding with those cards.
Discussion
==========
1. The RealTek cards were far the worst in this test.
Heavy udp load freezes the machine. The channel bonded 3 Realtek cards
sometimes locked up the machine so severely that only a hard reboot
(on/off switch) would bring it back. They are cheap, but it seems that
you get what you paid for.
2. The results for the tulip, 3Com, Intel cards for PIII/600 -> PIII/600
do not differ significantly. However they differ in the tests from
the P166 to the dual PII/400. The significance of this test is the
following: the P166 does not have the cpu power to handle 100Mbit/s.
Hence, the transfer rates in this case are not limited by the highest
throughput a particular card/driver combination can handle, but by
the cpu. However, some of the cards are "smarter" than others as they
can off load some of the cpu tasks. Therefore, a higher throughput in
this test indicates a "smarter" card. This should be particularly
important when you channel bond the cards. If the cpu is 100% busy
with maintaining a high throughput, it is impossible to do computation
and communication in parallel. In this area the 3C905B/3c59x outperforms
all the other card/driver combinations.
3. Channel bonding three 3C905B using the 3c59x driver works very well: the
throughput is basically three times the throughput of a single card.
Channel bonding the RealTek cards is out of the question with the
RealTek cards: they show a horrendous packet loss under udp. This
would bring NFS to a grinding halt. Also the reliability is poor
(the uptime of our beowulf is currently 150 days; and that's only
because I had to upgrade the kernel: cf. Linux capabilities bug).
Summary
=======
I'm planning to upgrade my cluster to 3 x 100BaseT using 3C905B and the
3c59x driver :-)
[This was the main purpose of this exercise].
Martin Siegert
Academic Computing Services phone: (604) 291-4691
Simon Fraser University fax: (604) 291-4242
Burnaby, British Columbia email: siegert at sfu.ca
Canada V5A 1S6
More information about the Beowulf
mailing list