Channel bonding Gigabit Ethernet

[Jakob Oestergaard]

On Mon, Nov 25, 2002 at 07:18:40PM +0900, In-Su Yoon wrote:
> Hi, Glad to meet you.
> I have an 8-node cluster and 3Com SuperStack Switch 4900 which supports Link Aggregation.
> Each node has 512 MB SDRAM (133 Mhz) and Intel Server Board SDS2 with dual Pentium III 1.4 GHz.
> Also It has two Intel 82544 (known as e1000) Gigabit Ethernet cards which use 64 bit/66 Mhz PCI bus.
> 
> I performed some experiments with NetPIPE benchmark under Linux kernel 2.4.19.
> When I use one Gigabit Ethernet, the maximum bandwidth is approximately 863 Mbps.
> But, When I use the channel bonding for two Gigabit Ethernets, the maximum bandwidth is approximately 650 Mbps.

This issue was raised on the linux-kernel mailing-list some time ago, if
memory serves...

The problem is, that the linux kernel bonding driver will round-robin
between the interfaces as packets are transmitted. This will allow
packets in a TCP stream to arrive out-of-order on the receiving node, on
which the TCP stack will then have to go thru some non-trivial work to
actually re-order the packets again.

You probably see high CPU utilization on the receiving node - is that
correct?

The way to fix it, would be to let the bonding driver select which NIC
to transmit on, based on some hash of the packet headers. This would
then not give you any benefit on a single TCP stream, but it would allow
multiple concurrent TCP streams to exceed the one-NIC bottleneck.

AFAIK, this has not been done. I haven't seen any patches for it - but
again, google is your friend  :)

For one single TCP stream, I think the answer is that you are pretty
much stuck with single-NIC performance. And possibly better of without
bonding at all (on GBit cards - on FE cards modern processors seem to
cope fairly well with the re-ordering on receive).

-- 
................................................................
:   jakob at unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: