Channel bonding GbE (Tigon3)

[In-Su Yoon]

Dear Steffen,

I read follwing your message.

> Dear list people, 
> 
> Lately I've been testing out a couple of Dell PowerEdge 2650 machines. 
> These babies have dual onboard BCM95701A10 NICs (Tigon3 chip) mounted 
> in the same PCI-X 133MHz 64 bit bus. 
> 
> Since they have dual onboard GbE, I've been trying to channel bond them 
> using just two crossover cables between two machines. The results I'm 
> seeing is at the first glance very strange. What I see is that the 
> performance when bonded (round robin) is about _half_ (and sometimes even 
> less) compared to just using a single interface. Here are some netpipe-2.4 
> results : 
> 
> 64k message size, single interface 
> 1: 65536 bytes 190 times --> 760.54 Mbps in 0.000657 sec 
> 
> 256k message size, single interface 
> 1: 262144 bytes 53 times --> 855.04 Mbps in 0.002339 sec 
> 
> 64 message size, both interfaces (using round robin) 
> 1: 65536 bytes 65 times --> 257.06 Mbps in 0.001945 sec 
> 
> 256k message size, both interfaces (using round robin) 
> 1: 262144 bytes 25 times --> 376.01 Mbps in 0.005319 sec 
> 
> Looking at the output of netstat -s after a testrun with 256k message 
> size, I see some differences (main items) : 
> 
> Single interface : 
> Tcp: 
> 0 segments retransmited 
> 
> TcpExt: 
> 109616 packets directly queued to recvmsg prequeue. 
> 52249581 packets directly received from backlog 
> 125694404 packets directly received from prequeue 
> 78 packets header predicted 
> 124999 packets header predicted and directly queued to user 
> TCPPureAcks: 93 
> TCPHPAcks: 22981 
> 
> 
> Bonded interfaces : 
> Tcp: 
> 234 segments retransmited 
> 
> TcpExt: 
> 1 delayed acks sent 
> Quick ack mode was activated 234 times 
> 67087 packets directly queued to recvmsg prequeue. 
> 6058227 packets directly received from backlog 
> 13276665 packets directly received from prequeue 
> 6232 packets header predicted 
> 4625 packets header predicted and directly queued to user 
> TCPPureAcks: 25708 
> TCPHPAcks: 4456 
> 
> 
> The biggest difference as far as I can see is the 'packtes header 
> predicted', 'packets header predicted and directly queued to user', 
> 'TCPPureAcks' and TCPHPAcks. 
> 
> I have an idea that this happens because the packets are comming out of 
> order into the receiving node (i.e the bonding device is alternating 
> between each interface when sending, and when the receiving node gets the 
> packets it is possible that the first interface get packets number 0, 2, 
> 4 and 6 in one interrupt and queues it to the network stack before packet 
> 1, 3, 5 is handled on the other interface). 
> 
> If this is the case, any ideas how to fix this... 
> 
> I would really love to get 2Gbit/sec on these machines.... 
> 
> 
> PS 
> 
> I've also seen this feature on the Intel GbE cards (e1000), but these 
> drivers has a parameter named RxIntDelay which can be set to 0 to get 
> interrupt for each packet. Is this possible with the tg3 driver too ? 
> 
> DS 
> 
> Regards,

I have also Intel Gigabit Ethernet cards (e1000).
I performed experiments with bonded e1000 Gigabit Ethernet cards.
The channel bonding performance was worse than a single Gigabit Ethernet performance.
The netstat -s results are as follows.

<Single interface>
15139284 segments received
15709077 segments send out
3740366 packets directly queued to recvmsg prequeue.
2981867090 packets directly received from prequeue
2025774 packets header predicted
10455151 packets header predicted and directly queued to user
TCPPureAcks: 1417
TCPHPAcks: 3000128

<bonded interface>
13208207 segments received
13035163 segments send out
255 segments retransmited
57 delayed acks sent
Quick ack mode was activated 20902 times
5036204 packets directly queued to recvmsg prequeue.
2817975024 packets directly received from prequeue
975822 packets header predicted
7253442 packets header predicted and directly queued to user
TCPPureAcks: 570464
TCPHPAcks: 3756814
TCPSackRecovery: 111
TCPFACKReorder: 1
TCPSACKReorder: 9
TCPFullUndo: 12
TCPPartialUndo: 35
TCPDSACKUndo: 23
TCPFastRetrans: 177
TCPForwardRetrans: 78
TCPDSACKOldSent: 20902
TCPDSACKOfoSent: 698
TCPDSACKRecv: 20794
TCPDSACKOfoRecv: 804

The bonded interface has more 'TCPPureAck' than single interface.
What is a difference between 'TCPPureAcks' and 'TCPHPAcks'?
Is the reordering cost of receiver a bottleneck of bonded interface?
I also performed the same experiment using eepro100 Fast Ethernet cards.
The results showed that the bonded FE cards has worse performance than a single FE card.

Is there any report that a bonded interface (using 100Mbps or 1Gbps) shows better performance than a single interface?

Thanks you.