[Beowulf] Re: [Bonding-devel] 802.3ad: Should it behave in this way??

[Jay Vosburgh]

Andrei Maslennikov <Andrei.Maslennikov at caspur.it> wrote:

[...]
>In the second case, one stream is ending up long before the
>three others, with the speed of 100+ MBytes/sec. And other
>3 end all at the same time, again with the aggregate speed of 
>100+ Mbytes/sec.
>
>I.e. instead of observing 2 streams over e1000-A and 2 other 
>streams over e1000-B we see 1 stream over e1000-A and 3 others
>over e1000-B.  
>
>As the result does not depend on the type of switch in use
>(CISCO or DELL), we assume that 802.3ad (which was originally
>designed for switch trunking interconnects) simply may not
>be a good solution for our case. 

	More likely the balance algorithm is simply seeing a collision.
The 802.3ad implementations I'm familiar with (linux bonding and Cisco
3550 switches) do selection based on a hash (an XOR) of the MAC address
(Cisco routers can also use the IP address as part of the hash).  The
Cisco lets you select between using the source or destination addresses,
but that's it.  I'm not familiar with any 802.3ad implementations that
do balancing according to traffic load.  There may be some; I'm just not
aware of any.

>We know that there are other intelligent switches around (starting
>with CISCO 6500 series) which may have a better balancing, but they 
>are a lot more expensive. Or one could use some kind of a 
>stand-alone balancer... 

	As far as I know (and I'm willing to be corrected here) Cisco
has two systems for 802.3ad balancing: hashing of the MAC address or
hashing of the IP addresses; the former is the only choice on switches,
the latter is an option on routers.  If the higher end gear has
additional (load related) options, I'm not aware of them.

>Are there any other people on the list who tried to configure
>a similar layout with 802.3ad, and succeded?  

	One obvious hacky test to see if this is your problem is to
adjust the low order byte of the MAC address of one of the "threesome"
up or down by small amounts and see if it's hash (and thus port
assignment) changes.

	This topic (suboptimal balance of traffic) comes up regularly on
bonding-devel; you might want to check the archives (available via
http://sourceforge.net/projects/bonding).

	Also, if you're not married to running 802.3ad, the bonding
balance-alb mode does slightly smarter balancing of traffic, and
requires no specific support from the switch.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar at us.ibm.com