[Beowulf] 802.3ad: Should it behave in this way??

[Andrei Maslennikov]

Hi all,

we are trying to set up a link aggregation on our server with two GigE
NICs Intel e1000 bonded together either with the bonding driver which
comes with the RHEL3 (kernel 2.4.21-27.0.2smp), or with the Intel IANS
teaming solution. This figure represents our layout:


          Client-1   Client-2   Client-3   Client-4   
             |         |          |          |
             |e1000    |e1000     |e1000     |e1000
         +-------------------------------------------+
         |       CISCO CATALYST 3570, 802.3ad        |
         |               - OR -                      |
         |            DELL 5324, 802.3ad             |
         +-------------------------------------------+
                    e1000-A |   | e1000-B
                            |   |
                            |   |
                      SERVER (Xeon 2x3.4GHz)

We would like to make sure that when all 4 clients talk to the
server, both e1000-A and e1000-B NICs operate at full speed and
are evenly serving 2 clients each at a time.

To test this configuration, we have used 4 ttcp streams in 2 ways:

  - four senders started on server, four clients receiving
  - four senders started on four clients, server receiving
   
In the first case (packets go *out* of the server), everything 
works as one would expect. All the streams end up at the
same time, and the aggregate throughput of all streams is
in excess of 200 MBytes/sec.

In the second case, one stream is ending up long before the
three others, with the speed of 100+ MBytes/sec. And other
3 end all at the same time, again with the aggregate speed of 
100+ Mbytes/sec.

I.e. instead of observing 2 streams over e1000-A and 2 other 
streams over e1000-B we see 1 stream over e1000-A and 3 others
over e1000-B.  

As the result does not depend on the type of switch in use
(CISCO or DELL), we assume that 802.3ad (which was originally
designed for switch trunking interconnects) simply may not
be a good solution for our case. 

We know that there are other intelligent switches around (starting
with CISCO 6500 series) which may have a better balancing, but they 
are a lot more expensive. Or one could use some kind of a 
stand-alone balancer... 

Are there any other people on the list who tried to configure
a similar layout with 802.3ad, and succeded?  

Thanks ahead for any comment - Andrei, Marco, Miguel.