[Beowulf] split traffic to two interfaces for two "subnets"

[Yaroslav Halchenko]

> My quick and dirty test was to do NFS reads from memory on the server (to rule out disk contention) to multiple 
> clients.  So I did this:

> o On client 1, 'tar cO $DATA | cat > /dev/null' ~4GB of data from the
>   server (4GB being the amount of memory in the server) to /dev/null.
>   This caches the data in the server's RAM.  Do the same from client 2.

> o On clients 1 and 2, tar the same data to /dev/null on both clients
>   simultaneously.

Are you sure did step 1 on a client? that would fill up the client's
buffer (if its RAM < server RAM, then only partially) with content from
the server, thus during step 2 it will first fetch from the buffer and
read the rest from the server... 

The reason I am asking because I did more tests on bonding and had to
postpone till I get all the nodes on the new switch.

My discoveries:

  with mode 0 I couldn't reach good throughput. Tried NFS from the nodes
  - the same story and it looked weird -- 1 netperf stream would occupy
    all 1Gbps and then only at the end let 2nd netperf to work for the
    few milliseconds, so 2nd netperf was getting close to 1 Mbps at the
    end ;-)

    I've read up more about mode 0 from kernel docs:

balance-rr: This mode is the only mode that will permit a single
        TCP/IP connection to stripe traffic across multiple
        interfaces. It is therefore the only mode that will allow a
        single TCP/IP stream to utilize more than one interface's
        worth of throughput. 

    As I understood it: it does split traffic through both interfaces,
    but if there are 2 TCP/IP streams, then for some reason it didn't do
    any ballancing, so 1st stream occupied both, saturated switches
    buffer and only after those gets delivered to the destination
    node#1, it allowed the 2nd stream to get to node#2. Thus at the end
    total throughput was limited by a single downlink throughput.
    Is it something wrong with my kernel setup or this is how it has to
    be?

    I've tried _xor bonding. That beast worked wonderfully -- occupying
    both interfaces to their capacity whenever mac's XORed properly (so
    reaching around 220 MBps total), and ofcause all traffic went through
    the single interface if both destination MACs XORed to the same
    number...

    If I don't resolve the issue with _rr I am afraid I would need to
    tune MAC addressed of NICs in the nodes so I have even split between
    them for _xor balancing
    
-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20060517/d96d0c74/attachment.sig>