[Beowulf] Channel bonding, again

[Henning Fehrmann]

Hello Greg,

On Thu, Oct 11, 2007 at 03:13:16PM -0700, Greg Lindahl wrote:
> I'm thinking about using "balance-alb" channel bonding on a
> medium-to-large Linux cluster; does anyone have experience with this?
> It seems that it might generate a lot of arp replies if a switch
> fails.
> 

We did some experiments with channel bonding.

The bond device inherits the MAC-address of the first interface, meaning, it comes with a single MAC- and a single IP-address.  The bonding mode determines the load-balance of the transition.

On the other hand, one needs to trunk (HP calls it trunking) the ports on the switches. 
We tried it on HP and Cisco switches. 
The switch collects the packages of the trunked ports and redistributes them according to a level 2 or level 3 hash policy.
Here starts the problem: the packages are coming with the same IP- and MAC-address. The switch gets confused and, furthermore, the switch wants to do the work which is already done by the node.

One of our studends found a workaround:

E.g., you want to bond 4 interfaces. Put each interface in a distinct VLAN (VLAN 1-4).  
The VLAN tag increases the Ethernets frame by 4 byte.  Subsequently, you bond the four interfaces.  The ports on the switch can be also configured to be a member of the coresponding VLAN. 

The load-balancing is done by the node, the switch transmits the packages according to the VLAN id and does not care about trunking.

In order to establish communication of a node with 4 network cards and nodes with 2 ore 1 networkcards you have to create on each node 4 virtual interfaces, put each one into a distinct VLAN and bond them together.

The ports on the switch can also be configured to be member of multiple VLANs.

Cheers
Henning