The FNN (Flat Neighborhood Network) paradox

[Martin Siegert]

On Wed, 21 Feb 2001, Jim Lux wrote:
<snip>
> > This discussion brings up an interesting question:
> >
> >
> > Say you had 2 8 port switches and wanted to interconnect, say, 8<=N<=4
> > processors with a pair of NICs in each.  What is the optimum arrangement for
> > channel bonding?  All processors send one NIC to one switch and the other to
> > the other?, or some sort of hybrid where you send both NICs to the same
> > switch. (obviously, one can scale this problem up... in today's market, the
> > correct answer would be to just buy a 16 port switch).  How well does
> 
On Wed, Feb 21, 2001 at 03:05:50PM -0500, Robert G. Brown wrote:
> I think that this is really the right answer though.  Just buy a bigger
> switch until you cannot afford anything any larger, and long before that
> time you'll likely have moved over into the regime of either GB ethernet
> or Myrinet or out in the esoteric region where the answer requires
> serious math to work out and is likely different for different kinds of
> network access.  I'd guess the >>optimum<< answer would be different for
> a parallel process with short range (one message to "nearest neighbors"
> with some topology), long range but symmetric (one message to N hosts)
> and long range buy asymmetric (N different messages to N hosts, and that
> the optimum answer starts to become very expensive regardless, so for
> big designs you really have to just work it out.

A word of caution:
Most (all?) switches will get horrendously confused, if you connect
two (or more) channel bonded NICs to the same switch.
Linux assigns the same MAC address to all channel bonded NICs (I believe
that by doing so Linux actually does not comply to the RFC on trunking,
hence even if the docs for a certain switch say that it supports trunking
it doesn't mean that that it supports Linux-type channel bonding; I'm
not really on safe ground here, please correct me, if I'm wrong).
Most (all?) switches cannot handle receiving the same MAC address from
different NICs. Hence if you plan on doing channel bonding buy as many
switches as you buy NICs/per box and connect each channel bonded NIC
in a box to a different switch. Even if you can configure your switch
(using VLANs, etc.) to accept identical MAC addresses from different NICs,
the setup must be a nightmare.

Cheers,
Martin

========================================================================
Martin Siegert
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: siegert at sfu.ca
Canada  V5A 1S6
========================================================================