The FNN (Flat Neighborhood Network) paradox

[Robert G. Brown]

On Wed, 21 Feb 2001, Miroslaw Gawenda wrote:

> Hi
>
> I just plan to make beowulf with 100M nic with network architecture
> described on the http://aggregate.org/FNN.
> But I am in trouble because I found so this is without sense for me :(
> I was count the prices for architectures prepared from 8/16/24/and
> greater switches.
> I was count the total network cost ie : cost of switches, nics,
> patchcoords and patchpanels.
> My results say : ONLY 1 switch giving good price !

The FNN trick is really intended for the HIGH end of switch-technology
space, not the low end.  Let us pretend for a moment that it is five
years ago and it costs $40,000 for a 100BT switch with at least 32
switched ports and that switch one gets for that money doesn't have
sufficient bisection bandwidth to allow all ports to be in use at once.
24 port switches, however, only cost $6000 and do have sufficient
bisection bandwidth.  (The difference is that 32 port switches in those
days came in big chassis with many expensive blades and an expensive
backplane, but 24 ports might come in a single -- still expensive --
box.  A 32 node beowulf with adequate bisection bandwidth and symmetric
connections to every other node can be built with only 3 24 port
switches for (say) $18K -- a big win!

Today, switches are dirt cheap up to 24 ports, and not horribly
expensive out to 80 ports.  As you have so aptly noted, it is really
dumb to build a 16 node beowulf out of lots of 8 port switches when a 16
port switch is cheap, cheap, cheap.  In fact, for the trick to work, the
S switches required to connect N nodes with up to P NICs per node must
cost less than a single switch with N ports, and that condition is
likely not satisfied these days for any mundane number of nodes (again,
as you noted). It thus is a bit more difficult to make the trick work,
but let's try.  We'll try to design a beowulf with (say) 512 nodes.  For
our imaginary purposes, it is absolutely essential that all nodes be
interconnected symmetrically, unblocked, with adequate bisection
bandwidth.

Looking around, the largest number of ports I can (easily -- this search
was not exhaustive) find in a layer 2 switch is 336 (one of the big
Foundry switches).  At a guess, the 336 port switch costs a gazillion
dollars.  I'm too lazy to wait for a quote so we'll have to make up a
number.  Pretend that it costs $100,000.  It doesn't matter -- the first
thing we note is that you cannot get to 512 symmetric connections for
any money with a 336 port switch.  So the cost for 512 symmetric ports
is infinity.

Now the FNN trick makes sense.  We can get to 512 nodes with 3 of these
336 port switches, no bottlenecks or uplinks anywhere.  It is expensive,
sure, at $300000 plus, but less expensive than infinity and our premise
is that it is worth it at any price.

It can also make sense if you have a bunch of leftover 24 port switches
(not an impossibility these days as wiring closets get redone in an
enterprise to use a single big switch and old switches are surplused)
and need to interconnect, say, 40 nodes and prefer no uplink.  A 40 port
switch isn't particularly expensive, but 4 24 port switches are (by
hypothesis) free.

To conclude, the trick isn't for anybody, but it will make sense for
some.  The FNN site is very cool, working out the nontrivial connection
topology problem for you (which likely has other applications as well).

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu