a yet another stupid network topology

[Eugene Leitl]

I've been thinking about network topologies for physical simulations
(spatially distributed over nodes, only locally coupled). The
assumption is that few-port FastEthernet switches are much less
expensive per port and typically have a larger backbone bandwidth then
larger switches. Is this still correct?

The solution before (4 NICs/node, projection from above, S: switch, N: 
node; single elementary cell shown):

  |   |
 -N   N-
   \ /
    S
   / \
 -N   N-
  |   |

This one is scalable, but it is rarely possible (due to simple budget
reasons) to go beyond 64 nodes, or so. Hence: the next stupid idea: we
have only 64 nodes, 3 NICs/node and 12 16-port
switches. (alternatively, same number of nodes, 4 NICs/node and 16
16-port switches). View the nodes as arranged on a 4x4x4 cubic
lattice. Think the switches arranged thusly (projection from above):

N N N N
   S
N N N N
 S   S
N N N N
   S
N N N N

We use each 16-port switch to crosslink the individual 2x2x2 cubes
along the larger cube axis. (In the larger 16 switch solution,
switches also crosslink the 2x2x2 cubes along the larger cube axis).

For all this to make sense, we need

1) 16-port switches are much cheaper/port than 32-port switches and
   larger 

2) above cheap 16-port switches can provide full 200 MBps*16 backbone 
   bandwidth

3) we can feed 3 (or even 4) FastEthernet NICs in a node

If each of the switches have 1 GBps uplink port, we obviously can
connect the whole system to a user pool without apparent bottleneks.

I would welcome any comments on this scheme. (yea, nay, wtf, etc.).

TIA,

-- Eugene Leitl