[Beowulf] Pony: not yours.

[Mark Hahn]

>> that since we exist in 3d, our networks need to be a 3d lattice at scale.
>> fighting power (flops, comm) seems like a noble, *engineering* fight,
>> but fighting our existence's dimensionality is silly...
>
> We already exist in 3d - racks. We limit rack heights due to weight already
> (another item to add to your list of limits above).

weight (and 2d footprint) are purely linear, and not very interesting,
to me at least.  it's demonstrably possible (not hard) to do racks 
in 3d - and in fact doing so generally draws more attention to the 
macro-structure of the cluster anyway (including airflow, for instance.)

> I understand that you are arguing for 3d is silicon (or other substrate),
> but there will still be a practical limit.

I think I'm actually making the opposite argument.  that at least for 
inter-unit networking, the topology for very scaled clusters needs to be
a 2 or 3d mesh.  if you're building a cluster laid out in 2d, as most are,
then there are scaling problems with any non-2d topology.  I love higher-
order fabrics (fat trees, hypercubes, dragonfly, kautz, etc), but they 
have a sweet spot in their scale, and the whole point of exascale seems 
to be to exceed all sweet spots.

within a "unit" (chassis, rack, take your pick), it's practical to exceed
the "consensual dimensionality" - especially since you get to use different 
materials (say, a backplane with a complex pattern of connections
implementing a 5d torus, etc).  inside the package, the coming win seems 
to be simply 2.5d integration to minimize compute-memory distances,
though this is nothing new to the IBM MCM world.

> I saw some presentation a year or two ago that showed an exascale system as
> a sphere with the switch complex in the center. It looked very 60s-ish,
> proto-EPCOTish...

works for me!  though if you're going to do that, it would be awefully 
tempting to try to do free-space optics.  but again, once you exceed 
the sweetspot for your free-space-optical-sphere-cluster, you have to 
pack multiple spheres, and that's going to be some kind of mesh...

>> yes, I would like to demo your new blackhole-based interconnect! ;)
>
> Shouldn't it be wormhole-based? The blackhole-based one will just destroy
> the data, no?

I knew I should have just said "singularity" there!