Beowulf: A theorical approach

[Robert G. Brown]

On Thu, 22 Jun 2000, James Cownie wrote:

> 
> >  The problem right now really isn't in link speeds (though better
> > link speeds are good), its in how close/far the network interface is
> > from the CPU.  COTS HW doesn't place a high value on direct access
> > to IO devices - there is a higher value on a standardized bus
> > interface to allow different system components to be integrated and
> > updated independently.  A "supercomputer" can have the network
> > engineered directly into the node architecture.  This is a huge
> > advantage.  Luckily, this advantage has the most effect in only some
> > programs.
> 
> If Infiniband does all that it is supposed to do, then it will rapidly
> become the network of choice, since it _does_ have support for direct
> (user-space) access to the comms, and has some nifty switches.
> 
> Of course in the short term it will be limited by the CPU side
> interfaces being PCI, but that's only the same limitation as
> for Quadrics, Myrinet, SCI and so on. 

Just as a matter of curiosity -- Once upon a time some two or three
years ago I suggested on the list that a development company consider
building a network communications device that plugged into the second
CPU slot of a dual CPU board.  After all, one would guess that a
cleverly designed controller (which might even have a full CPU on it and
be in a sense a "network-specialized single board computer") would then
be able to access and be accessible to the entire CPU/memory subsystem
at full memory speeds and latencies (that is, sub-microsecond latencies
and 100-200 MBytes (not bits) per second at least onto the device
itself, where it would presumably bottleneck through the actual
communications channel and switch).

I would >>think<< that such a design, with just the right firmware on
the "network communications processor" plugged into the slot and a
kernel module or two, could be made to provide de facto CC-NUMA
pseudo-smp operation.  After all, even on a dual system cache coherence
is already addressed, all that is needed in addition is an algorithm for
extending that across the attached network.  I'd bet that one could
design such a device/system to run with existing dual (Intel) CPU MoBo
chipsets and -- provided that my repeated (and hence documented)
description of the idea on this list suffice to prevent somebody else
from being able to patent it -- even qualify as a COTS technology.

Dual CPU motherboards are cheap - a few tens of dollars more than a
single.  The chipsets and firmware are well documented.  CPU's can
presumably generate interrupts and the like.  I'd expect that the
engineering would be straightforward and a marketable new device (or
marketable variant of an existing device that is bottlenecked at the PCI
bus) could be developed quickly and sold relatively inexpensively.  If
the idea is successful implemented in this way, it might even spawn a
specialized interface on motherboards derived from the existing second
CPU slot but even better suited toward commodity supercomputer assembly.

Is anybody even THINKING of doing this?  Yet?  Or is there something I'm
ignoring that makes this impossible or hideously expensive?

   rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu