Beowulf: A theorical approach

[Jakob Østergaard]

On Thu, 22 Jun 2000, Greg Lindahl wrote:

> > Just as a matter of curiosity -- Once upon a time some two or three
> > years ago I suggested on the list that a development company consider
> > building a network communications device that plugged into the second
> > CPU slot of a dual CPU board.
> 
> It's hard to build something that plugs into an Intel-designed CPU bus.
> Scali tried it and it didn't work so hot.

At least on Intel hardware, SMP systems have cache coherency in the hardware,
and that's _expensive_ wrt. inter-cpu communication.  There's write snooping
and all sorts of the strangest things happening.  I cannot imagine how this
could work in any way over a network with any reasonable performance. Even
given infinite bandwidth, I guess the latency from ten meters of copper wiring
would kill performance.  (any sub-c interconnect would and tunneling is a few
years away I guess  ;)

> 
> You could do it for the Alpha/AMD cpu bus, but it's a high speed design, and
> the estimates I got (I was seriously considering this) was 12 months and $2
> million.

I don't know the EV6 architecture.  But do you think cache coherency could
be done over reasonably over a network among several CPUs ??

Cache coherency is evil as h*ll to implement. ccNUMA machines even rely on
kernel support, they can't build it in hardware alone  (at least that's how
it's done on the Origin series AFAIK).  Linux is getting NUMA support in the
VM,  do you think this could be used with the special hardware you mention to
actually build a ccNUMA system of a cluster ?

> 
> In 12 months, you'll be able to get either PCI-X machines or Infiniband
> machines. PCI-X has split transactions, and Chuck Seitz, who knows more
> about networking than all of us combined, says that PCI-X will get rid of
> most of the PCI latency.
> 
> So, sit back, pop open a cold one, and wait.
> 

I'm skeptical, but I'll take your advice here   :)

-- 
................................................................
: jakob at ostenfeld.dtu.dk  : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: