networking options

[Peter Kjellström]

On Tue, 24 Sep 2002, Richard Walsh wrote:

> 
> All,
> 
> Whoops ... down is not alway bad ... in my prior note
> I meant gets worse (grows larger quickly).

Message read loud and clear :-) ...but I do disagree. SCI latency is not
as dependent on the size of the cluster as one might think considering
it's ring topology. Here are a few numbers (all application level,
mpi_send/recv):

two nodes same ring, ringsize 4: 4.5us
two nodes same ring, ringsize 8: 4.7us
two nodes one dim. change, ringsize 4, 8: 5.2us

In plain english, a ring size doubling from 4 to 8 nodes per ring only
gives you 0.2us extra latency. Changing dimension (the two nodes not on
the same ring) is a little more expensive at < 0.5us application level.

Thus, running a 512 node SCI 3d torus (8x8x8 ring dimension) would give
you latency in the neighbourhood of the 2nd example (ringsize 8) if you
use nodes on the same ring. Worst case (almost normal case) you would have
to change dimension two times adding an extra us at most. (guessing since
I don't have a 512 node cluster handy for testing)

Cheers,
 Peter

PS.
The numbers are from a small (8x4) athlon cluster with crappy PCI
(via). With better chipset you can get down to around 3.4us application
level.

>                              |
>                              |
>                              V
> >SCI's switchless latency drops off steeply as you scale to largish 
> >(> 128 nodes) sizes ...  you can of course use SCI switches to reduce 
> 
> rbw
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
------------------------------------------------------------
  Peter Kjellström               | E-mail: cap at nsc.liu.se
  National Supercomputer Centre  | Office: +46(0)13 281492
  Linköping University           | Fax   : +46(0)13 282535
  SE-581 83 Linköping            | 
  Sweden                         | http://www.nsc.liu.se