[Beowulf] torus versus (fat) tree topologies

[federica.pisani at quadrics.com]

Hi Craig, Mark

just a few clarifications:

the hot chips paper data showed the latency on a 4096 nodes machine

the latency from pad to pad on the Elite chip is about 21 nanoseconds, to go from anywhere to anywhere you pass through 11 Elite switch chips (5 on the way up and 6 on the way down) thus 21*11=231.

also: in the Hot chips paper the cable delay included the PBC trace delay on the PCB and within the midplane, thus the figure quoted for the Elite cable delay is the time required for the head of the packet to be fully routed across an unblocked network, in particular this includes all the arbitrations at each stage of the network.

Finally, 1.38 microseconds is the MPI  0 byte ping pong latency through a single Elite that we recently measured on 2.2 GigaHertz, Numa enabled Tyan Opteron Nodes. 

Hope this helps, 

Federica

-----Original Message-----
From: Craig Tierney [mailto:ctierney at HPTI.com]
Sent: 17 November 2004 16:25
To: Mark Hahn
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] torus versus (fat) tree topologies


On Tue, 2004-11-16 at 17:07, Mark Hahn wrote:
> > Mmm ... from your 2003 Hot Chips presentation on Elan 4 I see 231 
> > nanos.  Which is right, or are we talking about two different things?
> 
> AFAICT, the 25ns figure is for an individual 8-port xbar chip,
> and a full-sized switch is three stages of these.  but 6*25!=231.
> I believe there's at least one quadrics doc that quotes 300ns for 
> the switch.  perhaps the 231 number is derived from average latency
> (since some ports are just one xbar away)?
> 
> also, isn't SGI's numalink network a dual fat-tree?  they're claiming
> 1.1 us latency these days (though again, that might be averaged over
> all possible paths...)

Is that shmem latency or MPI latency?  I think that the MPI 
latency is closer to 2 u/s.

Craig

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf