PCI-64: how to find

[Patrick Geoffray]

Hi Mark,

Mark Hahn wrote:
>>>     Is there (objective) comparison chart (bandwidth + latency) of
>>>Gigabit ethernet vs Myrinet vs Dolphin ?
>>>
>>It's impossible to get an objective comparaison, there is too many 
>>
> 
> perhaps to get a complete and comprehensive picture.  but I don't see
> why some simple but useful metrics can't be had.  for instance,
> suppose I just want to know the latency for a zero-sized packet between
> just two machines.  I don't see why CPU/PCI/etc would make much of a 
> difference here.  sure, it's not enough of a picture to start ordering
> hardware, but it's a start (and a fundamental number).

It makes a big difference for bandwidth: PCI is the bottleneck and PCI 
chipset are all differents, I have seen variation from 45 MB/s (VIA 
32/33) to 450 MB/s (ServerWorks 64/66) doing DMA benchmarks. The 
bandwidth for SCI seems to be related to the CPU (PIO) and also PCI 
(speed, write-combining). For GigE, the PCI is a boulevard.

For latency, it's less important in absolute values but it still a large 
percentage: some PCI have horrible arbitration, write-combining is 
sometimes buggy, you divide the PCI overhead by 2 with a 66 MHz bus, PIO 
   depends of CPU clock. When you add it up, the amplitude can be 
several us. Not negligeable when what you measure is < 10 us.

If you want numbers out-of-the-blue, Myrinet/GM is 6.5 us for 0-sized 
packet on Serverworks. If you use BIP, latency is 4 us. On Sun's boxes, 
latency with GM is ~12 (PCI sucks). With a 32/33 PCI, latency is 8 us. 
Using IP, latency depends linearly of the CPU clock and the way you 
measure it...
See, it's quite volatile, and I am not even thinking about bandwidth.


The trap with such numbers is that people wants to reproduce them, and
there is always something different that change the context. You would
be surprised of the impact of temperature on sub 10 us latencies for 
example.

Patrick

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      patrick at myri.com
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------