PCI-64: how to find
Patrick Geoffray
patrick at myri.com
Mon Feb 25 09:56:55 PST 2002
Hi Mark,
Mark Hahn wrote:
>>> Is there (objective) comparison chart (bandwidth + latency) of
>>>Gigabit ethernet vs Myrinet vs Dolphin ?
>>>
>>It's impossible to get an objective comparaison, there is too many
>>
>
> perhaps to get a complete and comprehensive picture. but I don't see
> why some simple but useful metrics can't be had. for instance,
> suppose I just want to know the latency for a zero-sized packet between
> just two machines. I don't see why CPU/PCI/etc would make much of a
> difference here. sure, it's not enough of a picture to start ordering
> hardware, but it's a start (and a fundamental number).
It makes a big difference for bandwidth: PCI is the bottleneck and PCI
chipset are all differents, I have seen variation from 45 MB/s (VIA
32/33) to 450 MB/s (ServerWorks 64/66) doing DMA benchmarks. The
bandwidth for SCI seems to be related to the CPU (PIO) and also PCI
(speed, write-combining). For GigE, the PCI is a boulevard.
For latency, it's less important in absolute values but it still a large
percentage: some PCI have horrible arbitration, write-combining is
sometimes buggy, you divide the PCI overhead by 2 with a 66 MHz bus, PIO
depends of CPU clock. When you add it up, the amplitude can be
several us. Not negligeable when what you measure is < 10 us.
If you want numbers out-of-the-blue, Myrinet/GM is 6.5 us for 0-sized
packet on Serverworks. If you use BIP, latency is 4 us. On Sun's boxes,
latency with GM is ~12 (PCI sucks). With a 32/33 PCI, latency is 8 us.
Using IP, latency depends linearly of the CPU clock and the way you
measure it...
See, it's quite volatile, and I am not even thinking about bandwidth.
The trap with such numbers is that people wants to reproduce them, and
there is always something different that change the context. You would
be surprised of the impact of temperature on sub 10 us latencies for
example.
Patrick
----------------------------------------------------------
| Patrick Geoffray, Ph.D. patrick at myri.com
| Myricom, Inc. http://www.myri.com
| Cell: 865-389-8852 685 Emory Valley Rd (B)
| Phone: 865-425-0978 Oak Ridge, TN 37830
----------------------------------------------------------
More information about the Beowulf
mailing list