[Beowulf] Re: Re: Home beowulf - NIC latencies
patrick at myri.com
Tue Feb 15 02:12:35 PST 2005
Joachim Worringen wrote:
> This is an interesting issue. If you look at what Greg mentioned about
> dump NICs (like InfiniPath, or SCI) and the latency numbers Ole posted
> for ScaMPI on different interconnects (all(?) accessed through uDAPL),
> you see that the dumb interface SCI has the lowest latency for both,
Which is the original hardware Scali built its MPI upon, btw.
> pingpong and random, with random being about twice of pingpong. In
> contrast, the "smart" NIC Myrinet, which has much less CPU utilization,
> has twice the pingpong latency, and a slightly worse random-to-pingpong
No, it's not Myrinet, it's GM/Myrinet. There are many things that come
from the GM side of the equation, believe me.
> Why this? Maybe better pipelining in SCI, because it's write-and-forget
> for the CPU, with 16 outstanding transactions on the network level,
> while Myrinet obviously behaves differently here (although GM should
> also be PIO-write to the NIC memory for small messages).
Nope, no PIO for small messages with GM, DMA for everything.
A last remark. I really think that the argument of using the same
swiss-army-knive MPI implementation such as ScaMPI or Intel MPI or even
MPI/Pro to infere interconnect characteristics is even worse that
looking at latency and bandwidth alone. These implementations are never
going to be designed to use all hardware efficiently, their design is
either historic (Scali used to provided software for SCI alone) or
politicaly motivated (Intel is using uDapl, hummm, wonder why), or both.
They are by-products of the MPI forum failure to make the Standard
practical (compatible ABI).
More information about the Beowulf