[Beowulf] Re: Re: Home beowulf - NIC latencies

[Patrick Geoffray]

Joachim Worringen wrote:
> This is an interesting issue. If you look at what Greg mentioned about 
> dump NICs (like InfiniPath, or SCI) and the latency numbers Ole posted 
> for ScaMPI on different interconnects (all(?) accessed through uDAPL), 
> you see that the dumb interface SCI has the lowest latency for both, 

Which is the original hardware Scali built its MPI upon, btw.

> pingpong and random, with random being about twice of pingpong. In 
> contrast, the "smart" NIC Myrinet, which has much less CPU utilization, 
> has twice the pingpong latency, and a slightly worse random-to-pingpong 
> ratio.

No, it's not Myrinet, it's GM/Myrinet. There are many things that come 
from the GM side of the equation, believe me.

> Why this? Maybe better pipelining in SCI, because it's write-and-forget 
> for the CPU, with 16 outstanding transactions on the network level, 
> while Myrinet obviously behaves differently here (although GM should 
> also be PIO-write to the NIC memory for small messages).

Nope, no PIO for small messages with GM, DMA for everything.


A last remark. I really think that the argument of using the same 
swiss-army-knive MPI implementation such as ScaMPI or Intel MPI or even 
MPI/Pro to infere interconnect characteristics is even worse that 
looking at latency and bandwidth alone. These implementations are never 
going to be designed to use all hardware efficiently, their design is 
either historic (Scali used to provided software for SCI alone) or 
politicaly motivated (Intel is using uDapl, hummm, wonder why), or both. 
They are by-products of the MPI forum failure to make the Standard 
practical (compatible ABI).

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com