[Beowulf] interconnect and compiler ?

[Patrick Geoffray]

Hi Greg,

Greg Lindahl wrote:
> On Fri, Jan 30, 2009 at 07:07:34AM -0700, Michael H. Frese wrote:
 >
>> Johnn Adams said "Facts are stubborn things," and there just aren't  
>> enough of them in your example to determine whether bandwidth or latency 
>> dominates communication time.
> 
> Mark asked for an example, not a research paper. And we were

And Michael gave you a good practical example. You can always find a 
code that does something stupid like a 27-point (!) 3d stencil code 
sending tiny messages. I know that there are such naive codes out there, 
but reasonable stencil codes do exchange blocks that are not tiny, 
easily transforming a message rate problem into a bandwidth problem. And 
you could further transform the bandwidth problem into an overlap 
problem with ghost cells.

> network.  In the Berkeley "logp" model, for example, processor
> overhead and the "gap" betweeen messsages are fundamental parameters.
> The InfiniPath chip has a tiny "o" and a negative "g".  As a result,

Ok, my turn to bite :-) What is a negative "g" ?

> it can send a lot more small messages than other interconnects, and
> this number rises as you add more cores to a system (!).

Because the host library overhead (single core) is greater than the NIC 
HW overhead. And that's when the host is doing nothing but sending. Add 
some computation in there and the amount of messages a core will produce 
is not that high. Does multiply it by the number of cores make it big 
enough ? Maybe for a stencil code with tiny blocks where all the core 
are very synchronized, ie they all send and receive at the same time and 
wait for the data exchange before moving on.

In practice, stencil codes can start updating as soon as they receive a 
border block, so not only you overlap communication and computation, but 
  not all cores end up communicating at the same time.

When the number of core will be one order of magnitude larger than today 
(100s), then it will be a different discussion.

Patrick