[Beowulf] Q: IB message rate & large core counts (per node)?

[Greg Lindahl]

On Mon, Mar 15, 2010 at 07:30:15PM -0400, Patrick Geoffray wrote:

> However, things are different for tiny packets. The minimum packet size  
> on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is  
> 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC,  
> etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684  
> MB/s if you remove the 14B Ethernet header (54% efficiency).

There's the additional complexity, for tiny packets, that different
cards will have different outgoing inter-packet gaps, usually greater
than the minimum.  Switches can merge streams from multiple hosts and
reduce that inter-packet gap on the receiving side, if multiple hosts
talk to one.  This is true for both Ethernet and IB.

And if we're talking MPI, different MPIs have different header sizes.
That's where a graph of non-coalesced MPI bandwidth (not including all
the overheads) as a function of message size and core counts is
interesting.

The spreadsheet can kinda be useful for large messages, but not for
data sizes < 1/2 MTU. But even for large data sizes, there are plenty
of other factors which your real application can trip over.

-- greg