[Beowulf] Q: IB message rate & large core counts (per node)?

[Patrick Geoffray]

On 3/15/2010 5:24 PM, richard.walsh at comcast.net wrote:
> to best and worst case). It would be good to add Ethernet to the mix
> (1Gb, 10Gb, and 40Gb) as well.

10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw 
bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is 
still in draft. Last time I looked at 40 Gb/s, it was pretty much four 
10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate.

>  >There is a similar protocol efficiency at the IB or Ethernet level, but
>  >the MTU is large enough that it's much smaller compared to PCIe.
>
> Would you estimate less than 1%, 2%, 4% ... ??

It depends on the packet size. For example, 14 Bytes Ethernet header on 
1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less 
than that. I don't know the header size in IB, but with an MTU of 2K or 
4K, it's negligible.

However, things are different for tiny packets. The minimum packet size 
on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is 
14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC, 
etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684 
MB/s if you remove the 14B Ethernet header (54% efficiency).

I don't think you can put all that on an Excel sheet :-)

Patrick