[Beowulf] Q: IB message rate & large core counts (per node)?

Tom Ammon tom.ammon at utah.edu
Mon Mar 15 16:45:44 PDT 2010

If I understand correctly, 40GbE is 64/66 encoded.


On 3/15/2010 5:30 PM, Patrick Geoffray wrote:
> On 3/15/2010 5:24 PM, richard.walsh at comcast.net wrote:
>> to best and worst case). It would be good to add Ethernet to the mix
>> (1Gb, 10Gb, and 40Gb) as well.
> 10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw
> bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is
> still in draft. Last time I looked at 40 Gb/s, it was pretty much four
> 10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate.
>>   >There is a similar protocol efficiency at the IB or Ethernet level, but
>>   >the MTU is large enough that it's much smaller compared to PCIe.
>> Would you estimate less than 1%, 2%, 4% ... ??
> It depends on the packet size. For example, 14 Bytes Ethernet header on
> 1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less
> than that. I don't know the header size in IB, but with an MTU of 2K or
> 4K, it's negligible.
> However, things are different for tiny packets. The minimum packet size
> on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is
> 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC,
> etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684
> MB/s if you remove the 14B Ethernet header (54% efficiency).
> I don't think you can put all that on an Excel sheet :-)
> Patrick
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah

More information about the Beowulf mailing list