[Beowulf] Q: IB message rate & large core counts (per node)?

[Tom Ammon]

If I understand correctly, 40GbE is 64/66 encoded.

Tom


On 3/15/2010 5:30 PM, Patrick Geoffray wrote:
> On 3/15/2010 5:24 PM, richard.walsh at comcast.net wrote:
>    
>> to best and worst case). It would be good to add Ethernet to the mix
>> (1Gb, 10Gb, and 40Gb) as well.
>>      
> 10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw
> bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is
> still in draft. Last time I looked at 40 Gb/s, it was pretty much four
> 10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate.
>
>    
>>   >There is a similar protocol efficiency at the IB or Ethernet level, but
>>   >the MTU is large enough that it's much smaller compared to PCIe.
>>
>> Would you estimate less than 1%, 2%, 4% ... ??
>>      
> It depends on the packet size. For example, 14 Bytes Ethernet header on
> 1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less
> than that. I don't know the header size in IB, but with an MTU of 2K or
> 4K, it's negligible.
>
> However, things are different for tiny packets. The minimum packet size
> on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is
> 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC,
> etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684
> MB/s if you remove the 14B Ethernet header (54% efficiency).
>
> I don't think you can put all that on an Excel sheet :-)
>
> Patrick
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>    


-- 
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu