[Beowulf] IB in the real world

[Shane Canon]

I suspect he is talking about the dual port cards....  You can find the
result in this set of slides.  The number is actually almost 1500 MB/s.

http://nowlab.cis.ohio-state.edu/publications/conf-presentations/2004/liuj_sc04.pdf

--Shane

Patrick Geoffray wrote:
> Hi Josh,
> 
> Josh England wrote:
> 
>> On Thu, 2005-05-12 at 14:32 -0700, Bill Broadley wrote:
> 
> 
>>> 6.  What real world latencies and bandwidths are you observing on
>>> production
>>>    clusters with MPI?  How much does that change when all nodes are
>>> running
>>>    the latency or bandwidth benchmark?
>>
>>
>>
>> I don't have the numbers offhand, but I recall about 7-9ns latency and
>> ~=800MB/s on PCI-X and ~=1200MB/s on PCIe.
> 
> 
> ~1200 MB/s on a single IB 4x link ?
> 
> The IB 4x link is 4 links at 2.5 Gb/s signal rate, so that 10 Gb/s
> signal rate total. After the 8b/10b encoding, that's 8 Gb/s data rate.
> Even without counting the efficiency of the protocol (ratio payload size
> / packet size wich should cost ~3% conservatively assuming a header of
> 64 Bytes and a max payload of 2KB), there is no way to get more than 1
> GB/s *on paper*.
> 
> How did you get 1200 MB/s ?
> 
> Patrick