[Beowulf] IB in the real world

[Josh England]

On Wed, 2005-05-25 at 15:47 -0400, Patrick Geoffray wrote:
> Hi Josh,
> 
> Josh England wrote:
> > On Thu, 2005-05-12 at 14:32 -0700, Bill Broadley wrote:
> 
> >>6.  What real world latencies and bandwidths are you observing on production
> >>    clusters with MPI?  How much does that change when all nodes are running
> >>    the latency or bandwidth benchmark?
> > 
> > 
> > I don't have the numbers offhand, but I recall about 7-9ns latency and
> > ~=800MB/s on PCI-X and ~=1200MB/s on PCIe.
> 
> ~1200 MB/s on a single IB 4x link ?
> 
> The IB 4x link is 4 links at 2.5 Gb/s signal rate, so that 10 Gb/s 
> signal rate total. After the 8b/10b encoding, that's 8 Gb/s data rate. 
> Even without counting the efficiency of the protocol (ratio payload size 
> / packet size wich should cost ~3% conservatively assuming a header of 
> 64 Bytes and a max payload of 2KB), there is no way to get more than 1 
> GB/s *on paper*.
> 
> How did you get 1200 MB/s ?

I should have been more explicit.  I don't mean the dual port cards.
This is only on a single port.

The latency is around 3.5us and bw is around 900MB/s unidirectional and
1600MB/s bidirectional on PCIe.

PCI-X is 6.0-7.0us for latency and 750-850MB/s (uni), 750-875 (bi).

These numbers may be a bit off, but I think they're close.

-JE