[Beowulf] Infiniband modular switches

[Gilad Shainer]

Joe Landman wrote:

> On a different note ...
> 
> Gilad Shainer wrote:
> 
> > routing capabilities. And if you are bandwidth bounded, 
> using IB QDR 
> > will help. You will be able to drive more than 3GB/s from 
> each server.
> 
> This is theoretical max, correct?  What sort of real observed 
> DMA bandwidth are people seeing with the QDR adapters?  That 
> is, how quickly can I copy 32MB from main memory to the PCIe 
> (even gen2) based card?  Or how quickly (e.g. what is the 
> real observed data rate) for copying streaming data, say from 
> a nice NFS over RDMA session, to the card?
> 


No, this is the actual number. IB QDR  theoretical max is 4GB/s
(uni-directional BW)


> Please correct me if I am wrong, but I suspect that this will 
> be one of rate limiting factors at least in the near term.
> 
> Not that this is a bad thing, 3GB/s is quite good, but if the 
> practical best case memory->PCIe bandwidth only let us get 
> 2GB/s, or even 1.5 GB/s, this is important to know.
> 

PCIe x8 Gen1 is maxed in around 1.5-1.6GB/s, and PCIe x8 Gen2 in around
3.3-3.4GB/s. There are multiple PCIe Gen2 servers out there from
multiple vendors. I am using Supermicro based for my testing, but there
are other solutions. 
Here is some non-tuned MPI results  bandwidth (IB QDR on PCIe x8 Gen2
servers):
Bytes		PingPong (MB/s)	SendRecv (MB/s)
512		695.71		707.2
1024		1160.13		1178.18
2048		1664.83		1972.34
4096		1957.76		2597.41
8192		2165.76		3025.58
16384		2642.91		4187.19
32768		2962.5		5076.31
65536		3165.03		5693.37
131072	3274.05		6078.59
262144	3332.28		6287.49
524288	3362.12		6399.46
1048576	3377.28		6458.3
2097152	3385.08		6489.52
4194304	3388.02		6457.68

 

> If you or someone out there with QDR could measure this, I 
> think a few of us (at least on the storage side) would like 
> to know this.  If someone wants to loan us cards and cables 
> (cough cough), we would be happy to do this measurement 
> between pairs of JackRabbits.


I might be able to help you here. 

Gilad.