[Beowulf] Infiniband: beyond 24 ports

[Jan Heichler]

Hallo Joe,

Dienstag, 26. August 2008, meintest Du:

>> Questions>> node.
:

>> 1) What is the cost-effective yet efficient way to connect this cluster 
>> with IB?

JL> Understand that cost-effective is not necessarily the highest
JL> performance.  This is where over-subscription comes in.

>> 2) How many switches are required, and of which size?

JL> With the new 36 port switch chips from Mellanox, you should need 1.  For
JL> a reasonable oversubscription (16 down, 8 up), you would need 3 switches
JL> ... 1 master and 2 leaf switches.

2 Switches should be enough - why would you need a master Switch? 

And you could do better:

2 Switches with each 13 down and 11 uplinks where the Uplink-Ports are directly connected with each other. So you have exactly the number of ports you need and an oversubscription of 1 : 1.2

And if you take into account that a PCIe gen1 IB NIC is capable of handling 1400 MB/s and a DDR-IB Link gives you something like 1800 MB/s you have 13 downlinks with 1400 MB/s each = 18200 MB/s and 11 uplinks with 1800 MB/s = 19800 MB/s - so no oversupscription at all!

(The last point is just a quick idea that came to my mind - probably not well thought through... maybe someone could add something ;-) )

Regards,
Jan Heichler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20080826/0f91fc86/attachment.html>