[Beowulf] Infiniband: beyond 24 ports
jan.heichler at gmx.net
Mon Aug 25 23:43:14 PDT 2008
Dienstag, 26. August 2008, meintest Du:
>> Questions>> node.
>> 1) What is the cost-effective yet efficient way to connect this cluster
>> with IB?
JL> Understand that cost-effective is not necessarily the highest
JL> performance. This is where over-subscription comes in.
>> 2) How many switches are required, and of which size?
JL> With the new 36 port switch chips from Mellanox, you should need 1. For
JL> a reasonable oversubscription (16 down, 8 up), you would need 3 switches
JL> ... 1 master and 2 leaf switches.
2 Switches should be enough - why would you need a master Switch?
And you could do better:
2 Switches with each 13 down and 11 uplinks where the Uplink-Ports are directly connected with each other. So you have exactly the number of ports you need and an oversubscription of 1 : 1.2
And if you take into account that a PCIe gen1 IB NIC is capable of handling 1400 MB/s and a DDR-IB Link gives you something like 1800 MB/s you have 13 downlinks with 1400 MB/s each = 18200 MB/s and 11 uplinks with 1800 MB/s = 19800 MB/s - so no oversupscription at all!
(The last point is just a quick idea that came to my mind - probably not well thought through... maybe someone could add something ;-) )
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf