[Beowulf] Infiniband modular switches

[andrew holway]

+1 for the 24 port flextronics switches. They are very cost effective
for half bisectional networks upto 32 ports. It starts to get messy
after that.

I wonder how long we will be waiting for switches based on the 36p asic?

On Thu, Jun 12, 2008 at 4:08 PM, Don Holmgren <djholm at fnal.gov> wrote:
>
> Ramiro -
>
> You might want to also consider buying just a single 24-port switch for your
> 22 nodes, and then when you expand either replace with a larger switch, or
> build a distributed switch fabric with a number of leaf switches connecting
> into a central spine switch (or switches).  By the time you expand to the
> larger cluster, switches based on the announced 36-port Mellanox crossbar
> silicon will be available and perhaps per port prices will have dropped
> sufficiently to justify the purchase delay and the disruption at the time of
> expansion.
>
> If your applications can tolerate some oversubscription (less than a 1:1
> ratio of leaf-to-spine uplinks to leaf-to-node connections), a distributed
> switch fabric (leaf and spine) has the advantage of shorter (and cheaper)
> cables between the leaf switches and your nodes, and relatively fewer longer
> cables from the leaves back to the spine, compared with a single central
> switch.
>
> We have many Flextronics switches - SDR and DDR, 24-port and 144-port - on a
> pair of large clusters (520 nodes, and 600 nodes) built in 2005 and 2006. No
> complaints.  But, we have been self-supporting, and I would guess you would
> have very different support structures with Voltaire or Qlogic.  With the
> Flextronics
> switches you will definitely be using the OFED stack, and you will have to
> run
> a subnet manager on one of your nodes (dedicated is probably best).  You
> could
> optionally buy an embedded subnet manager on the Voltaire or Qlogic
> switches,
> depending upon model, though I believe for a large fabric an external subnet
> manager is still recommended.
>
> Don Holmgren
> Fermilab
>
>
>
>
> On Tue, 10 Jun 2008, Ramiro Alba Queipo wrote:
>
>> Hello everybody:
>>
>> We are about to build an HPC cluster with infiniband network starting
>> from 22 dual socket nodes with AMD QUAD core processors and in a year or
>> so we will be having about 120 nodes. We will be using infiniband both
>> for calculation as for storage.
>> The question is that we need a modular solution and we are having 3
>> candidates:
>>
>> a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)->
>> seems very good and well supported, but very expensive.
>>
>> b) Qlogic SilverStorm 9120 (144 ports) -> no price and support
>> information yet
>>
>> c) Flextronics 10U 144 Port Modular-> very good at price but little
>> support => risky option?.
>>
>> I am in a mess. What is your opinion about this matter? Are you using
>> any of this products.
>>
>> Regards
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>