[Beowulf] IB switches: managed or not?

[Bill Broadley]

Andrew Robbie (GMail) wrote:
> Hi,
> 
> I am building a small (~16) node cluster with an IB interconnect. I need 
> to decide whether I will buy a cheaper, dumb switch and run OpenSM, or 
> get a more expensive switch with a built in subnet manager. The largest 
> this system would every grow is 32 nodes (two 24 port switches).

Be warned, if you depend on a managed switch, that unless you have
a signed contract with the switch vendor you might get stranded with
a broken subnet manager.  We bought a topspin switch and they agreed
to "support" for 1 year and firmware updates for the life of the switch.

Then cisco bought topspin, and won't talk to us without a signed support
contract.  So now we have a switch that often hangs and fragile subnet
manager.  I can see requiring an updated license for new features, but seems
very strange to charge for fixes to something inherently broken.

I have high hopes for OpenSM, I plan to give it a shot this summer.

We also have a newer silverstorm IB (288 port) switch that is dramatically 
more reliable than the cisco and have had zero problems with the built
in subnet manager.

So if your going to depend on a built in subnet manager make sure you have
a support contract for the switch and a license for the related updates
(which strangely isn't the same thing).  I now have very specific wording
for all requests for cluster bids.

Best of luck.

> My other query is about diagnostic software. With an ethernet switch it 
> is pretty easy to fire up Ethereal (sorry Wireshark, but it is such a 
> silly name) or Etherape and get a look at what is going on. If I buy a 
> Cisco or Voltaire etc do they come with tools that let me get accurate 
> representations of what is going on? Or are their tools really for large 
> IB networks?

I'm aware of no similar tools, but various IB nics have diagnostics available.
I'm most familiar with Infinipath, they have an ipathstats command that
makes a fair number of statistics available, email me if you want an email
(it's about 80x25 window worth).

The management interfaces of the topspin and silverstorm also report
statistics as well.  I could likely gather a screenshot or two if needed.