[Beowulf] IB switches: managed or not?

[Steve Phillips (stevep)]

Hi Bill,

There is a process that enables you to get a software emtitlement to
receive software updates from Cisco, but it appears that it didn't reach
all our customers - please accept our apologies. I have escalated to the
Cisco SFS product managers (Topspin = SFS) to look into the situation
and you should receive an email from one of them shortly. If anyone else
is in a similar position, please send an email to me and I will pass on
to the appropriate people.

Best regards

Steve Phillips 
Market Manager, Server Switching 

-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
On Behalf Of Bill Broadley
Sent: Thursday, March 08, 2007 2:07 PM
To: Andrew Robbie (GMail)
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] IB switches: managed or not?

Andrew Robbie (GMail) wrote:
> Hi,
> 
> I am building a small (~16) node cluster with an IB interconnect. I 
> need to decide whether I will buy a cheaper, dumb switch and run 
> OpenSM, or get a more expensive switch with a built in subnet manager.

> The largest this system would every grow is 32 nodes (two 24 port
switches).

Be warned, if you depend on a managed switch, that unless you have a
signed contract with the switch vendor you might get stranded with a
broken subnet manager.  We bought a topspin switch and they agreed to
"support" for 1 year and firmware updates for the life of the switch.

Then cisco bought topspin, and won't talk to us without a signed support
contract.  So now we have a switch that often hangs and fragile subnet
manager.  I can see requiring an updated license for new features, but
seems very strange to charge for fixes to something inherently broken.

I have high hopes for OpenSM, I plan to give it a shot this summer.

We also have a newer silverstorm IB (288 port) switch that is
dramatically more reliable than the cisco and have had zero problems
with the built in subnet manager.

So if your going to depend on a built in subnet manager make sure you
have a support contract for the switch and a license for the related
updates (which strangely isn't the same thing).  I now have very
specific wording for all requests for cluster bids.

Best of luck.

> My other query is about diagnostic software. With an ethernet switch 
> it is pretty easy to fire up Ethereal (sorry Wireshark, but it is such

> a silly name) or Etherape and get a look at what is going on. If I buy

> a Cisco or Voltaire etc do they come with tools that let me get 
> accurate representations of what is going on? Or are their tools 
> really for large IB networks?

I'm aware of no similar tools, but various IB nics have diagnostics
available.
I'm most familiar with Infinipath, they have an ipathstats command that
makes a fair number of statistics available, email me if you want an
email (it's about 80x25 window worth).

The management interfaces of the topspin and silverstorm also report
statistics as well.  I could likely gather a screenshot or two if
needed.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org To change your subscription
(digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf