[Beowulf] IB switches: managed or not?

[Markus Baumgartner]

Andrew Robbie (GMail) wrote:
> Various vendors (integrators, not switch OEMs) have stated to me that
> managed switches are the go, and that OpenSM is (a) buggy, and (b)
> very time consuming to set up. But, a managed name brand switch seems
> to cost a lot more than a non-managed one using the Mellanox reference
> design kit (rebadged, but I suspect made by Flextronics...).
>
We have a small (3 nodes) cluster with an IB interconnect here. The
switch is unmanaged. I cannot confirm either (a) nor (b). OpenSM runs
without problems and the set-up is not too complicated.

On the other hand, we also have a much more expensive shared-memory
system from a well-known vendor that also features an IB interconnect.
We never had to use any of the extra features of the managed switch we
have there. And in contrast to the open-source and unsupported drivers
that we use in the small cluster, the commercial driver stack is buggy
and causes our machines to crash every now and then (usually under high
load).

My advice is to take the unmanaged switch.

> My other query is about diagnostic software. With an ethernet switch
> it is pretty easy to fire up Ethereal (sorry Wireshark, but it is such
> a silly name) or Etherape and get a look at what is going on. If I buy
> a Cisco or Voltaire etc do they come with tools that let me get
> accurate representations of what is going on? Or are their tools
> really for large IB networks?
>
If you run "IPoIB" you can use Ethernet monitoring tools to get
diagnostics of the emulated ethernet devices. Our managed switch did not
come with extra diagnostics software. The switch was shipped with the
whole system, though (OEM). I do not know what software you would get if
you buy a retail IB switch.
 
Regards,
Markus