[Beowulf] Mutiple IB networks in one cluster

Alex Chekholko alex.chekholko at gmail.com
Thu Jan 30 16:15:37 PST 2014

Hi Prentice,

Today, IB probably means Mellanox, so why not get their pre-sales
engineer to draw you up a fabric configuration for your intended use

Certainly you can have a fabric where each host has two links, and
then you segregate the different types of traffic on the different
links.  But what would that accomplish if they're using the same

Certainly you can have totally separate fabrics and each host could
have links to one or more of those.

If this was Ethernet, you'd comparing separate networks vs multiple
interfaces on the same network vs bonded interfaces on the same
network.  Not all the concepts translate directly, the main one being
the default network layout, Mellanox will suggest a strict fat tree.

Furthermore, your question really just comes down to performance.
Leave IB out of it.  You're asking: is an interconnect with such and
such throughput and latency sufficient for my heterogeneous workload
comprised of bulk data transfers and small messages.  Only you can
answer that.


On Thu, Jan 30, 2014 at 8:33 AM, Prentice Bisbal
<prentice.bisbal at rutgers.edu> wrote:
> Beowulfers,
> I was talking to a colleague the other day about cluster architecture and
> big data, and this colleague was thinking that it would be good to have two
> separate FDR IB clusters within a single cluster: one for message-passing,
> and the other purely for data movement. I'm a bit skeptical of this myself.
> I was always under the impression that IB has more than enough bandwidth for
> message-passing and I/O. I have some questions about this idea:
> 1. Does this make sense?
> 2. Does anyone have first hand experience with doing this, or can point me
> to someone who does (articles on line, papers on the topic will suffice)?
> 3. Would the present any issues for managing the fabric? I know IB is
> designed to detect loops automatically, but what about making sure  certain
> traffic stays on certain IB interfaces.
> 4. Since IB uses cross-bar switches (please correct me if I'm wrong), we
> wouldn't need to duplicate switchgear, just double IB connections on each
> host, correct?
> --
> Prentice
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list