[Beowulf] Mutiple IB networks in one cluster

[Alex Chekholko]

Hi Prentice,

Today, IB probably means Mellanox, so why not get their pre-sales
engineer to draw you up a fabric configuration for your intended use
case?

Certainly you can have a fabric where each host has two links, and
then you segregate the different types of traffic on the different
links.  But what would that accomplish if they're using the same
fabric?

Certainly you can have totally separate fabrics and each host could
have links to one or more of those.

If this was Ethernet, you'd comparing separate networks vs multiple
interfaces on the same network vs bonded interfaces on the same
network.  Not all the concepts translate directly, the main one being
the default network layout, Mellanox will suggest a strict fat tree.

Furthermore, your question really just comes down to performance.
Leave IB out of it.  You're asking: is an interconnect with such and
such throughput and latency sufficient for my heterogeneous workload
comprised of bulk data transfers and small messages.  Only you can
answer that.

Regards,
Alex


On Thu, Jan 30, 2014 at 8:33 AM, Prentice Bisbal
<prentice.bisbal at rutgers.edu> wrote:
> Beowulfers,
>
> I was talking to a colleague the other day about cluster architecture and
> big data, and this colleague was thinking that it would be good to have two
> separate FDR IB clusters within a single cluster: one for message-passing,
> and the other purely for data movement. I'm a bit skeptical of this myself.
> I was always under the impression that IB has more than enough bandwidth for
> message-passing and I/O. I have some questions about this idea:
>
> 1. Does this make sense?
>
> 2. Does anyone have first hand experience with doing this, or can point me
> to someone who does (articles on line, papers on the topic will suffice)?
>
> 3. Would the present any issues for managing the fabric? I know IB is
> designed to detect loops automatically, but what about making sure  certain
> traffic stays on certain IB interfaces.
>
> 4. Since IB uses cross-bar switches (please correct me if I'm wrong), we
> wouldn't need to duplicate switchgear, just double IB connections on each
> host, correct?
>
> --
> Prentice
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf