[Beowulf] Mutiple IB networks in one cluster

[Lux, Jim (337C)]

On 2/5/14 11:49 PM, "Greg Lindahl" <lindahl at pbm.com> wrote:

>In the usual Clos network, 1/2 of the nodes can make a single call to
>the other 1/2 of the nodes. That's what's non-blocking. Nothing else
>is. Running any real code, every node talks to more than one other
>node, and the network is not non-blocking.
>
>It makes perfect sense in a telephone network. In the real world, a
>Clos network is good but not non-blocking.

In this case, though, leaving aside multicast, isn't it not so much that
the network cannot connect, but that any given node can only "receive" one
call at a time, so the blocking is when the second caller tries to call
the node when it's "busy"

You have N/2 independent paths available (or some equivalent) and that is
the most you could possibly use since any path can have only two endpoints.

If the nodes are (at some granularity) single threaded, then having
multiple interfaces on each node wouldn't help, since the CPU can only
talk to one at a time.  (Yes, real processors have DMA and so do I/O
interfaces, so you can actually have data coming from multiple sources at
exactly the same time)

Maybe the issue is that *real* codes don't have a single thread on a
processor? (even if the multiple parallel threads are things like CPU and
DMA I/O devices).


>