[Beowulf] Q: IB message rate & large core counts (per node) ?

[Mark Hahn]

>> Regardless of how tight the seastar per-hop latency is,
>> IB has 2.5x the per-hop fanout (2or 3 outgoing 9.6 GB links
>> versus 18 outgoing 4 GB/s links). higher radix means an
>> advantage that increases with size.
>
> Doesn't this assume worst case all-to-all type communication
> patterns.

I'm assuming random point-to-point communication, actually.

> If you are just trading ghost cell data with your neighbors
> and you have placed your job smartly on the torus the fan out
> advantage mentioned is irrelevant. No?

if your comms are nearest-neighbor, 
then yes, a nearest-neighbor fabric is your friend ;)

how often does that actually happen?  to work out so neatly would 
preclude, for instance, adaptive meshes, right?  it seems like mostly
I see jobs with no obvious regular structure to their communication.