[Beowulf] Q: IB message rate & large core counts (per node) ?

[richard.walsh at comcast.net]

Mark Hahn wrote: 

>> Doesn't this assume worst case all-to-all type communication 
>> patterns. 
> 
>I'm assuming random point-to-point communication, actually. 


A sub-case of all-to-all (possibly all-to-all). So you are assuming 
random point-to-point is a common pattern in HPC ... mmm ... I 
would call it a worse case pattern, something more typical of 
graph searching codes like they run at the NSA. Sure a high 
radix switch (or better yet a global memory address space, Cray 
X1E) is good and designed for this worst-case, but not sure this 
is the common case data reference pattern in HPC ... if it were 
they would be selling more global memory systems at Cray and 
SGI (not just to the NSA). 


There you might also want a machine like the Cray XMT where 
the memory is flat and stalled threads can be switched out for 
another thread. 

>> If you are just trading ghost cell data with your neighbors 
>> and you have placed your job smartly on the torus the fan out 
>> advantage mentioned is irrelevant. No? 
> 
>if your comms are nearest-neighbor, then yes, a nearest-neighbor 
>fabric is your friend ;) 


I think that if you look at the HPC space globally there is still a lot 
of locality that you can rely on. Familiar with the "7 dwarves" paper 
from Berkeley? 

>how often does that actually happen? to work out so neatly would 
>preclude, for instance, adaptive meshes, right? it seems like mostly 
>I see jobs with no obvious regular structure to their communication. 



Really ... must be doing a lot of turbulent flow simulations with shedding 
vortices, crash simulations with self-penetrating meshes ... tough stuff 
for your average cluster or even your above average cluster. Even AMR 
codes usually attempt to discover new neighbors and localize them. 


Not disrespecting switches, but they are in a sense designed for worse 
case scenarios (the design asserts that "there are no neighborhoods") 
... a torus design appeals to the middle ground were locality is not banished. 


rbw 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20100226/c70b3e8d/attachment.html>