<html>
<body>
<font size=3>At 07:05 PM 1/29/2009, Greg Lindahl wrote:<br>
<blockquote type=cite class=cite cite="">On Thu, Jan 29, 2009 at
07:22:10PM -0500, Mark Hahn wrote:<br>
> I'll bite: suppose I run large MPI jobs (say, 1k rank)<br>
> and have 8 cores/node and 1 nic/node. under what
circumstances<br>
> would a node be primarily worried about message rate, rather than
latency?<br><br>
Well, let's say that you're doing a stencil computation on a 3D
grid,<br>
diagonals included. Then each cycle each core needs to send to 26<br>
neighbors, and then receive from 26 neighbors. Even if you have<br>
fat-ish nodes (~ 8 cores) and a clever layout of the cores onto the
3D<br>
grid, that's a lot of off-node messages in a row. And that's message<br>
rate.</blockquote><br>
Johnn Adams said "Facts are stubborn things," and there just
aren't enough of them in your example to determine whether bandwidth or
latency dominates communication time. I have a 3-d code that does
quite a lot of that. Assuming each processor has a 100 x 100 x 100
grid, the communication to the six face neighbors of 10,000 elements may
dominate. If the basic grid dimension is 10 instead of 100, the
communication to the 12 edge neighbors and the 8 corner neighbors may
take three times as those 6 messages.<br><br>
The critical thing from the hardware is the size of the message that
requires twice the latency time to transmit. For GigE at 30
microseconds, and this is 30 kilobits. For 64 bit floating point
that's about 2000 numbers. For messages smaller than that the
latency dominates, while for messages longer than that the bandwidth
does.<br><br>
The latency dominates in time-stepping finite-difference codes where you
need to do a time step in a few seconds. For steady state
finite-difference codes where you can spend an hour on a single solution,
the bandwidth determines how big a problem you can do.<br><br>
As the list is wont to say, YMMV.<br><br>
<br>
Mike </font></body>
</html>