Hi,<br>please see below<br><br><div class="gmail_quote">On Wed, Sep 2, 2009 at 6:57 PM, Mark Hahn <span dir="ltr"><<a href="mailto:hahn@mcmaster.ca">hahn@mcmaster.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Wed, 2 Sep 2009, amjad ali wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi All,<br>
I have 4-Nodes ( 4 CPUs Xeon3085, total 8 cores) Beowulf cluster on ROCKS-5<br>
with GiG-Ethernet. I tested runs of a 1D CFD code both serial and parallel<br>
on it.<br>
Please reply following:<br>
<br>
1) When I run my serial code on the dual-core head node (or parallel code<br>
with -np 1); it gives results in about 2 minutes. What I observe is that<br>
"System Monitor" application show that some times CPU1 become busy 80+% and<br>
CPU2 around 10% busy. After some time CPU1 gets share around 10% busy while<br>
the CPU2 becomes 80+% busy. Such fluctuations/swap-of-busy-ness continue<br>
till end. Why this is so? Does this busy-ness shifts/swaping harms<br>
performance/speed?<br>
</blockquote>
<br></div>
the kernel decides where to run processes based on demand. if the machine<br>
were otherwise idle, your process would stay on the same CPU. depending on<br>
the particular kernel release, the kernel uses various heuristics to decide<br>
how much to "resist" moving the process among cpus.<br>
<br>
the cost of moving among cpus depends entirely on how much your code depends<br>
on the resources tied to one cpu or the other. for instance, if your code has a very small memory footprint, moving will have only trivial cost.<br>
if your process has a larger working set size, but fits in onchip cache,<br>
it may be relatively expensive to move to a different processor in the system that doesn't share cache. consider a 6M L3 in a 2-socket system,<br>
for instance: the inter-socket bandwidth will be approximately memory speed,<br>
which on a core2 system is something like 6 GB/s. so migration will incur<br>
about a 1ms overhead (possibly somewhat hidden by concurrency.)<br>
<br>
in your case (if I have the processor spec right), you have 2 cores<br>
sharing a single 4M L2. L1 cache is unshared, but trivial in size,<br>
so migration cost should be considered near-zero.<br>
<br>
the numactl command lets you bind a cpu to a processor if you wish.<br>
this is normally valuable on systems with more complex topologies,<br>
such as combinations of shared and unshared caches, especially when divided over multiple sockets, and with NUMA memory (such as opterons and nehalems.)<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
2) When I run my parallel code with -np 2 on the dual-core headnode only;<br>
it gives results in about 1 minute. What I observe is that "System Monitor"<br>
application show that all the time CPU1 and CPU2 remain busy 100%.<br>
</blockquote>
<br></div>
no problem there. normally, though, it's best to _not_ run extraneous processes, and instead only look at the elapsed time that the job takes to run. that is the metric that you should care about.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
3) When I run my parallel code with "-np 4" and "-np 8" on the dual-core<br>
headnode only; it gives results in about 2 and 3.20 minutes respectively.<br>
What I observe is that "System Monitor" application show that all the time<br>
CPU1 and CPU2 remain busy 100%.<br>
</blockquote>
<br></div>
sure. with 4 cpus, you're overloading the cpus, but they timeslice fairly<br>
efficiently, so you don't lose. once you get to 8 cpus, you lose because the overcommitted processes start interfering (probably their working set<br>
is blowing the L2 cache.)<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
4) When I run my parallel code with "-np 4" and "-np 8" on the 4-node (8<br>
cores) cluster; it gives results in about 9 (NINE) and 12 minutes. What I<br>
</blockquote>
<br></div>
well, then I think it's a bit hyperbolic to call it a parallel code ;)<br>
seriously, all you've learned here is that your interconnect is causing<br>
your code to not scale. the problem could be your code or the interconnect.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
observe is that "System Monitor" application show CPU usage fluctuations<br>
somewhat as in point number 1 above (CPU1 remains dominant busy most of the<br>
time), in case of -np 4. Does this means that an MPI-process is shifting to<br>
different cores/cpus/nodes? Does these shiftings harm performance/speed?<br>
</blockquote>
<br></div>
MPI does not shift anything. the kernel may rebalance runnable processes within a single node, but not across nodes. it's difficult to tell how much<br>
your monitoring is harming the calculation or perturbing the load-balance.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
5) Why "-np 4" and "-np 8" on cluster is taking too much time as compare to<br>
-np 2 on the headnode? Obviously its due to communication overhead! but how<br>
to get better performance--lesser run time? My code is not too complicated<br>
only 2 values are sent and 2 values are received by each process after each<br>
stage.<br>
</blockquote>
<br></div>
then do more work between sends and receives. hard to say without knowing exactly what the communication pattern is.<br></blockquote><div><br>Here is my subroutine:<br><br> IF (myrank /= p-1) CALL MPI_ISEND(u_local(Np,E),1,MPI_REAL8,myrank+1,55,MPI_COMM_WORLD,right(1), ierr)<br>
IF (myrank /= 0) CALL MPI_ISEND(u_local(1,B),1,MPI_REAL8,myrank-1,66,MPI_COMM_WORLD,left(1), ierr)<br> <br> IF (myrank /= 0) CALL MPI_IRECV(u_left_exterior,1, MPI_REAL8, myrank-1, 55, MPI_COMM_WORLD,right(2), ierr)<br>
IF (myrank /= p-1) CALL MPI_IRECV(u_right_exterior,1, MPI_REAL8, myrank+1, 66, MPI_COMM_WORLD,left(2), ierr)<br><br> u0_local=RESHAPE(u_local,(/Np*K_local/))<br> du0_local=RESHAPE(du_local,(/Nfp*Nfaces*K_local/))<br>
q_local = rx_local*MATMUL(Dr,u_local)<br> <br> DO I = shift*Nfp*Nfaces+1+1 , shift*Nfp*Nfaces+Nfp*Nfaces*K_local-1<br> du0_local(I) = (u0_local(vmapM_local(I))-u0_local(vmapP_local(I)))/2.0_8<br> ENDDO<br>
<br> I = shift*Nfp*Nfaces+1<br> IF (myrank == 0) du0_local(I) = (u0_local(vmapM_local(I))-u0_local(vmapP_local(I)))/2.0_8<br> <br><br> IF (myrank /= p-1) CALL MPI_WAIT(right(1), status, ierr)<br> IF (myrank /= 0) CALL MPI_WAIT(left(1), status, ierr)<br>
IF (myrank /= 0) CALL MPI_WAIT(right(2), status, ierr)<br> IF (myrank /= p-1) CALL MPI_WAIT(left(2), status, ierr)<br> <br><br> IF (myrank /= 0) THEN<br> du0_local(I) = (u0_local(vmapM_local(I))-u_left_exterior)/2.0_8<br>
ENDIF<br> <br> I = shift*Nfp*Nfaces+Nfp*Nfaces*K_local<br> IF (myrank == p-1) du0_local(I) = (u0_local(vmapM_local(I))-u0_local(vmapP_local(I)))/2.0_8<br> <br> IF (myrank /= p-1) THEN<br> du0_local(I) = (u0_local(vmapM_local(I))-u_right_exterior)/2.0_8 <br>
ENDIF<br> <br> IF (myrank == 0) du0_local(mapI) = 0.0_8<br> IF (myrank == p-1) du0_local(mapO) = 0.0_8<br> <br> du_local = RESHAPE(du0_local,(/Nfp*Nfaces,K_local/))<br> <br> q_local = q_local-MATMUL(LIFT,Fscale_local*(nx_local*du_local))<br>
<br> IF (myrank /= p-1) CALL MPI_ISEND(q_local(Np,E),1,MPI_REAL8,myrank+1,551,MPI_COMM_WORLD,right(1), ierr)<br> IF (myrank /= 0) CALL MPI_ISEND(q_local(1,B),1,MPI_REAL8,myrank-1,661,MPI_COMM_WORLD,left(1), ierr)<br>
<br> IF (myrank /= 0) CALL MPI_IRECV(q_left_exterior,1, MPI_REAL8, myrank-1, 551, MPI_COMM_WORLD,right(2), ierr)<br> IF (myrank /= p-1) CALL MPI_IRECV(q_right_exterior,1, MPI_REAL8, myrank+1, 661, MPI_COMM_WORLD,left(2), ierr)<br>
<br> q0_local=RESHAPE(q_local,(/Np*K_local/))<br> dq0_local=RESHAPE(dq_local,(/Nfp*Nfaces*K_local/))<br><br> rhsu_local= rx_local*MATMUL(Dr,q_local) + u_local*(u_local-a1)*(1.0_8-u_local) - v_local<br><br> DO I = shift*Nfp*Nfaces+1+1 , shift*Nfp*Nfaces+Nfp*Nfaces*K_local-1<br>
dq0_local(I) = (q0_local(vmapM_local(I))-q0_local(vmapP_local(I)))/2.0_8<br> ENDDO<br><br> I = shift*Nfp*Nfaces+1<br> <br> IF (myrank /= p-1) CALL MPI_WAIT(right(1), status, ierr)<br> IF (myrank /= 0) CALL MPI_WAIT(left(1), status, ierr)<br>
IF (myrank /= 0) CALL MPI_WAIT(right(2), status, ierr)<br> IF (myrank /= p-1) CALL MPI_WAIT(left(2), status, ierr)<br> <br> IF (myrank /= 0) THEN<br> dq0_local(I) = (q0_local(vmapM_local(I))-q_left_exterior)/2.0_8<br>
ENDIF<br> <br> <br> I = shift*Nfp*Nfaces+Nfp*Nfaces*K_local<br> <br> IF (myrank /= p-1) THEN<br> dq0_local(I) = (q0_local(vmapM_local(I))-q_right_exterior)/2.0_8 <br> ENDIF<br>
<br> IF (myrank == 0) dq0_local(mapI) = q0_local(vmapI)+0.225_8<br> IF (myrank == p-1) dq0_local(mapO) = q0_local(vmapO)<br><br> dq_local = RESHAPE(dq0_local,(/Nfp*Nfaces,K_local/))<br><br> rhsu_local= rhsu_local-MATMUL(LIFT,(Fscale_local*(nx_local*dq_local)))<br>
<br>END SUBROUTINE<br>=====================================================================<br><br>Is it suffieciently goog or there are some serious problems with the communication pattern. Here Arrays are not very big because Nfp*Nfaces = 2 and K_local = < 30. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
I think you should first validate your cluster to see that the Gb is running as fast as expected. actually, that everything is running right.<br>
that said, Gb is almost not a cluster interconnect at all, since it's so much slower than the main competitors (IB mostly, to some extent 10GE).<br>
fatter nodes (dual-socket quad-core, for instance) would at least decrease<br>
the effect of slow interconnect.<br>
<br>
you might also try instaling openMX, which is an ethernet protocol<br>
optimized for MPI (rather than your current MPI which is presumably layered on top of the usual TCP stack, which is optimized for wide-area<br>
streaming transfers.) heck, you can probably obtain some speedup by tweaking your coalesce settings via ethtool.<br>
</blockquote></div><br>