<div dir="ltr"><div><div><div><br></div>Hi everyone,<br><br></div> In case anyone else has seen similar problems, the latest is we've shown the problem exists not just for IB, but also Gbit networks, and the current theory is that it has to do with latencies introduced by the kernel scheduler -- a theory I'm hoping to test over the next day or two once I'm given root access to a set of nodes to play with. Basically, this appears to be a case of high jitter on the nodes impacting network performance, as seen in this table:<br>
<br></div><div> Quantile<br></div>TestNodes 50% 90% 97% 99%<br><div>C5.5-IB 180.7950 189.3320 197.9239 226.4684<br>
</div><div>RH62a-IB 219.0050 800.7130 2049.0532 3329.3111<br>
</div><div>RH62b-IB 258.0000 306.6360 2614.8770 4996.9770<br>C5.5-GB 5995.0000 6388.9800 6630.9400 6981.4500<br></div><div>RH6.2-GB 4689.6000 21746.7100 29692.5000 32588.4800</div>
<div><br></div><div> [Apologies for the poor formatting! Gmail doesn't do fixed-width fonts!]<br><br></div><div> The take-away from this is that the CentOS5.5 nodes (C5.5) show relatively low jitter in their timings. These results are from running 2000 runs of a 64K Allreduce on 8 nodes / 64 cores via the Intel MPI Benchmarks (v3.2.4), and using R's 'quantile' command to get the statistics on the timings. If anyone wants the full script I can send it to them. I removed the 10% column just to simplify the horrid formatting, but basically the 10% and 50% don't differ much, meaning that half the runs in <i>any</i> configuration are close to the minimum -- that is, they behave perfectly fine. For some of the tests, that's even true for 90% of the runs. But the CentOS5.5 runs are good up to the 99% quantile, whereas the others balloon up above 90%. (The difference between the two RH6.2 tests is likely largely due to the different hardware on those clusters, ... and in part probably just because I need to randomize test times, etc., to be more fair statistically.)<br>
</div><div><div><br></div><div> I believe the original CentOS5.5 kernel (2.6.18) was upgraded to 2.6.32-400.34.4uek (Oracle Linux? I'll check with our IT guys), which is pretty close to the 2.6.32-358 on the RHEL nodes, <i>but</i> the scheduler features enabled in both systems are very different. I don't have a list of what's turned on or off yet, but will once I have root and can mount the debug info. If anyone else has seen this and solved it via kernel tweaks, though, I'd be interested in knowing what settings you used. If not, and you're running a stock RHEL6.x 2.6.32 kernel, ... well, I'd be interested to hear if you see any similar jitter. For jobs with a high communication:computation ratio, it makes a pretty big difference -- we've seen applications run at 25-30% of previous speeds on the RH6 clusters due to this issue. For jobs with much less frequent communication, it doesn't matter as much, obviously.<br>
<br></div><div> I'll post an update once I'm able to play with /proc/sys/kernel/sched_features a bit.<br></div><div><br></div><div> Thanks,<br></div><div> - Brian<br><br></div><div>(PS. Most testing was done with the Intel 2013.3.163 compilers and OpenMPI 1.6.4, though I did also do one or two tests with MVAPICH and OpenMPI 1.8)<br>
</div><div><br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Apr 26, 2014 at 10:36 PM, Chris Samuel <span dir="ltr"><<a href="mailto:samuel@unimelb.edu.au" target="_blank">samuel@unimelb.edu.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Thu, 24 Apr 2014 12:24:23 PM Joe Landman wrote:<br>
<br>
> Yeah, that was the other thing I'd forgotten about. Might want to tweak<br>
> Cstates to C1. It could be going over-power and throttling. Turn of<br>
> ASPM and other (fairly painful) things.<br>
<br>
</div>This Mellanox document has some useful information on that (and other settings<br>
that can affect performance).<br>
<br>
<a href="http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf" target="_blank">http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf</a><br>
<br>
cheers,<br>
Chris<br>
<span><font color="#888888">--<br>
Christopher Samuel Senior Systems Administrator<br>
VLSCI - Victorian Life Sciences Computation Initiative<br>
Email: <a href="mailto:samuel@unimelb.edu.au" target="_blank">samuel@unimelb.edu.au</a> Phone: <a href="tel:%2B61%20%280%293%20903%2055545" value="+61390355545" target="_blank">+61 (0)3 903 55545</a><br>
<a href="http://www.vlsci.org.au/" target="_blank">http://www.vlsci.org.au/</a> <a href="http://twitter.com/vlsci" target="_blank">http://twitter.com/vlsci</a><br>
</font></span><div><div><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</div></div></blockquote></div><br></div></div>