<div dir="ltr"><div>Joe - Leela? I did not know you were a Dr Who fan.</div><div><br></div><div>Faraz, you really should log into your switch and look at the configuration of the ports.</div><div>Find the port to which that compute node is connected by listing the MAC address table.</div><div>(If you are using Bright there is an easy way to do this).</div><div>Look at the port configuration - is it capped to a certain rate?</div><div>The next step is to bring the interface down then up to see if it renegotiates.</div><div>Probably won't so then it is a trip to the data centre to reseat the connection. (Tha tis the posh phrase for pulling the cable out and sticking it back in).</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 17 August 2017 at 20:40, Gus Correa <span dir="ltr"><<a href="mailto:gus@ldeo.columbia.edu" target="_blank">gus@ldeo.columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 08/17/2017 12:35 PM, Joe Landman wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
<br>
On 08/17/2017 12:00 PM, Faraz Hussain wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
I noticed an mpi job was taking 5X longer to run whenever it got the compute node lusytp104 . So I ran qperf and found the bandwidth between it and any other nodes was ~100MB/sec. This is much lower than ~1GB/sec between all the other nodes. Any tips on how to debug further? I haven't tried rebooting since it is currently running a single-node job.<br>
<br>
[hussaif1@lusytp114 ~]$ qperf lusytp104 tcp_lat tcp_bw<br>
tcp_lat:<br>
latency = 17.4 us<br>
tcp_bw:<br>
bw = 118 MB/sec<br>
[hussaif1@lusytp114 ~]$ qperf lusytp113 tcp_lat tcp_bw<br>
tcp_lat:<br>
latency = 20.4 us<br>
tcp_bw:<br>
bw = 1.07 GB/sec<br>
<br>
This is separate issue from my previous post about a slow compute node. I am still investigating that per the helpful replies. Will post an update about that once I find the root cause!<br>
</blockquote>
<br>
Sounds very much like it is running over gigabit ethernet vs Infiniband. Check to make sure it is using the right network ...<br>
</blockquote>
<br></span>
Hi Faraz<br>
<br>
As others have said answering your previous posting about Infiniband:<br>
<br>
- Check if the node is configured the same way as the other nodes,<br>
in the case of Infinband, if the MTU is the same,<br>
using connected or datagram mode, etc.<br>
<br>
**<br>
<br>
Besides, for Open MPI you can force it at runtime not to use tcp:<br>
--mca btl ^tcp<br>
or with the syntax in this FAQ:<br>
<a href="https://www.open-mpi.org/faq/?category=openfabrics#ib-btl" target="_blank" rel="noreferrer">https://www.open-mpi.org/faq/?<wbr>category=openfabrics#ib-btl</a><br>
<br>
If that node has an Infinband interface with a problem,<br>
this should at least give a clue.<br>
<br>
**<br>
<br>
In addition, check the limits in the node.<br>
That may be set by your resource manager,<br>
or in /etc/security/limits.conf<br>
or perhaps in the actual job script.<br>
The memlock limit is key to Open MPI over Infiniband.<br>
See FAQ 15, 16, 17 here:<br>
<a href="https://www.open-mpi.org/faq/?category=openfabrics" target="_blank" rel="noreferrer">https://www.open-mpi.org/faq/?<wbr>category=openfabrics</a><br>
<br>
**<br>
<br>
Moreover, check if the mlx4_core.conf (assuming it is Mellanox HW)<br>
is configured the same way across the nodes:<br>
<br>
/etc/modprobe.d/mlx4_core.conf<br>
<br>
See FAQ 18 here:<br>
<a href="https://www.open-mpi.org/faq/?category=openfabrics" target="_blank" rel="noreferrer">https://www.open-mpi.org/faq/?<wbr>category=openfabrics</a><br>
<br>
**<br>
<br>
To increase the btl diagnostic verbosity (that goes to STDERR, IRRC):<br>
<br>
--mca btl_base_verbose 30<br>
<br>
That may point out which interfaces are actually being used, etc.<br>
<br>
See this FAQ:<br>
<br>
<a href="https://www.open-mpi.org/faq/?category=all#diagnose-multi-host-problems" target="_blank" rel="noreferrer">https://www.open-mpi.org/faq/?<wbr>category=all#diagnose-multi-ho<wbr>st-problems</a><br>
<br>
**<br>
<br>
Finally, as John has suggested before, you may want to<br>
subscribe to the Open MPI mailing list,<br>
and ask the question there as well:<br>
<br>
<a href="https://www.open-mpi.org/community/help/" target="_blank" rel="noreferrer">https://www.open-mpi.org/commu<wbr>nity/help/</a><br>
<a href="https://www.open-mpi.org/community/lists/" target="_blank" rel="noreferrer">https://www.open-mpi.org/commu<wbr>nity/lists/</a><br>
<br>
There you will get feedback from the Open MPI developers +<br>
user community, and that often includes insights from<br>
Intel and Mellanox IB hardware experts.<br>
<br>
**<br>
<br>
I hope this helps.<span class="HOEnZb"><font color="#888888"><br>
<br>
Gus Correa</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</blockquote>
<br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</div></div></blockquote></div><br></div>