<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Cheers SLURM people,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’re seeing some intermittent job failures in our SLURM cluster, all with the same 137 exit code. I’m having difficulty in determining whether this error code is coming from SLURM (timeout?) or the Linux OS (process killed, maybe memory).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In this example, there’s the WEXITSTATUS in the slurmctld.log, error:0 status 35072 in the slurd.log, and ExitCode 9:0 in the accounting log….???<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does anyone have insight into how all these correlate? I’ve spent a significant amount of time digging through the documentation, and I don’t see a clear way on how to interpret all these…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Example: Job: 62791<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@XXXXXXXXXXXXX] /var/log/slurm# grep -ai jobid=62791 slurmctld.log<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T10:58:28.599] _slurm_rpc_submit_batch_job: JobId=62791 InitPrio=4294845347 usec=679<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T10:58:29.080] sched: Allocate JobId=62791 NodeList= XXXXXXXXXXXXX #CPUs=1 Partition=normal<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T11:17:45.275] _job_complete: JobId=62791 <span style="background:yellow;mso-highlight:yellow">
WEXITSTATUS 137</span><o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T11:17:45.294] _job_complete: JobId=62791 done<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@ XXXXXXXXXXXXX] /var/log/slurm# grep 62791 slurmd.log<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T10:58:29.090] _run_prolog: prolog with lock for job 62791 ran for 0 seconds<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T10:58:29.090] Launching batch job 62791 for UID 847694<o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T11:17:45.280] [62791.batch] sending REQUEST_COMPLETE_BATCH_SCRIPT,
<span style="background:yellow;mso-highlight:yellow">error:0 status 35072</span><o:p></o:p></p>
<p class="MsoNormal">[2020-08-13T11:17:45.405] [62791.batch] done with job<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@XXXXXXXXXXXXX] /var/log/slurm# sacct -j 62791<o:p></o:p></p>
<p class="MsoNormal"> JobID JobName Partition Account AllocCPUS State ExitCode
<o:p></o:p></p>
<p class="MsoNormal">------------ ---------- ---------- ---------- ---------- ---------- --------
<o:p></o:p></p>
<p class="MsoNormal">62791 nf-normal+ normal (null) 0 FAILED
<span style="background:yellow;mso-highlight:yellow">9:0</span> <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@XXXXXXXXXXXXX] /var/log/slurm# sacct -lc | tail -n 100 | grep 62791<o:p></o:p></p>
<p class="MsoNormal">JobID UID JobName Partition NNodes NodeList State Start End Timelimit
<o:p></o:p></p>
<p class="MsoNormal">62791 847694 nf-normal+ normal 1 XXXXXXXXXXX.+ FAILED 2020-08-13T10:58:29 2020-08-13T11:17:45 UNLIMITED<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Anthony<b><u><span style="font-size:9.0pt;color:#3333FF"> <o:p>
</o:p></span></u></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<br>
________________________________________<br>
<span style="font-size:9pt; font-family: 'Calibri','times new roman','garamond',serif; color:#928E8E;"><b>IMPORTANT</b> - PLEASE READ: This electronic message, including its attachments, is CONFIDENTIAL and may contain PROPRIETARY or LEGALLY PRIVILEGED or PROTECTED
information and is intended for the authorized recipient of the sender. If you are not the intended recipient, you are hereby notified that any use, disclosure, copying, or distribution of this message or any of the information included in it is unauthorized
and strictly prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and permanently delete this message and its attachments, along with any copies thereof, from all locations received (e.g., computer, mobile
device, etc.). To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns
of network traffic and managing client relationships. For further information see: https://www.iqvia.com/about-us/privacy/privacy-policy. Thank you.
</span>
<p></p>
</body>
</html>