<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Indeed, and you can quite easily get into a “boulders and sand” scheduling problem; if you allow the small interactive jobs (the sand) free access to everything, the scheduler tends to find them easy to schedule, partially fills nodes with them, and then finds
it can’t find contiguous resources large enough for the big parallel jobs (the boulders), and you end up with the large batch jobs pending forever.
<div class=""><br class="">
</div>
<div class="">I’ve tried various approaches to this in the past; for example pre-emption of large long running jobs, but that causes resource starvation (suspended jobs are still consuming virtual memory) and then all sorts of issues with timeouts on TCP connections
and so on and so forth, these being genomics jobs with lots of not-normal-HPC activities like talking to relational databases etc.</div>
<div class=""><br class="">
</div>
<div class="">I think you always end up having to ring-fence hardware for the large parallel batch jobs, and not allow the interactive stuff on it.</div>
<div class=""><br class="">
</div>
<div class="">This of course is what leads some users to favour the cloud, because it appears to be infinite, and so the problem appears to go away. But let's not get into that argument here.</div>
<div class=""><br class="">
</div>
<div class="">Tim<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 16 Jan 2020, at 23:50, Alex Chekholko via Beowulf <<a href="mailto:beowulf@beowulf.org" class="">beowulf@beowulf.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hey Jim,</div>
<div class=""><br class="">
</div>
There is an inverse relationship between latency and throughput. Most supercomputing centers aim to keep their overall utilization high, so the queue always needs to be full of jobs.
<div class=""><br class="">
</div>
<div class="">If you can have 1000 nodes always idle and available, then your 1000 node jobs will usually take 10 seconds. But your overall utilization will be in the low single digit percent or worse.</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class="">Alex</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jan 16, 2020 at 3:25 PM Lux, Jim (US 337K) via Beowulf <<a href="mailto:beowulf@beowulf.org" class="">beowulf@beowulf.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" class="">
<div class="gmail-m_-7849349986852910975WordSection1">
<p class="MsoNormal"><span style="font-size:11pt" class="">Are there any references out there that discuss the tradeoffs between interactive and batch scheduling (perhaps some from the 60s and 70s?) –
<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">Most big HPC systems have a mix of giant jobs and smaller ones managed by some process like PBS or SLURM, with queues of various sized jobs.
<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">What I’m interested in is the idea of jobs that, if spread across many nodes (dozens) can complete in seconds (<1 minute) providing essentially “interactive” access, in the context of large jobs taking
days to complete. It’s not clear to me that the current schedulers can actually do this – rather, they allocate M of N nodes to a particular job pulled out of a series of queues, and that job “owns” the nodes until it completes. Smaller jobs get run on
(M-1) of the N nodes, and presumably complete faster, so it works down through the queue quicker, but ultimately, if you have a job that would take, say, 10 seconds on 1000 nodes, it’s going to take 20 minutes on 10 nodes.<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class="">Jim<u class=""></u><u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt" class=""><u class=""></u> <u class=""></u></span></p>
<div class="">
<div class="">
<p class="MsoNormal"><span style="font-size:11pt" class="">-- <u class=""></u><u class=""></u></span></p>
</div>
</div>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
</div>
_______________________________________________<br class="">
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank" class="">
Beowulf@beowulf.org</a> sponsored by Penguin Computing<br class="">
To change your subscription (digest mode or unsubscribe) visit <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__beowulf.org_cgi-2Dbin_mailman_listinfo_beowulf&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=xK7X4jUX3oG8IizF_lTh0GNrYM4sF9nUCxNKq6vi97c&s=rnNXVoLqTeEFVWB-0Jr0hJC0BgpH2_jm2s51IZb0H8o&e=" rel="noreferrer" target="_blank" class="">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf [beowulf.org]</a><br class="">
</blockquote>
</div>
_______________________________________________<br class="">
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" class="">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br class="">
To change your subscription (digest mode or unsubscribe) visit <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__beowulf.org_cgi-2Dbin_mailman_listinfo_beowulf&d=DwIGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=xK7X4jUX3oG8IizF_lTh0GNrYM4sF9nUCxNKq6vi97c&s=rnNXVoLqTeEFVWB-0Jr0hJC0BgpH2_jm2s51IZb0H8o&e=" class="">
https://urldefense.proofpoint.com/v2/url?u=https-3A__beowulf.org_cgi-2Dbin_mailman_listinfo_beowulf&d=DwIGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=xK7X4jUX3oG8IizF_lTh0GNrYM4sF9nUCxNKq6vi97c&s=rnNXVoLqTeEFVWB-0Jr0hJC0BgpH2_jm2s51IZb0H8o&e=</a>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
</body>
</html>