[Beowulf] Avoiding/mitigating fragmentation of systems by small jobs?

[Andrew Mather]

Hi Chris,


> Message: 2
> Date: Fri, 08 Jun 2018 17:21:56 +1000
> From: Chris Samuel <chris at csamuel.org>
> To: beowulf at beowulf.org
> Subject: [Beowulf] Avoiding/mitigating fragmentation of systems by
>         small   jobs?
> Message-ID: <2427060.afPWsf2KXH at quad>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi all,
>
> I'm curious to know what/how/where/if sites do to try and reduce the
> impact of
> fragmentation of resources by small/narrow jobs on systems where you also
> have
> to cope with large/wide parallel jobs?
>
> For my purposes a small/narrow job is anything that will fit on one node
> (whether a single core job, multi-threaded or MPI).
>
>
Somewhat ancient history for me now and we didn't have to deal with
multi-node jobs...  :)

Hopefully my memory doesn't let me down and no doubt my successor has
tweaked things :)

But on our Torque/Moab system at $JOB -1 we just used to place a higher
priority multiplier on larger jobs, which had the effect of the scheduler
shuffling things around so they'd run as soon as possible.  We had a fairly
complex prioritisation setup there, so job size was only one factor.


........ 8< Snip .....

But really I'm curious to know what people do about this, or do you not
> worry
> about it at all and just let the scheduler do its best?
>

Pretty much this really, given the other priority multipliers



> All the best,
> Chris
>

Andrew
(wondering why he can't let this list go)


> --
>  Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC
>
> -
 https://picasaweb.google.com/107747436224613508618
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
“We need the tonic of wildness.... We can never have enough of nature.”― Henry
David Thoreau
<http://www.goodreads.com/author/show/10264.Henry_David_Thoreau>, Walden:
Or, Life in the Woods <http://www.goodreads.com/work/quotes/2361393>
-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180608/7acf5fe6/attachment.html>