<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>IMHO, this PAM solution is a very neat solution. It only lacks a
network limitation (maybe just add a [traffic]shaper solution,
like tc ?).</p>
<p>Best regards<br>
</p>
<div class="moz-cite-prefix">Le 26/03/2021 à 17:30, Lohit Valleru
via Beowulf a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CA+Timp+b1B=zaXDSf2B7NLnaGEYfW8u6LW=62-BCg8pS1UyLRw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>I have just used a simple PAM script to apply cgroup rules
to every user who logs into a CentOS7 login node<br>
Something like this:</div>
<div><br>
</div>
<div>#!/bin/sh -e<br>
<br>
PAM_UID=$(getent passwd "${PAM_USER}" | cut -d: -f3)<br>
<br>
if [ "${PAM_UID}" -ge 1000 ]; then<br>
/bin/systemctl set-property "user-${PAM_UID}.slice" \<br>
CPUQuota=100% MemoryLimit=2G<br>
fi<br>
</div>
<div><br>
</div>
<div>This is not as sophisticated or does not change parameters
depending on dynamic load, But it does set static limits for
every user as per cgroups.</div>
<div><br>
</div>
<div>However, the above does not cover every scenario, and does
not restrict the number of threads, network load, network file
system load ( NFS/GPFS/Lustre). or paging etc.</div>
<div>I have actually seen cases where cgroups were causing more
stress trying to limit resources such as memory for users, who
happen to run hundreds of threads and still be able to stay
within the memory/cpu limit. It so happens that Cgroup does
not kill every application that goes beyond limits, as long as
the application tries to stay within its limits. </div>
<div>I tried limiting the number of threads with cgroups, and it
caused issues where it kills ssh connections when threads go
beyond a limit.<br>
Also, I recently realized about how Java does not recognize
cgroups for its garbage collection, and instead assumes that
all of physical memory is available.</div>
<div><br>
</div>
<div>I do not know if Arbiter somehow resolved the above issues,
and behaves much better than simple cgroup limits, or if
Redhat 8 happens to be better.</div>
<div><br>
</div>
<div>I do want to mention that for an ideal solution - i go with
Chris Dagdigian response, that it is best to educate users and
follow up respectively. </div>
<div><br>
</div>
<div>At the same time, I do wish there was a good solution. I
also thought about cases, where i could write an ssh wrapper
with bsub/qsub interactive job command that will allow users
to use compute nodes as interactive nodes for a while, to
compile/edit or submit there scripts but this would only be
easy if all the compute nodes can be directly reachable over
network, and not be restricted on a private network. </div>
<div><br>
</div>
<div>Thank you,</div>
<div>Lohit</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Mar 26, 2021 at 10:27
AM Prentice Bisbal via Beowulf <<a
href="mailto:beowulf@beowulf.org" moz-do-not-send="true">beowulf@beowulf.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yes,
there's a tool developed specifically for this called Arbiter
that <br>
uses Linux cgroups to dynamically limit resources on a login
node based <br>
on it's current load. It was developed at the University of
Utah:<br>
<br>
<a
href="https://dylngg.github.io/resources/arbiterTechPaper.pdf"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://dylngg.github.io/resources/arbiterTechPaper.pdf</a><br>
<br>
<a href="https://gitlab.chpc.utah.edu/arbiter2/arbiter2"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://gitlab.chpc.utah.edu/arbiter2/arbiter2</a><br>
<br>
Prentice<br>
<br>
On 3/26/21 9:56 AM, Michael Di Domenico wrote:<br>
> does anyone have a recipe for limiting the damage people
can do on<br>
> login nodes on rhel7. i want to limit the allocatable
cpu/mem per<br>
> user to some low value. that way if someone kicks off a
program but<br>
> forgets to 'srun' it first, they get bound to a single
core and don't<br>
> bump anyone else.<br>
><br>
> i've been poking around the net, but i can't find a
solution, i don't<br>
> understand what's being recommended, and/or i'm
implementing the<br>
> suggestions wrong. i haven't been able to get them
working. the most<br>
> succinct answer i found is that per user cgroup controls
have been<br>
> implemented in systemd v239/240, but since rhel7 is still
on v219<br>
> that's not going to help. i also found some wonkiness
that runs a<br>
> program after a user logs in and hacks at the cgroup
files directly,<br>
> but i couldn't get that to work.<br>
><br>
> supposedly you can override the user-{UID}.slice unit
file and jam in<br>
> the cgroup restrictions, but I have hundreds of users
clearly that's<br>
> not maintainable<br>
><br>
> i'm sure others have already been down this road. any
suggestions?<br>
> _______________________________________________<br>
> Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org" target="_blank"
moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by
Penguin Computing<br>
> To change your subscription (digest mode or unsubscribe)
visit <a
href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit
<a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Rémy Dernat
Chef de projet SI
IR CNRS - ISI / ISEM</pre>
<div id="grammalecte_menu_main_button_shadow_host" style="width:
0px; height: 0px;"></div>
</body>
</html>