<div dir="ltr"><div><div><div><div><div>Thanks Robert, <br><br></div>You have given me a lot to think about. <br></div>Most of our nodes have around 250GB SSDs largely unpopulated so I am guessing there is no harm in just installing the libraries in every node with ansible. Also, in our department we have a wealth of old HDDs we could repurpose <br></div>My time indeed has a cost, hence I will favour a "cheap and dirty" solution to get the ball rolling and try something fancy later. <br></div>Though I was intrigued by your tip about LXC, I have used LXC locally on my workstation for the longest time, but I have not considered running it on a Beowulf cluster context, that would be a neat thing to investigate. <br><br></div><div>anyway thanks for the tips<br></div><div><br></div><div>Cheers</div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 28 Jun 2022 at 16:01, Robert G. Brown <<a href="mailto:rgb@phy.duke.edu">rgb@phy.duke.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, 28 Jun 2022, leo camilo wrote:<br>
<br>
> I see, so if I understand it correctly I have to make sure that there is a<br>
> copy of the library, environments and modules on every computational node?<br>
> <br>
> I am wondering if I can get around it by using nfs.<br>
<br>
The answer is yes, although it is a bit of a pain.<br>
<br>
Two ways to proceed:<br>
<br>
Export the library directory(s) from your head node -- at least /usr/lib<br>
(this assumes, BTW, that the head node and worker nodes are running<br>
exactly the same version of linux updated to exactly the same level --<br>
especially the kernel). Mount it on an alternative path e.g.<br>
/usr/local/lib or /usr/work/lib e.g. during/after boot. Learn how to<br>
use ldconfig and run it to teach the kernel how to find the libraries<br>
there. This approach is simple in that you don't need to worry about<br>
whether or not any particular library is there or isn't there -- you are<br>
provisioning "everything" present on your head node, so if it works one<br>
place it works everywhere else.<br>
<br>
The second way may be easier if you are already exporting e.g. a home<br>
directory or work directory, and only need to provision a few<br>
applications. Use Unix tools (specifically ldd) to figure out what<br>
libraries are needed for your application. Put copies of those<br>
libraries in a "personal" link library directory -- e.g.<br>
/home/joeuser/lib -- and again, use ldconfig as part of your startup/login<br>
script(s) to teach the kernel where to find them when you run your<br>
application.<br>
<br>
A third way is to look into containers -- <a href="https://linuxcontainers.org/" rel="noreferrer" target="_blank">https://linuxcontainers.org/</a><br>
-- which allow you to build "containerized" binaries that contain all of<br>
their dependencies and in principle run across DIFFERENT linuces,<br>
kernels, update levels, etc. The idea there is a containerized app<br>
doesn't depend directly on the parent operating system "at all" beyond<br>
running on the right CPU. An immediate advantage is that if somebody<br>
decides to change or drop some key library in the future, you don't<br>
care. It's containerized. I have only played with them a bit, mind<br>
you, but they are supposedly pretty robust and suitable for commercial<br>
cloud apps etc so they should be fine for you too.<br>
<br>
A fourth way -- and this would be my own preference -- would be to just<br>
install the requisite libraries on the worker nodes (all of which should<br>
be automagically updated from the primary repo streams anyway to remain<br>
consistent and up to date). Hard storage is sooooo cheap. You could<br>
put the entire functional part of the OS including all libraries on<br>
every system for $10 to $20 via a USB thumb drive, assuming that the<br>
worker nodes don't ALREADY have enormous amounts of unused space. Speed<br>
is not likely to be a major issue here as the OS will cache the<br>
libraries after the initial load assuming that your nodes are<br>
well-provisioned with RAM, and it has to load the application itself<br>
anyway. I can't think of a good reason any more -- with TB hard drives<br>
very nearly the SMALLEST ones readily available to limit what you put on<br>
a worker node unless you are trying to run it entirely diskless (and for<br>
the cost, why would you want to do that?).<br>
<br>
Remember, YOUR TIME has a cost. You have 7 worker nodes. Putting a 128<br>
GB hard drive on the USB port of each will cost you (say) $15 each, for<br>
a total of $105 -- assuming that somehow the nodes currently have only 8<br>
GB and can't easily hold the missing libraries "permanently". I did<br>
beowulfery back in the day when storage WAS expensive, and run entirely<br>
diskless nodes in many cases that booted from the network, and I assure<br>
you, it is a pain in the ass and pointless when storage is less than<br>
$0.10/GB. There is simply no point in installing "limited" worker<br>
nodes, picking and choosing what libraries to include or trying to<br>
assemble and OS image that lacks e.g. GUI support just because you won't<br>
be putting a monitor and keyboard on them. Just come up with a standard<br>
post-install script to run after you do the primary OS install to e.g.<br>
"dnf -y install gsl" to add in the Gnu scientific library or whatever<br>
and ensure that the nodes are all updated at the same time for<br>
consistency, then forget it.<br>
<br>
rgb<br>
<br>
> <br>
> On Tue, 28 Jun 2022 at 11:42, Richard <<a href="mailto:ejb@trick-1.net" target="_blank">ejb@trick-1.net</a>> wrote:<br>
> For what it?s worth I use an easy8 licensed bright cluster (now<br>
> part of NVidia) and I continually find I need to make sure the<br>
> module packages, environment variables etc are installed/set in<br>
> the images that are deployed to the nodes<br>
><br>
> Bright supports slurm, k8, jupyter and a lot more<br>
><br>
> Richard<br>
><br>
> Sent from my iPhone<br>
><br>
> > On 28 Jun 2022, at 19:32, leo camilo <<a href="mailto:lhcamilo@gmail.com" target="_blank">lhcamilo@gmail.com</a>><br>
> wrote:<br>
> ><br>
> ><br>
> > # Background<br>
> ><br>
> > So, I am building this small beowulf cluster for my<br>
> department. I have it running on ubuntu servers, a front node<br>
> and at the moment 7 x 16 core nodes. I have installed SLURM as<br>
> the scheduler and I have been procrastinating to setup<br>
> environment modules.<br>
> ><br>
> > In any case, I ran in this particular scenario where I was<br>
> trying to schedule a few jobs in slurm, but for some reason<br>
> slurm would not find this library (libgsl). But it was in fact<br>
> installed in the frontnode, I checked the path with ldd and I<br>
> even exported the LD_LIBRARY_PATH .<br>
> ><br>
> > Oddly, if I ran the application directly in the frontnode, it<br>
> would work fine.,<br>
> ><br>
> > Though it occured to me that the computational nodes might not<br>
> have this library and surely once I installed this library in<br>
> the nodes the problem went away.<br>
> ><br>
> > # Question:<br>
> ><br>
> > So here is the question, is there a way to cache the<br>
> frontnode's libraries and environment onto the computational<br>
> nodes when a slurm job is created?<br>
> ><br>
> > Will environment modules do that? If so, how?<br>
> ><br>
> > Thanks in advance,<br>
> ><br>
> > Cheers<br>
> > _______________________________________________<br>
> > Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin<br>
> Computing<br>
> > To change your subscription (digest mode or unsubscribe) visit<br>
> <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
> <br>
> <br>
><br>
<br>
Robert G. Brown <a href="http://www.phy.duke.edu/~rgb/" rel="noreferrer" target="_blank">http://www.phy.duke.edu/~rgb/</a><br>
Duke University Dept. of Physics, Box 90305<br>
Durham, N.C. 27708-0305<br>
Phone: 1-919-660-2567 Fax: 919-660-2525 <a href="mailto:email%3Argb@phy.duke.edu" target="_blank">email:rgb@phy.duke.edu</a><br>
<br>
<br>
</blockquote></div>