<div dir="ltr"><div><div>hi there, <br><br></div>Thanks for the advice. <br><br></div><div>From the messages here I think I have grokked on how to proceed</div><div><br></div><div>- Swap the HDDs with NVME<br></div><div>- replace 1GB ethernet IB <br>- Configure NFS to use IPoIB or RDMA<br></div><div>- Tune NFS <br><br></div><div>I will need to get my hands on Lustre eventually, but that can wait. <br><br></div><div>Thanks for the help<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 10 Aug 2023 at 23:47, Renfro, Michael <<a href="mailto:Renfro@tntech.edu">Renfro@tntech.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-8163274010129663889">
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="m_-8163274010129663889WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">As the definitely-not-proud owner of a 2016 purchase of a 60-bay disk shelf attached to a single server with an Infiniband connection back to 54 compute nodes, NFS on spinning disks can definitely handle 5
40-core jobs, but your particular setup really can’t. Mine has hit its limits at times as well, but it’s about the IOPS from the disk array, the speed of the SAS cable connecting the disk shelf to the server, everything *<b>but</b>* NFS itself.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Swapping to NVMe should make a world of difference on its own, as long as you don’t have a bottleneck of 1 Gb Ethernet between your storage and the compute capacity.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div id="m_-8163274010129663889mail-editor-reference-message-container">
<div>
<div style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Beowulf <<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a>> on behalf of leo camilo <<a href="mailto:lhcamilo@gmail.com" target="_blank">lhcamilo@gmail.com</a>><br>
<b>Date: </b>Thursday, August 10, 2023 at 3:04 PM<br>
<b>To: </b>Jeff Johnson <<a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a>><br>
<b>Cc: </b>Bernd Schubert <<a href="mailto:bernd.schubert@fastmail.fm" target="_blank">bernd.schubert@fastmail.fm</a>>, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> <<a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a>><br>
<b>Subject: </b>Re: [Beowulf] NFS alternative for 200 core compute (beowulf) cluster<u></u><u></u></span></p>
</div>
<p style="margin:0in;text-align:center;background:white" align="center"><b><span style="font-size:12pt;color:red;background:white">External Email Warning</span></b><u></u><u></u></p>
<p style="margin-right:12pt;margin-bottom:0in;margin-left:12pt;text-align:center;background:white" align="center">
<b><span style="font-size:12pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><u></u><u></u></p>
<div class="MsoNormal" style="text-align:center" align="center"><span style="font-size:11pt">
<hr width="100%" size="0" align="center">
</span></div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">Awesome, thanks for the info!
<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt">Best, <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">leo<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Thu, 10 Aug 2023 at 22:01, Jeff Johnson <<a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal"><span style="font-size:11pt">Leo, <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Both BeeGFS and Lustre require a backend file system on the disks themselves. Both Lustre and BeeGFS support ZFS backend. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">--Jeff<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Thu, Aug 10, 2023 at 1:00 PM leo camilo <<a href="mailto:lhcamilo@gmail.com" target="_blank">lhcamilo@gmail.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">Hi there,
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">thanks for your response. <u></u>
<u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">BeeGFS indeed looks like a good call option, though realistically I can only afford to use a single node/server for it.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">Would it be feasible to use zfs as volume manager coupled with BeeGFS for the shares, or should I write zfs off all together?
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">thanks again,
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">best,
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">leo<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Thu, 10 Aug 2023 at 21:29, Bernd Schubert <<a href="mailto:bernd.schubert@fastmail.fm" target="_blank">bernd.schubert@fastmail.fm</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt"><br>
<br>
On 8/10/23 21:18, leo camilo wrote:<br>
> Hi everyone,<br>
> <br>
> I was hoping I would seek some sage advice from you guys.<br>
> <br>
> At my department we have build this small prototyping cluster with 5 <br>
> compute nodes,1 name node and 1 file server.<br>
> <br>
> Up until now, the name node contained the scratch partition, which <br>
> consisted of 2x4TB HDD, which form an 8 TB striped zfs pool. The pool is <br>
> shared to all the nodes using nfs. The compute nodes and the name node <br>
> and compute nodes are connected with both cat6 ethernet net cable and <br>
> infiniband. Each compute node has 40 cores.<br>
> <br>
> Recently I have attempted to launch computation from each node (40 tasks <br>
> per node), so 1 computation per node. And the performance was abysmal. <br>
> I reckon I might have reached the limits of NFS.<br>
> <br>
> I then realised that this was due to very poor performance from NFS. I <br>
> am not using stateless nodes, so each node has about 200 GB of SSD <br>
> storage and running directly from there was a lot faster.<br>
> <br>
> So, to solve the issue, I reckon I should replace NFS with something <br>
> better. I have ordered 2x4TB NVMEs for the new scratch and I was <br>
> thinking of :<br>
> <br>
> * using the 2x4TB NVME in a striped ZFS pool and use a single node<br>
> GlusterFS to replace NFS<br>
> * using the 2x4TB NVME with GlusterFS in a distributed arrangement<br>
> (still single node)<br>
> <br>
> Some people told me to use lustre,but I reckon that might be overkill. <br>
> And I would only use a single fileserver machine(1 node).<br>
> <br>
> Could you guys give me some sage advice here?<br>
> <br>
<br>
So glusterfs is using fuse, which doesn't have the best performance <br>
reputation (although hopefully not for long - feel free to search for <br>
"fuse" + "uring").<br>
<br>
If you want to avoid complexity of Lustre, maybe look into BeeGFS. Well, <br>
I would recommend to look into it anyway (as former developer I'm biased <br>
again ;) ).<br>
<br>
<br>
Cheers,<br>
Bernd<u></u><u></u></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11pt">_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><u></u><u></u></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span class="m_-8163274010129663889gmailsignatureprefix"><span style="font-size:11pt">--
</span></span><span style="font-size:11pt"><u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a><br>
<a href="http://www.aeoncomputing.com/" target="_blank">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f: 858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite C - San Diego, CA 92117 <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">High-Performance Computing / Lustre Filesystems / Scale-out Storage<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>