<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I'm surprised no one here has mentioned tuning kernel/network
parameters. I would take at which of these parameters you can tune
to improve performance first because it's free, quick, and the
least labor-intensive way to improve performance. I would take a
look at the website below and see what parameters you can tweak to
improve your performance. <br>
</p>
<p><a class="moz-txt-link-freetext" href="https://fasterdata.es.net/">https://fasterdata.es.net/</a><br>
</p>
<pre class="moz-signature" cols="72">Prentice </pre>
<div class="moz-cite-prefix">On 8/10/23 3:35 PM, Jeff Johnson wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFCYAsesubJORz4y_8SY0CvPcV+oaMuBcoiNftW038LxtP-Tug@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Leo,
<div><br>
</div>
<div>NFS can be a hindrance but if tuned and configured properly
might not be as terrible. Some thoughts...</div>
<div>
<ul>
<li>What interface are the nodes accessing NFS via? Ethernet
or Infiniband?</li>
<li>Have you tuned the number of NFS server threads above
defaults? </li>
<li>As a test, you could deploy a single Lustre node that
would act as MGS/MDS and OSS simultaneously to test for
performance gains via Infiniband. </li>
<li>Your scratch volume must really be scratch because you
are running with no parity protection (two disk os SSD
stripe)</li>
<li>You're probably better off with tuned NFS as opposed to
GlusterFS</li>
</ul>
<div>--Jeff</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at
12:19 PM leo camilo <<a href="mailto:lhcamilo@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">lhcamilo@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi everyone, <br>
<br>
</div>
I was hoping I would seek some sage advice from
you guys. <br>
<br>
</div>
At my department we have build this small
prototyping cluster with 5 compute nodes,1 name
node and 1 file server. <br>
<br>
</div>
Up until now, the name node contained the scratch
partition, which consisted of 2x4TB HDD, which form
an 8 TB striped zfs pool. The pool is shared to all
the nodes using nfs. The compute nodes and the name
node and compute nodes are connected with both cat6
ethernet net cable and infiniband. Each compute node
has 40 cores.<br>
<br>
</div>
Recently I have attempted to launch computation from
each node (40 tasks per node), so 1 computation per
node. And the performance was abysmal. I reckon I
might have reached the limits of NFS.<br>
<br>
</div>
I then realised that this was due to very poor
performance from NFS. I am not using stateless nodes, so
each node has about 200 GB of SSD storage and running
directly from there was a lot faster. <br>
<br>
</div>
So, to solve the issue, I reckon I should replace NFS
with something better. I have ordered 2x4TB NVMEs for the
new scratch and I was thinking of :<br>
<br>
</div>
<ul>
<li>using the 2x4TB NVME in a striped ZFS pool and use a
single node GlusterFS to replace NFS</li>
<li>using the 2x4TB NVME with GlusterFS in a distributed
arrangement (still single node)</li>
</ul>
<div>Some people told me to use lustre,but I reckon that
might be overkill. And I would only use a single
fileserver machine(1 node).<br>
<br>
</div>
<div>Could you guys give me some sage advice here?<br>
<br>
</div>
<div>Thanks in advance<br>
</div>
<div>
<div>
<div>
<div>
<div><br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit
<a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a href="mailto:jeff.johnson@aeoncomputing.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">jeff.johnson@aeoncomputing.com</a><br>
<a href="http://www.aeoncomputing.com" target="_blank"
moz-do-not-send="true">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f: 858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite C - San Diego, CA 92117
<div><br>
</div>
<div>High-Performance Computing / Lustre Filesystems /
Scale-out Storage</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
</body>
</html>