<div dir="ltr"><div dir="ltr">Will,<div><br>We setup BeeGFS for a potential scientist work and found it reasonably easy to setup. The software was built by admins, so administration was fairly easy and adding nodes worked well. We used SSD's for the metadata servers and regular SAS drives for data stores. In testing, we got some decent numbers:</div><div><br></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">dd to local disks, 1GB per second
dd to local disks, 1GB per second -- potential 2GB/s max throughput</span><br></div><div><br></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">dd to beegfs volume, sequential write, 1 stream, ~600MB/s
dd to beegfs volume, sequential write, 4 streams, 1700MB/s total, individual streams were slower
i iperf test to another node over IB, 1 stream, 800-900MB/s
iperf test to another node over IB, 4 stream, ~3200MB/s total, individual streams were slower</span><br></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap"><br></span></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">This was with two servers and 12x 10K SAS drives (using ZFS to pool them together) in each server, with SSD boot drive and SSD meta data drive. The one thing we did notice, over a long period of time, the system needed to be rebooted in order to maintain consistency. We were using it as a scratch space, so we didn't mind the problem.</span></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap"><br></span></div><div><font color="#333333" face="sans-serif"><span style="font-size:14px;white-space:pre-wrap">We talked to a local vendor and they liked the technology, but they did recommend having a support contract for the system. Also, have BeeGFS help you with the initial setup of the system. </span></font></div><div><br></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">This information was from 4 years ago.</span></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">
Alas, the scientist didn't end up here, so we keep the information on file and hopefully someday we will re-engage.</span></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">
Good luck,</span></div><div><span style="color:rgb(51,51,51);font-family:sans-serif;font-size:14px;white-space:pre-wrap">Craig</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 18, 2019 at 12:02 PM Prentice Bisbal via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Will, <br>
</p>
<p>Several years ago,when I was at Rutgers, Joe Landman's company,
Scalable Informatics (RIP), was trying to sell be on BeeGFS over
Lustre and GPFS. At the time, I was not interested. Why not?
BeeGFS was still relatively new, and Lustre and GPFS had larger
install bases, and therefore bigger track records. I was the only
System admin in a group with aspirations to be the one-stop shop
for a very large research institution, and to become a
national-level HPC center. As a result, I was more risk-adverse
than I normally would be. I didn't want to take a risk with a
relatively unproven system, no matter how good it's performance
was. I also wanted to use a system where there was an abundance of
other sys admins with expertise I could lean on if I needed to. <br>
</p>
<p>Fast forward 4-5 years, and the situation has completely changed.
At SC18, it seemed every booth was using or promoting BeeGFS, and
everyone was saying good things about it. If were in the same
situation today, I wouldn't hesitate to consider BeeGFS. <br>
</p>
<p>In fact, I feel bad for not giving it a closer look at the time,
because it's clear Joe and his team at his late company were on to
something and were clearly ahead of their time with promoting
BeeGFS. <br>
</p>
<p>--<br>
Prentice<br>
</p>
<p><br>
</p>
<div class="gmail-m_-8812320764687312380moz-cite-prefix">On 3/18/19 11:50 AM, Will Dennis wrote:<br>
</div>
<blockquote type="cite">
<div class="gmail-m_-8812320764687312380WordSection1">
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am considering using BeeGFS for a
parallel file system for one (and if successful, more) of our
clusters here. Just wanted to get folks’ opinions on that, and
if there is any “gotchas” or better-fit solutions out there...
The first cluster I am considering it for has ~50TB storage
off a single ZFS server serving the data over NFS currently;
looking to increase not only storage capacity, but also I/O
speed. The cluster nodes that are consuming the storage have
10GbaseT interconnects, as does the ZFS server. As we are a
smaller shop, want to keep the solution simple. BeeGFS was
recommended to me as a good solution off another list, and
wanted to get people’s opinions off this list.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">Will<u></u><u></u></p>
</div>
<br>
<fieldset class="gmail-m_-8812320764687312380mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-8812320764687312380moz-quote-pre">_______________________________________________
Beowulf mailing list, <a class="gmail-m_-8812320764687312380moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="gmail-m_-8812320764687312380moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowul" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowul</a></pre>
</blockquote>
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">___________________________________________<br>Craig Andrew<br>Manager of Systems Administration<br>Whitehead Institute for Biomedical Research<br></div></div></div>