[Beowulf] Considering BeeGFS for parallel file system

[Craig Andrew]

Will,

We setup BeeGFS for a potential scientist work and found it reasonably easy
to setup. The software was built by admins, so administration was fairly
easy and adding nodes worked well. We used SSD's for the metadata servers
and regular SAS drives for data stores. In testing, we got some decent
numbers:

dd to local disks, 1GB per second dd to local disks, 1GB per second --
potential 2GB/s max throughput

dd to beegfs volume, sequential write, 1 stream, ~600MB/s dd to beegfs
volume, sequential write, 4 streams, 1700MB/s total, individual streams
were slower i iperf test to another node over IB, 1 stream, 800-900MB/s
iperf test to another node over IB, 4 stream, ~3200MB/s total, individual
streams were slower

This was with two servers and 12x 10K SAS drives (using ZFS to pool them
together) in each server, with SSD boot drive and SSD meta data drive. The
one thing we did notice, over a long period of time, the system needed to
be rebooted in order to maintain consistency. We were using it as a scratch
space, so we didn't mind the problem.

We talked to a local vendor and they liked the technology, but they did
recommend having a support contract for the system. Also, have BeeGFS help
you with the initial setup of the system.

This information was from 4 years ago.
Alas, the scientist didn't end up here, so we keep the information on file
and hopefully someday we will re-engage.
Good luck,
Craig

On Mon, Mar 18, 2019 at 12:02 PM Prentice Bisbal via Beowulf <
beowulf at beowulf.org> wrote:

> Will,
>
> Several years ago,when I was at Rutgers, Joe Landman's company, Scalable
> Informatics (RIP), was trying to sell be on BeeGFS over Lustre and GPFS. At
> the time, I was not interested. Why not? BeeGFS was still relatively new,
> and Lustre  and GPFS had larger install bases, and therefore bigger track
> records. I was the only System admin in a group with aspirations to be the
> one-stop shop for a very large research institution, and to become a
> national-level HPC center. As a result, I was more risk-adverse than I
> normally would be.  I didn't want to take a risk with a relatively unproven
> system, no matter how good it's performance was. I also wanted to use a
> system where there was an abundance of other sys admins with expertise I
> could lean on if I needed to.
>
> Fast forward 4-5 years, and the situation has completely changed. At SC18,
> it seemed every booth was using or promoting BeeGFS, and everyone was
> saying good things about it. If were in the same situation today, I
> wouldn't hesitate to consider BeeGFS.
>
> In fact, I feel bad for not giving it a closer look at the time, because
> it's clear Joe and his team at his late company were on to something and
> were clearly ahead of their time with promoting BeeGFS.
>
> --
> Prentice
>
>
> On 3/18/19 11:50 AM, Will Dennis wrote:
>
> Hi all,
>
>
>
> I am considering using BeeGFS for a parallel file system for one (and if
> successful, more) of our clusters here. Just wanted to get folks’ opinions
> on that, and if there is any “gotchas” or better-fit solutions out there...
> The first cluster I am considering it for has ~50TB storage off a single
> ZFS server serving the data over NFS currently; looking to increase not
> only storage capacity, but also I/O speed. The cluster nodes that are
> consuming the storage have 10GbaseT interconnects, as does the ZFS server.
> As we are a smaller shop, want to keep the solution simple. BeeGFS was
> recommended to me as a good solution off another list, and wanted to get
> people’s opinions off this list.
>
>
>
> Thanks!
>
> Will
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowul
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>


-- 
___________________________________________
Craig Andrew
Manager of Systems Administration
Whitehead Institute for Biomedical Research
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20190318/2c5aa3a9/attachment-0001.html>