[Beowulf] Suggestions to what DFS to use

[Bogdan Costescu]

I can second the recommendation for BeeGFS. We have it in use for ~4
years with very good results, by now on 3 different FSes. We also run
it on SuperMicro hardware and Infiniband, but use the "classic"
combination with ext4 for metadata and xfs for storage servers - of
course, with RAID controllers underneath :) With 6TB SATA disks, we
get around 330TB usable capacity per storage server, which is composed
of an actual server and a JBOD unit. Maybe the most useful feature for
us is that it can be very easily expanded by adding another such
storage server, though this could become a bit problematic due to the
lack of rebalancing. (please note: I'm not saying that other FSes
don't have this feature, just that it was very useful for us :)) We
too see high performance under a very mixed load without much tuning,
however this also has a reverse side: it puts a high load on the
hardware and the software stack underneath, exposing faults in them,
such that f.e. we needed to downgrade the IB stack in one case, or
upgrade the firmware of the RAID controller and SSDs on a metadata
server in another case. The users are especially happy about the
metadata performance when working with very many small files;
streaming reads or writes of large data is basically limited by
hardware or underlying stacks, not by BeeGFS.

Cheers,
Bogdan

On Mon, Feb 13, 2017 at 5:54 PM, John Hanks <griznog at gmail.com> wrote:
> We've had pretty good luck with BeeGFS lately running on SuperMicro vanilla
> hardware with ZFS as the underlying filesystem. It works pretty well for the
> cheap end of the hardware spectrum and BeeGFS is free and pretty amazing. It
> has held up to abuse under a very mixed and heavy workload and we can stream
> large sequential data into it fast enough to saturate a QDR IB link, all
> without any in depth tuning. While we don't have redundancy (other than
> raidz3), BeeGFS can be set up with some redundancy between metadata servers
> and mirroring between storage. http://www.beegfs.com/content/
>
> jbh
>
> On Mon, Feb 13, 2017 at 7:40 PM Alex Chekholko <alex.chekholko at gmail.com>
> wrote:
>>
>> If you have a preference for Free Software, GlusterFS would work, unless
>> you have many millions of small files. It would also depend on your
>> available hardware, as there is not a 1-to-1 correspondence between a
>> typical GPFS setup and a typical GlusterFS setup. But at least it is free
>> and easy to try out. The mailing list is active, the software is now mature
>> ( I last used GlusterFS a few years ago) and you can buy support from Red
>> Hat if you like.
>>
>> Take a look at the RH whitepapers about typical GlusterFS architecture.
>>
>> CephFS, on the other hand, is not yet mature enough, IMHO.
>> On Mon, Feb 13, 2017 at 8:31 AM Justin Y. Shi <shi at temple.edu> wrote:
>>>
>>> Maybe you would consider Scality (http://www.scality.com/) for your
>>> growth concerns. If you need speed, DDN is faster in rapid data ingestion
>>> and for extreme HPC data needs.
>>>
>>> Justin
>>>
>>> On Mon, Feb 13, 2017 at 4:32 AM, Tony Brian Albers <tba at kb.dk> wrote:
>>>>
>>>> On 2017-02-13 09:36, Benson Muite wrote:
>>>> > Hi,
>>>> >
>>>> > Do you have any performance requirements?
>>>> >
>>>> > Benson
>>>> >
>>>> > On 02/13/2017 09:55 AM, Tony Brian Albers wrote:
>>>> >> Hi guys,
>>>> >>
>>>> >> So, we're running a small(as in a small number of nodes(10), not
>>>> >> storage(170TB)) hadoop cluster here. Right now we're on IBM Spectrum
>>>> >> Scale(GPFS) which works fine and has POSIX support. On top of GPFS we
>>>> >> have a GPFS transparency connector so that HDFS uses GPFS.
>>>> >>
>>>> >> Now, if I'd like to replace GPFS with something else, what should I
>>>> >> use?
>>>> >> It needs to be a fault-tolerant DFS, with POSIX support(so that users
>>>> >> can move data to and from it with standard tools).
>>>> >>
>>>> >> I've looked at MooseFS which seems to be able to do the trick, but
>>>> >> are
>>>> >> there any others that might do?
>>>> >>
>>>> >> TIA
>>>> >>
>>>> >
>>>>
>>>> Well, we're not going to be doing a huge amount of I/O. So performance
>>>> requirements are not high. But ingest needs to be really fast, we're
>>>> talking tens of terabytes here.
>>>>
>>>> /tony
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Tony Albers
>>>> Systems administrator, IT-development
>>>> Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
>>>> Tel: +45 2566 2383 / +45 8946 2316
>>>> _______________________________________________
>>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>>> To change your subscription (digest mode or unsubscribe) visit
>>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> ‘[A] talent for following the ways of yesterday, is not sufficient to
> improve the world of today.’
>  - King Wu-Ling, ruler of the Zhao state in northern China, 307 BC
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>