[Beowulf] Suggestions to what DFS to use

[Jon Tegner]

BeeGFS sounds interesting. Is it possible to say something general about 
how it compares to Lustre regarding performance?

/jon

On 02/13/2017 05:54 PM, John Hanks wrote:
> We've had pretty good luck with BeeGFS lately running on SuperMicro 
> vanilla hardware with ZFS as the underlying filesystem. It works 
> pretty well for the cheap end of the hardware spectrum and BeeGFS is 
> free and pretty amazing. It has held up to abuse under a very mixed 
> and heavy workload and we can stream large sequential data into it 
> fast enough to saturate a QDR IB link, all without any in depth 
> tuning. While we don't have redundancy (other than raidz3), BeeGFS can 
> be set up with some redundancy between metadata servers and mirroring 
> between storage. http://www.beegfs.com/content/
>
> jbh
>
> On Mon, Feb 13, 2017 at 7:40 PM Alex Chekholko 
> <alex.chekholko at gmail.com <mailto:alex.chekholko at gmail.com>> wrote:
>
>     If you have a preference for Free Software, GlusterFS would work,
>     unless you have many millions of small files. It would also depend
>     on your available hardware, as there is not a 1-to-1
>     correspondence between a typical GPFS setup and a typical
>     GlusterFS setup. But at least it is free and easy to try out. The
>     mailing list is active, the software is now mature ( I last used
>     GlusterFS a few years ago) and you can buy support from Red Hat if
>     you like.
>
>     Take a look at the RH whitepapers about typical GlusterFS
>     architecture.
>
>     CephFS, on the other hand, is not yet mature enough, IMHO.
>     On Mon, Feb 13, 2017 at 8:31 AM Justin Y. Shi <shi at temple.edu
>     <mailto:shi at temple.edu>> wrote:
>
>         Maybe you would consider Scality (http://www.scality.com/) for
>         your growth concerns. If you need speed, DDN is faster in
>         rapid data ingestion and for extreme HPC data needs.
>
>         Justin
>
>         On Mon, Feb 13, 2017 at 4:32 AM, Tony Brian Albers <tba at kb.dk
>         <mailto:tba at kb.dk>> wrote:
>
>             On 2017-02-13 09:36, Benson Muite wrote:
>             > Hi,
>             >
>             > Do you have any performance requirements?
>             >
>             > Benson
>             >
>             > On 02/13/2017 09:55 AM, Tony Brian Albers wrote:
>             >> Hi guys,
>             >>
>             >> So, we're running a small(as in a small number of
>             nodes(10), not
>             >> storage(170TB)) hadoop cluster here. Right now we're on
>             IBM Spectrum
>             >> Scale(GPFS) which works fine and has POSIX support. On
>             top of GPFS we
>             >> have a GPFS transparency connector so that HDFS uses GPFS.
>             >>
>             >> Now, if I'd like to replace GPFS with something else,
>             what should I use?
>             >> It needs to be a fault-tolerant DFS, with POSIX
>             support(so that users
>             >> can move data to and from it with standard tools).
>             >>
>             >> I've looked at MooseFS which seems to be able to do the
>             trick, but are
>             >> there any others that might do?
>             >>
>             >> TIA
>             >>
>             >
>
>             Well, we're not going to be doing a huge amount of I/O. So
>             performance
>             requirements are not high. But ingest needs to be really
>             fast, we're
>             talking tens of terabytes here.
>
>             /tony
>
>             --
>             Best regards,
>
>             Tony Albers
>             Systems administrator, IT-development
>             Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C,
>             Denmark.
>             Tel: +45 2566 2383 <tel:%2B45%202566%202383> / +45 8946
>             2316 <tel:%2B45%208946%202316>
>             _______________________________________________
>             Beowulf mailing list, Beowulf at beowulf.org
>             <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>             To change your subscription (digest mode or unsubscribe)
>             visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>         _______________________________________________
>         Beowulf mailing list, Beowulf at beowulf.org
>         <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>         To change your subscription (digest mode or unsubscribe) visit
>         http://www.beowulf.org/mailman/listinfo/beowulf
>
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>
> -- 
> ‘[A] talent for following the ways of yesterday, is not sufficient to 
> improve the world of today.’
>  - King Wu-Ling, ruler of the Zhao state in northern China, 307 BC
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20170214/68d2b266/attachment-0001.html>