[Beowulf] Lustre Upgrades

[Joe Landman]

On 07/25/2018 04:36 PM, Prentice Bisbal wrote:
>
> Paging Dr. Joe Landman, paging Dr. Landman...
>

My response was

"I'd seen/helped build/benchmarked some very nice/fast CephFS based 
storage systems in $dayjob-1.  While it is a neat system, if you are 
focused on availability, scalability, and performance, its pretty hard 
to beat BeeGFS.  We'd ($dayjob-1) deployed several very large/fast file 
systems with it on our spinning rust, SSD, and NVMe units."

at the bottom of the post.

Yes, BeeGFS compares very favorably to Lustre across performance, 
management, resiliency dimensions.  Distributed replicated metadata and 
data is possible, atop zfs, xfs, etc.  We sustained >  40GB/s in a 
single rack of spinning disk in 2014 at a customer site using it, no 
SSD/cache implicated, and using 56Gb IB throughout.  Customer wanted to 
see us sustain 46+GB/s writes, and we did.

These are some of our other results with it:

https://scalability.org/2014/05/massive-unapologetic-firepower-2tb-write-in-73-seconds/

https://scalability.org/2014/10/massive-unapologetic-firepower-part-2-the-dashboard/
(that was my first effort with Grafana, and look at the writes ... 
vertical scale is 10k MB/s, aka 10GB/s increments.

W.r.t. BeeGFS, very easy to install, you can set it up trivially on 
extra hardware to see it in action.  Won't be as fast as my old stuff, 
but that's the price people pay for not buying the good stuff when it 
was available.

> Prentice
> On 07/24/2018 10:19 PM, James Burton wrote:
>> Does anyone have any experience with how BeeGFS compares to Lustre? 
>> We're looking at both of those for our next generation HPC storage 
>> system.
>>
>> Is CephFS a valid option for HPC now? Last time I played with CephFS 
>> it wasn't ready for prime time, but that was a few years ago.
>>
>> On Tue, Jul 24, 2018 at 10:58 AM, Joe Landman <joe.landman at gmail.com 
>> <mailto:joe.landman at gmail.com>> wrote:
>>
>>
>>
>>     On 07/24/2018 10:31 AM, John Hearns via Beowulf wrote:
>>
>>         Forgive me for saying this, but the philosophy for software
>>         defined storage such as CEPH and Gluster is that forklift
>>         style upgrades should not be necessary.
>>         When a storage server is to be retired the data is copied
>>         onto the new server then the old one taken out of service.
>>         Well, copied is not the correct word, as there are
>>         erasure-coded copies of the data. Rebalanced is probaby a
>>         better word.
>>
>>
>>     This ^^
>>
>>     I'd seen/helped build/benchmarked some very nice/fast CephFS
>>     based storage systems in $dayjob-1.  While it is a neat system,
>>     if you are focused on availability, scalability, and performance,
>>     its pretty hard to beat BeeGFS.  We'd ($dayjob-1) deployed
>>     several very large/fast file systems with it on our spinning
>>     rust, SSD, and NVMe units.
>>
>>
>>     -- 
>>     Joe Landman
>>     e: joe.landman at gmail.com <mailto:joe.landman at gmail.com>
>>     t: @hpcjoe
>>     w: https://scalability.org
>>     g: https://github.com/joelandman
>>     l: https://www.linkedin.com/in/joelandman
>>     <https://www.linkedin.com/in/joelandman>
>>
>>
>>     _______________________________________________
>>     Beowulf mailing list, Beowulf at beowulf.org
>>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>>     To change your subscription (digest mode or unsubscribe) visit
>>     http://www.beowulf.org/mailman/listinfo/beowulf
>>     <http://www.beowulf.org/mailman/listinfo/beowulf>
>>
>>
>>
>>
>> -- 
>> James Burton
>> OS and Storage Architect
>> Advanced Computing Infrastructure
>> Clemson University Computing and Information Technology
>> 340 Computer Court
>> Anderson, SC 29625
>> (864) 656-9047
>>
>>
>> _______________________________________________
>> Beowulf mailing list,Beowulf at beowulf.org  sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visithttp://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joe Landman
e: joe.landman at gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman