[Beowulf] Lustre Upgrades

[Paul Edmon]

This was several years back so the current version of Gluster may be in 
better shape.  We tried to use it for our primary storage but ran into 
scalability problems.  It especially was the case when it came to 
healing bricks and doing replication.  It just didn't scale well.  
Eventually we abandoned it for NFS and Lustre, NFS for deep storage and 
Lustre for performance.  We tried it for hosting VM images which worked 
pretty well but we've since moved to Ceph for that.

Anyways I have no idea about current Gluster in terms of scalability so 
the issues we ran into may not be an problem anymore.  However it has 
made us very gun shy about trying Gluster again.  Instead we've decided 
to use Ceph as we've gained a bunch of experience with Ceph in our 
OpenNebula installation.

-Paul Edmon-

On 07/24/2018 11:02 AM, John Hearns via Beowulf wrote:
> Paul, thanks for the reply.
> I would like to ask, if I may. I rather like Glustre, but have not 
> deployed it in HPC. I have heard a few people comment about Gluster 
> not working well in HPC. Would you be willing to be more specific?
>
> One research site I talked to did the classic 'converged 
> infrastructure' idea of attaching storage drives to their compute 
> nodes and distributing Glustre storage. They were not happy with that 
> IW as told, and I can very much understand why. But Gluster on 
> dedicated servers I would be interested to hear about.
>
>
> On Tue, 24 Jul 2018 at 16:41, Paul Edmon <pedmon at cfa.harvard.edu 
> <mailto:pedmon at cfa.harvard.edu>> wrote:
>
>     While I agree with you in principle, one also has to deal with the
>     reality as you find yourself in.  In our case we have more
>     experience with Lustre than Ceph in an HPC and we got burned
>     pretty badly by Gluster.  While I like Ceph in principle I haven't
>     seen it do what Lustre can do in a HPC setting over IB.  Now it
>     may be able to do that, which is great.  However then you have to
>     get your system set up to do that and prove that it can.  After
>     all users have a funny way of breaking things that work amazingly
>     well in controlled test environs, especially when you have no
>     control how they will actually use the system (as in a research
>     environment).  Certainly we are working on exploring this option
>     too as it would be awesome and save many headaches.
>
>     Anyways no worries about you being a smartarse, it is a valid
>     point.  One just needs to consider the realities on the ground in
>     ones own environment.
>
>     -Paul Edmon-
>
>
>     On 07/24/2018 10:31 AM, John Hearns via Beowulf wrote:
>>     Forgive me for saying this, but the philosophy for software
>>     defined storage such as CEPH and Gluster is that forklift style
>>     upgrades should not be necessary.
>>     When a storage server is to be retired the data is copied onto
>>     the new server then the old one taken out of service. Well,
>>     copied is not the correct word, as there are erasure-coded copies
>>     of the data. Rebalanced is probaby a better word.
>>
>>     Sorry if I am seeming to be a smartarse. I have gone through the
>>     pain of forklift style upgrades in the past when storage arrays
>>     reach End of Life.
>>     I just really like the Software Defined Storage mantra - no
>>     component should be a point of failure.
>>
>>
>>     _______________________________________________
>>     Beowulf mailing list,Beowulf at beowulf.org <mailto:Beowulf at beowulf.org>  sponsored by Penguin Computing
>>     To change your subscription (digest mode or unsubscribe) visithttp://www.beowulf.org/mailman/listinfo/beowulf
>
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180724/49fe7234/attachment-0001.html>