[Beowulf] Lustre on google cloud

[Chris Dagdigian]

A lot of production HPC runs on cloud systems.

AWS is big for this via their AWS Parallelcluster stack which does 
include lustre support via vfXT for lustre service although they are 
careful to caveat it as staging/scratch space not suitable for 
persistant storage.  AWS has some cool node types now with 25gig, 50gig 
and 100-gigabit network support.

Microsoft Azure is doing amazing things now that they have the 
cyclecomputing folks on board, integrated and able to call shots within 
the product space. They actually offer bare metal HPC and infiniband 
SKUs now and have some interesting parallel filesystem offerings as well.

Can't comment on google as I've not touched or used it professionally 
but AWS and Azure for sure are real players now to consider if you have 
an HPC requirement.


That said, however, a sober cost accounting still shows on-prem or 
"owned' HPC is best from a financial perspective if your workload is 
24x7x365 constant.  The cloud based HPC is best for capability,  bursty 
workloads, temporary workloads, auto-scaling, computing against 
cloud-resident data sets or the neat new model where instead of on-prem 
multi-user shared HPC you go out and decide to deliver individual 
bespoke HPC clusters to each user or team on the cloud.

The big paradigm shift for cloud HPC is that it does not make a lot of 
sense to make a monolithic stack shared by multiple competing users and 
groups. The automated provisioning and elasticity of the cloud make it 
more sensible to build many clusters so that you can tune each cluster 
specifically for the cluster or workload and then blow it up when the 
work is done.

My $.02 of course!

Chris


> Jonathan Aquilina <mailto:jaquilina at eagleeyet.net>
> July 22, 2019 at 1:48 PM
>
> Hi Guys,
>
> I am looking at 
> https://cloud.google.com/blog/products/storage-data-transfer/introducing-lustre-file-system-cloud-deployment-manager-scripts
>
> This basically allows you to deploy a lustre cluster on google cloud. 
> In your HPC setups have you considered moving towards cloud based 
> clusters?
>
> Regards,
>
> Jonathan
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20190722/95d3197e/attachment.html>