[Beowulf] HPC workflows

[Gerald Henriksen]

On Tue, 27 Nov 2018 07:51:06 -0500, you wrote:

>On Mon, Nov 26, 2018 at 9:50 PM Gerald Henriksen <ghenriks at gmail.com> wrote:
>> On Mon, 26 Nov 2018 16:26:42 +0100, you wrote:
>> If on premise HPC doesn't change to reflect the way the software is
>> developed today then the users will in the future prefer cloud HPC.
>>
>> I guess it is a brave new world for on premise HPC as far as that the
>> users now, and likely more in the future, will have alternatives thus
>> forcing the on premise HPC to "compete" in order to survive.
>
>this seems a bit too stringent of a statement for me.  i don't dismiss
>or disagree with your premise, but i don't entirely agree that HPC
>"must" change in order to compete.  We've all heard this kind of stuff
>in the past if x doesn't change y will take over the world!

HPC, like most things, exists to get something done.

If HPC doesn't change to reflect the changes in society and the way
the software is developed (*) then the users will look for more modern
ways to replace traditional HPC.  As noted the software is no longer
developed on workstations that are connected to the lab/company
network but rather on laptops that stay with the user wherever they
go.

This in turn is at least in part what has driven to the rise of
distributed version control, git in particular.

If HPC doesn't make it easy for these users to transfer their workflow
to the cluster, and the cloud providers do, then the users will move
to using the cloud even if it costs them 10%, 20% more because at the
end of the day it is about getting the job done and not about spending
time to work to antiquated methods of putting jobs in a cluster.

And of course if the users would rather spend their department budgets
with Amazon, Azure, Google, or others then the next upgrade cycle
their won't be any money for the in house cluster...


* - note the HPC isn't unique in this regard.  The Linux distributions
are facing their own version of this, where much of the software is no
longer packagable in the traditional sense as it instead relies on
language specific packaging systems and languages that don't lend
themselves to the older rpm/deb style system.