[Beowulf] HPC workflows

[John Hearns]

> * - note the HPC isn't unique in this regard.  The Linux distributions
> are facing their own version of this, where much of the software is no
> longer packagable in the traditional sense as it instead relies on
> language specific packaging systems and languages that don't lend
> themselves to the older rpm/deb style system.

Gerald, very well said. The term in the UK At the moment would be
'friction'. On-premise HPC has to be frictionless as cloud HPC.

 Note that I referred to Julia, which has a packaging system.
The Julia community has given a lot of thought to the packaging system for
1.0 and it has concepts such as environments different projects.

I hate to single out Python, but have experience of users using Anaconda
which means a huge variation in what everyone has.
And more importantly for HPC systems the packages are placed in the users
home directory (by default).
On the system I am thinking about there was very limited space on /home and
it was an NFS mount. MEaning any parallel program startup would
pull lots of data from NFS.

On Wed, 28 Nov 2018 at 01:42, Gerald Henriksen <ghenriks at gmail.com> wrote:

> On Tue, 27 Nov 2018 07:51:06 -0500, you wrote:
>
> >On Mon, Nov 26, 2018 at 9:50 PM Gerald Henriksen <ghenriks at gmail.com>
> wrote:
> >> On Mon, 26 Nov 2018 16:26:42 +0100, you wrote:
> >> If on premise HPC doesn't change to reflect the way the software is
> >> developed today then the users will in the future prefer cloud HPC.
> >>
> >> I guess it is a brave new world for on premise HPC as far as that the
> >> users now, and likely more in the future, will have alternatives thus
> >> forcing the on premise HPC to "compete" in order to survive.
> >
> >this seems a bit too stringent of a statement for me.  i don't dismiss
> >or disagree with your premise, but i don't entirely agree that HPC
> >"must" change in order to compete.  We've all heard this kind of stuff
> >in the past if x doesn't change y will take over the world!
>
> HPC, like most things, exists to get something done.
>
> If HPC doesn't change to reflect the changes in society and the way
> the software is developed (*) then the users will look for more modern
> ways to replace traditional HPC.  As noted the software is no longer
> developed on workstations that are connected to the lab/company
> network but rather on laptops that stay with the user wherever they
> go.
>
> This in turn is at least in part what has driven to the rise of
> distributed version control, git in particular.
>
> If HPC doesn't make it easy for these users to transfer their workflow
> to the cluster, and the cloud providers do, then the users will move
> to using the cloud even if it costs them 10%, 20% more because at the
> end of the day it is about getting the job done and not about spending
> time to work to antiquated methods of putting jobs in a cluster.
>
> And of course if the users would rather spend their department budgets
> with Amazon, Azure, Google, or others then the next upgrade cycle
> their won't be any money for the in house cluster...
>
>
> * - note the HPC isn't unique in this regard.  The Linux distributions
> are facing their own version of this, where much of the software is no
> longer packagable in the traditional sense as it instead relies on
> language specific packaging systems and languages that don't lend
> themselves to the older rpm/deb style system.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20181128/597c0df7/attachment.html>