[Beowulf] HPC workflows

[Gerald Henriksen]

On Wed, 28 Nov 2018 13:51:05 +0100, you wrote:

>Now I am all for connecting divers and flexible workflows to true HPC systems and grids that feel different if not experienced
>with (otherwise what is the use of a computer if there are no users making use of it?), but do not make the mistake of thinking
>everything is cloud or will be cloud soon that fast. 

The "cloud" is a massive business that is currently growing fast.

Will it take over everything or continue its growth forever, of course
not.

But dismissing it is equally a dangerous thing to do, particularly if
your job relies on something not being in the cloud.

>So, one could say bare metal cloud have arisen mostly because of this but they also do come with expenses. Somehow I find that a
>simple rule always seems to apply; if more people in a scheme need to be paid, the scheme is probably more expensive than
>alternatives, if available. Or state differently; If you can do things yourself, it is always a cheaper option than let some
>others do things (under normal 'open market' rules and excluding the option of slavery :)).

But this is one area where the cloud can often win - the scale of the
Azure/Google/AWS operations means that you get 24/7/365 coverage with
essentially the lowest possible labour overhead.

And the fact is that while much of society insists on making decisions
purely based on cost - see airfares for example - there are a lot of
cases where people are willing to pay a premium for a service/product
that "just works".

>One has to note that in academia one often is in the situation that grants are obtained to buy hardware and that running costs
>(i.e. electricity and rack space) are matched by the university making the case of spending the grant money on paying amazone or
>google to do your 'compute' not so sensible if you can do things yourself.

Currently.

If on premise HPC doesn't reflect the ease of use that can be found
elsewhere, combined with some lobbying by the existing or specialized
cloud providers, and those grants could become a lot more flexible.

And given that many/most/all universities are often short on space and
they may well welcome an opportunity to be able to repurpose an
existing cluster space...

>There is also another aspect when for example dealing with sensitive data you are to be helt responsible for. The Cloud model is
>not so friendly under those circumstances either. Again your data is put "on someone else's computer". Thinking of GDPR and
>such.

I don't think this is so clear an advantage to on premise as some
think.

I think the fact that we are all on this mailing list in order to
learn and discuss issues puts us as an outlier - there are very few
people participating on this list, and even allowing for discussions
happening on other sites I (sadly) suspect you will find that the
majority of people running HPC aren't as informed as they should be.

Who do you trust more to keep your data safe - to keep systems
patched, to keep firewalls up to date, to properly configure
everything, etc.?  Is it your local HPC, where maybe they are
struggling to hire staff, or can't afford to offer a "good enough"
salary, or simply can't justify hiring a security specialist?  Or
perhaps you go with Google or Microsoft, who have entire departments
of staff dealing with these issues, who monitor their networks full
time looking for flaws?