[Beowulf] HPC workflows
hearnsj at googlemail.com
Mon Nov 26 07:26:42 PST 2018
This may not be the best place to discuss this - please suggest a better
forum if you have one.
I have come across this question in a few locations. Being specific, I am a
fan of the Julia language. Ont he Juia forum a respected developer recently
asked what the options were for keeping code developed on a laptop in sync
with code being deployed on an HPC system.
There was some discussion of having Git style repositories which can be
My suggestion was an ssh mount of the home directory on the HPC system,
which I have configured effectively int he past when using remote HPC
At a big company I worked with recently, the company provided home
directories on NFS Servers. But the /home/username directory on the HPC was
different - on higher performance storage. The 'company' home was mounted -
so you could copy between them. But we did have the inevitable incidents of
jobs being run from company NFS - and pulling code across the head node
Developers these days are used to carrying their Mac laptops around and
working at hotdesks, at home, at conferences. ME too - and I love it.
Though I have a lovely HP Spectre Ultrabook.
Again their workflow is to develop on the laptop and upload code to Github
type repositories. Then when running on a cloud service the software ids
downloaded from the Repo.
There are of course HPC services on the cloud, with gateways to access them.
This leads me to ask - shoudl we be presenting HPC services as a 'cloud'
service, no matter that it is a non-virtualised on-premise setup?
In which case the way to deploy software would be via downloading from
I guess this is actually more common nowadays.
I think out loud that many HPC codes depend crucially on a $HOME directory
being presnet on the compute nodes as the codes look for dot files etc. in
$HOME. I guess this can be dealt with by fake $HOMES which again sync back
to the Repo.
And yes I know containerisation may be the saviour here!
Sorry for a long post.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf