<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal">This is interesting, because the figure of merit is “time to first answer” – that’s much like my iterative antenna modeling task – The cost of the computing is negligible compared to the cost of the engineer(s) waiting for the results.
<br>
<br>
Running the AWS calculator, a EC instance is $74/month when used 2hrs/day, so let’s call that about $1/hr. With an engineer billing out at $100/hr, spinning up 50 instances isn’t all that unreasonable.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Beowulf <beowulf-bounces@beowulf.org> on behalf of Tim Cutts <tjrc@sanger.ac.uk><br>
<b>Date: </b>Tuesday, September 21, 2021 at 5:02 AM<br>
<b>To: </b>John Hearns <hearnsj@gmail.com><br>
<b>Cc: </b>"beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Subject: </b>[EXTERNAL] Re: [Beowulf] Rant on why HPC isn't as easy as I'd like it to be. [EXT]<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">I think that’s exactly the situation we’ve been in for a long time, especially in life sciences, and it’s becoming more entrenched. My experience is that the average user of our scientific computing systems has been becoming less technically
savvy for many years now. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The presence of the cloud makes that more acute, in particular because it makes it easy for the user to effectively throw more hardware at the problem, which reduces the incentive to make their code particularly fast or efficient. Cost
is the only brake on it, and in many cases I’m finding the PI doesn’t actually care about that. They care that a result is being obtained (and it’s time to first result they care about, not time to complete all the analysis), and so they typically don’t have
much time for those of us who are telling them they need to invest in time up front developing and optimising efficient code.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And cost is not necessarily the brake I thought it was going to be anyway. One recent project we’ve done on AWS has impressed me a great deal. It’s not terribly CPU efficient, and would doubtless, with sufficient effort, run much more
efficiently on premise. But it’s extremely elastic in its nature, and so a good fit for the cloud. Once a week, the project has to completely re-analyse the 600,000+ COVID genomes we’e sequenced so far, looking for new branches in the phylogenetic tree,
and to complete that analysis inside 8 hours. Initial attempts to naively convert the HPC implementation to run on AWS looked as though they were going to be very expensive (~$50k per weekly run). But a fundamental reworking of the entire workflow to make
it as cloud native as possible, by which I mean almost exclusively serverless, has succeeded beyond what I expected. The total cost is <$5,000 a month, and because there is essentially no statically configured infrastructure at all, the security is fairly
easy to be comfortable about. And all of that was done with no detailed thinking about whether the actual algorithms running in the containers are at all optimised in a traditional HPC sense. It’s just not needed for this particular piece of work. Did it
need software developers with hardcore knowledge of performance optimisation? No. Was it rapid to develop and deploy? Yes. Is the performance fast enough for UK national COVID variant surveillance? Yes. Is it cost effective? Yes. Sold! The one thing
it did need was knowledgeable cloud architects, but the cloud providers can and do help with that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Tim<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">Tim Cutts<br>
Head of Scientific Computing<br>
Wellcome Sanger Institute<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 21 Sep 2021, at 12:24, John Hearns <<a href="mailto:hearnsj@gmail.com">hearnsj@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Some points well made here. I have seen in the past job scripts passed on from graduate student to graduate student - the case I am thinking on was an Abaqus script for 8 core systems, being run on a new 32 core system. Why WOULD a graduate
student question a script given to them - which works. They should be getting on with their science. I guess this is where Research Software Engineers come in.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Another point I would make is about modern processor architectures, for instance AMD Rome/Milan. You can have different Numa Per Socket options, which affect performance. We set the preferred IO path - which I have seen myself to have an
effect on latency of MPI messages. IF you are not concerned about your hardware layout you would just go ahead and run, missing a lot of performance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am now going to be controversial and common that over in Julia land the pattern seems to be these days people develop on their own laptops, or maybe local GPU systems. There is a lot of microbenchmarking going on. But there seems to be
not a lot of thought given to CPU pinning or shat happens with hyperthreading. I guess topics like that are part of HPC 'Black Magic' - though I would imagine the low latency crowd are hot on them.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I often introduce people to the excellent lstopo/hwloc utilities which show the layout of a system. Most people are pleasantly surprised to find this.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1
2BE. <o:p></o:p></p>
</div>
</body>
</html>