<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>I'd be interested<br>
    </p>
    <pre class="moz-signature" cols="72">Prentice </pre>
    <div class="moz-cite-prefix">On 9/23/21 10:37 AM, Pizarro, Angel via
      Beowulf wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:C2E75C90-71E4-4BD0-A539-132CF07EC8A5@amazon.com">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <meta name="Generator" content="Microsoft Word 15 (filtered
        medium)">
      <style>@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}span.EmailStyle20
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}div.WordSection1
        {page:WordSection1;}</style>
      <div class="WordSection1">
        <p class="MsoNormal">DISCLOSURE: I work for AWS HPC Developer
          Relations in the services team. We developer AWS Batch, AWS
          ParallelCluster, NICE DCV, etc.
          <o:p></o:p></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p class="MsoNormal">Lambda’s limits today are 128MB to 10,240MB
          (~10GB) and billed in 1MB per ms increments. 15 minute max
          runtime for the function invocation.  <o:p></o:p></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p class="MsoNormal">Would you all be interested in a hands-on
          self-paced workshop on creating (or porting) an application to
          serverless environment? E.g. Monte-Carlo simulation, a genome
          alignment or variant call, or some other problem? We have some
          basic data processing documentation but nothing that speaks to
          real-world HPC use case and that is a something I want to fill
          the gap on if folks are interested in it.  <o:p></o:p></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p class="MsoNormal">Dr. Denis Bauer at CSIRO is also doing
          interesting things with serverless.
          <o:p></o:p></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p class="MsoNormal">-angel     <o:p></o:p></p>
        <p class="MsoNormal">--  <o:p></o:p></p>
        <p class="MsoNormal">Angel Pizarro | Principal Developer
          Advocate, HPC @ AWS <o:p>
          </o:p></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <div style="border:none;border-top:solid #B5C4DF
          1.0pt;padding:3.0pt 0in 0in 0in">
          <p class="MsoNormal"><b><span
                style="font-size:12.0pt;color:black">From: </span></b><span
              style="font-size:12.0pt;color:black">Beowulf
              <a class="moz-txt-link-rfc2396E" href="mailto:beowulf-bounces@beowulf.org"><beowulf-bounces@beowulf.org></a> on behalf of Guy
              Coates <a class="moz-txt-link-rfc2396E" href="mailto:guy.coates@gmail.com"><guy.coates@gmail.com></a><br>
              <b>Date: </b>Thursday, September 23, 2021 at 8:46 AM<br>
              <b>To: </b>Tim Cutts <a class="moz-txt-link-rfc2396E" href="mailto:tjrc@sanger.ac.uk"><tjrc@sanger.ac.uk></a><br>
              <b>Cc: </b>Beowulf <a class="moz-txt-link-rfc2396E" href="mailto:beowulf@beowulf.org"><beowulf@beowulf.org></a><br>
              <b>Subject: </b>RE: [EXTERNAL] [Beowulf] Rant on why HPC
              isn't as easy as I'd like it to be. [EXT]<o:p></o:p></span></p>
        </div>
        <div>
          <p class="MsoNormal"><o:p> </o:p></p>
        </div>
        <div>
          <table class="MsoNormalTable" style="border-collapse:collapse"
            cellspacing="0" cellpadding="0" border="0">
            <tbody>
              <tr style="height:15.25pt">
                <td style="width:842.35pt;border:solid #ED7D31
                  1.5pt;padding:0in 5.4pt 0in 5.4pt;height:15.25pt"
                  width="1123" valign="top">
                  <p><strong><span
style="font-family:"Calibri",sans-serif;color:black;background:#FFFF99">CAUTION</span></strong><span
                      style="color:black;background:#FFFF99">: This
                      email originated from outside of the organization.
                      Do not click links or open attachments unless you
                      can confirm the sender and know the content is
                      safe.</span><o:p></o:p></p>
                </td>
              </tr>
            </tbody>
          </table>
        </div>
        <p class="MsoNormal"><o:p> </o:p></p>
        <div>
          <div>
            <p class="MsoNormal">Out of interest, how large are the
              compute jobs (memory, runtime etc)?  How easy to get them
              to fit into a serverless environment?
              <o:p></o:p></p>
            <div>
              <p class="MsoNormal"><o:p> </o:p></p>
            </div>
            <div>
              <p class="MsoNormal">Thanks,<o:p></o:p></p>
            </div>
            <div>
              <p class="MsoNormal"><br>
                Guy<o:p></o:p></p>
            </div>
          </div>
          <p class="MsoNormal"><o:p> </o:p></p>
          <div>
            <div>
              <p class="MsoNormal">On Tue, 21 Sept 2021 at 13:02, Tim
                Cutts <<a href="mailto:tjrc@sanger.ac.uk"
                  moz-do-not-send="true">tjrc@sanger.ac.uk</a>>
                wrote:<o:p></o:p></p>
            </div>
            <blockquote style="border:none;border-left:solid #CCCCCC
              1.0pt;padding:0in 0in 0in
              6.0pt;margin-left:4.8pt;margin-right:0in">
              <div>
                <p class="MsoNormal">I think that’s exactly the
                  situation we’ve been in for a long time, especially in
                  life sciences, and it’s becoming more entrenched.  My
                  experience is that the average user of our scientific
                  computing systems has been becoming less technically
                  savvy for many years now. <o:p></o:p></p>
                <div>
                  <p class="MsoNormal"><o:p> </o:p></p>
                </div>
                <div>
                  <p class="MsoNormal">The presence of the cloud makes
                    that more acute, in particular because it makes it
                    easy for the user to effectively throw more hardware
                    at the problem, which reduces the incentive to make
                    their code particularly fast or efficient.  Cost is
                    the only brake on it, and in many cases I’m finding
                    the PI doesn’t actually care about that.  They care
                    that a result is being obtained (and it’s time to
                    first result they care about, not time to complete
                    all the analysis), and so they typically don’t have
                    much time for those of us who are telling them they
                    need to invest in time up front developing and
                    optimising efficient code.<o:p></o:p></p>
                </div>
                <div>
                  <p class="MsoNormal"><o:p> </o:p></p>
                </div>
                <div>
                  <p class="MsoNormal">And cost is not necessarily the
                    brake I thought it was going to be anyway.  One
                    recent project we’ve done on AWS has impressed me a
                    great deal.  It’s not terribly CPU efficient, and
                    would doubtless, with sufficient effort, run much
                    more efficiently on premise.  But it’s extremely
                    elastic in its nature, and so a good fit for the
                    cloud.   Once a week, the project has to completely
                    re-analyse the 600,000+ COVID genomes we’e sequenced
                    so far, looking for new branches in the phylogenetic
                    tree, and to complete that analysis inside 8 hours.
                      Initial attempts to naively convert the HPC
                    implementation to run on AWS looked as though they
                    were going to be very expensive (~$50k per weekly
                    run).  But a fundamental reworking of the entire
                    workflow to make it as cloud native as possible, by
                    which I mean almost exclusively serverless, has
                    succeeded beyond what I expected.  The total cost is
                    <$5,000 a month, and because there is essentially
                    no statically configured infrastructure at all, the
                    security is fairly easy to be comfortable about. 
                    And all of that was done with no detailed thinking
                    about whether the actual algorithms running in the
                    containers are at all optimised in a traditional HPC
                    sense.  It’s just not needed for this particular
                    piece of work.  Did it need software developers with
                    hardcore knowledge of performance optimisation? 
                    No.  Was it rapid to develop and deploy?  Yes.  Is
                    the performance fast enough for UK national COVID
                    variant surveillance?  Yes.  Is it cost effective? 
                    Yes.  Sold!  The one thing it did need was
                    knowledgeable cloud architects, but the cloud
                    providers can and do help with that.<o:p></o:p></p>
                </div>
                <div>
                  <p class="MsoNormal"><o:p> </o:p></p>
                </div>
                <div>
                  <p class="MsoNormal">Tim<o:p></o:p></p>
                </div>
                <div>
                  <p class="MsoNormal"><o:p> </o:p></p>
                </div>
                <div>
                  <div>
                    <div>
                      <p class="MsoNormal"><span style="color:black">-- <o:p></o:p></span></p>
                    </div>
                    <div>
                      <p class="MsoNormal" style="margin-bottom:12.0pt"><span
                          style="color:black">Tim Cutts<br>
                          Head of Scientific Computing<br>
                          Wellcome Sanger Institute<o:p></o:p></span></p>
                    </div>
                  </div>
                  <div>
                    <p class="MsoNormal"><br>
                      <br>
                      <o:p></o:p></p>
                    <blockquote
                      style="margin-top:5.0pt;margin-bottom:5.0pt">
                      <div>
                        <p class="MsoNormal">On 21 Sep 2021, at 12:24,
                          John Hearns <<a
                            href="mailto:hearnsj@gmail.com"
                            target="_blank" moz-do-not-send="true">hearnsj@gmail.com</a>>
                          wrote:<o:p></o:p></p>
                      </div>
                      <p class="MsoNormal"><o:p> </o:p></p>
                      <div>
                        <div>
                          <p class="MsoNormal">Some points well made
                            here. I have seen in the past job scripts
                            passed on from graduate student to graduate
                            student - the case I am thinking on was an
                            Abaqus script for 8 core systems, being run
                            on a new 32 core system. Why WOULD a
                            graduate student question a script given to
                            them - which works. They should be getting
                            on with their science. I guess this is where
                            Research Software Engineers come in.
                            <o:p></o:p></p>
                          <div>
                            <p class="MsoNormal"><o:p> </o:p></p>
                          </div>
                          <div>
                            <p class="MsoNormal">Another point I would
                              make is about modern processor
                              architectures, for instance AMD
                              Rome/Milan. You can have different Numa
                              Per Socket options, which affect
                              performance. We set the preferred IO path
                              - which I have seen myself to have an
                              effect on latency of MPI messages. IF you
                              are not concerned about your hardware
                              layout you would just go ahead and run,
                              missing  a lot of performance.<o:p></o:p></p>
                          </div>
                          <div>
                            <p class="MsoNormal"><o:p> </o:p></p>
                          </div>
                          <div>
                            <p class="MsoNormal">I am now going to be
                              controversial and common that over in
                              Julia land the pattern seems to be these
                              days people develop on their own laptops,
                              or maybe local GPU systems. There is a lot
                              of microbenchmarking going on. But there
                              seems to be not a lot of thought given to
                              CPU pinning or shat happens with
                              hyperthreading. I guess topics like that
                              are part of HPC 'Black Magic' - though I
                              would imagine the low latency crowd are
                              hot on them.<o:p></o:p></p>
                          </div>
                          <div>
                            <p class="MsoNormal"><o:p> </o:p></p>
                          </div>
                          <div>
                            <p class="MsoNormal">I often introduce
                              people to the excellent lstopo/hwloc
                              utilities which show the layout of a
                              system. Most people are pleasantly
                              surprised to find this.<o:p></o:p></p>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <p class="MsoNormal"><o:p> </o:p></p>
                </div>
                <p class="MsoNormal">-- The Wellcome Sanger Institute is
                  operated by Genome Research Limited, a charity
                  registered in England with number 1021457 and a
                  company registered in England with number 2742969,
                  whose registered office is 215 Euston Road, London,
                  NW1 2BE. <o:p></o:p></p>
              </div>
              <p class="MsoNormal">_______________________________________________<br>
                Beowulf mailing list, <a
                  href="mailto:Beowulf@beowulf.org" target="_blank"
                  moz-do-not-send="true">Beowulf@beowulf.org</a>
                sponsored by Penguin Computing<br>
                To change your subscription (digest mode or unsubscribe)
                visit <a
                  href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
                  target="_blank" moz-do-not-send="true">
                  https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><o:p></o:p></p>
            </blockquote>
          </div>
          <p class="MsoNormal"><br clear="all">
            <o:p></o:p></p>
          <div>
            <p class="MsoNormal"><o:p> </o:p></p>
          </div>
          <p class="MsoNormal">-- <o:p></o:p></p>
          <div>
            <div>
              <p class="MsoNormal">Dr. Guy Coates<br>
                +44(0)7801 710224 <o:p></o:p></p>
              <div>
                <p class="MsoNormal"><o:p> </o:p></p>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
</pre>
    </blockquote>
  </body>
</html>