<div dir="ltr">Or to put it simply: "Alexa - sequence my genome"</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 25 Nov 2020 at 09:45, John Hearns <<a href="mailto:hearnsj@gmail.com">hearnsj@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Tim, that is really smart. Over on the Julia discourse forum I have blue skyed about using Lambdas to run Julia functions (it is an inherently functional language) (*)<div>Blue skying further, for exascale compute needs can we think of 'Science as a Service'?</div><div>As in your example the scientist thinks about the analysis and how it is performed. Then sends it off to be executed. Large chunks are run using Lambda functions.</div><div>Crucially, if a Lambda (or whatever) fails the algorithm should be able to continue. People building web scale applications think like this today anyway.</div><div>Do you REALLY think you are connected to Amazon's single web server when you make a purchase? But it looks that way.</div><div>Also if you are about to purchase something and your Wifi goes down - as a customer you would be very angry if you were billed for this item.</div><div><br></div><div>(*) It is possible to insert your own 'payload' in a Lambda. There are standard ones like Python obviously.</div><div>However at the time I looked there was a small size limit on the payload.</div><div><br></div><div>Re-reading my won response <a href="https://discourse.julialang.org/t/lambda-or-cloud-functions-eventually-possible/39128/5" target="_blank">https://discourse.julialang.org/t/lambda-or-cloud-functions-eventually-possible/39128/5</a></div><div>you CAN have a larger payload, but this has to be in an S3 bucket</div><div><a href="https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html" target="_blank">https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html</a><br></div><div><br></div><div>BTW, I am sure everyone knows this but if you have a home assistant such as Alexa everytime you ask Alexa it is a lambda which is spun up </div><div><br><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 25 Nov 2020 at 09:27, Tim Cutts <<a href="mailto:tjrc@sanger.ac.uk" target="_blank">tjrc@sanger.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<br>
<div><br>
<blockquote type="cite">
<div>On 24 Nov 2020, at 18:31, Alex Chekholko via Beowulf <<a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a>> wrote:</div>
<br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
If you can run your task on just one computer, you should always do that rather than having to build a cluster of some kind and all the associated headaches.</div>
<br>
</div>
</blockquote>
</div>
<div><br>
</div>
<div>If you take on the cloud message, that of course isn’t necessarily the case. If you use very high level cloud services like lambda, you don’t have to build that infrastructure. It’s very unlikely to be anywhere near as efficient, of course,
but throughput efficiency is not what your average scientist cares about. What they care about is getting their answer quickly (and to a lesser extent, cheaply)</div>
<div><br>
</div>
<div>I saw a recent example where someone took a fairly simple sequencing read alignment process, which normally runs on a single 16-core node in about 6 hours, and split the input files small enough that the alignment code execution time and memory
use would fit with AWS Lambda’s envelope. The result executed in a couple of minutes, elapsed, but used about four times as many core-hours as the optimised single node version. Of course, this is an embarrassingly parallel problem, so this is a relatively
easy analysis to move to this sort of design.</div>
<div><br>
</div>
<div>From the scientist’s point of view, which is better? Getting their answer in 5 minutes or 6 hours? Especially if they’ve also reduced their development time as well because they don’t have to worry so much about infrastructure and optimisation.</div>
<div><br>
</div>
<div>The total value is hard to work out, many of these considerations are hard to put a dollar value on. When I saw that article, I did ask the author how much the analysis actually cost, and she didn’t have a number. But I don’t think we can dogmatically
say that we should always run a task on a single machine if we can.</div>
<div><br>
</div>
<div>Tim</div>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div>
</blockquote></div>