<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Well, the example use case is: More HP needed to get the cycle time down to “interactive”-ish than you can get from a single computer. (particularly a *<b>quiet</b>* computer)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I was giving a specific example from a couple years ago (2019 in the COVID before times – that’s how long ago it was) That is, I had a fairly powerful desktop machine and I wound up farming it out to the cluster (JPL Halo, then TACC LoneStar5)
– even though the cores on Halo were actually slower than my desktop machine – but far more numerous..<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Jonathan Engwall <engwalljonathanthereal@gmail.com><br>
<b>Date: </b>Tuesday, August 24, 2021 at 10:59 AM<br>
<b>To: </b>Jim Lux <james.p.lux@jpl.nasa.gov><br>
<b>Cc: </b>Douglas Eadline <deadline@eadline.org>, "jaquilina@eagleeyet.net" <jaquilina@eagleeyet.net>, "beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Subject: </b>[EXTERNAL] Re: [Beowulf] Deskside clusters<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Hello Jim,<o:p></o:p></p>
</div>
<p class="MsoNormal">Use IPMI to throttle the fans. <o:p></o:p></p>
<div>
<p class="MsoNormal">With a personal 10K budget you can get an overkill dell emc...and need the fans. Scale back to your specs to what you need, 64 gb ram maybe. I can tell you how to access the ipmi controller, if you need to know.
<o:p></o:p></p>
<div>
<p class="MsoNormal">I worry this may cause a vulnerability, like opening a door, but my r630 is very nice. Trucks passing by make more noise. And a 300 watt PSU is sufficient.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This machine I use for RTC, video coursework, amateur astronomy, authoring, coding, and graphic design. Moving an entire adventure map, for an example in design consisting of a hundred or so individual tiny, colored shapes as a unit out
of the page margin is not realistic. But I can do it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Watching htop as my 10-core bleeds for a couple seconds, as the map creeps, is also fun.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">My specs are actually low, power consumption is low. The r630 is EOL sadly.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Jonathan Engwall<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Aug 24, 2021, 10:04 AM Lux, Jim (US 7140) via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">I've been looking at "small scale" clusters for a long time (2000?) and talked a lot with the folks from Orion, as well as on this list.<br>
They fit in a "hard to market to" niche.<br>
<br>
My own workflow tends to have use cases that are a big "off-nominal" - one is the rapid iteration of a computational model while experimenting - That is, I have a python code that generates input to Numerical Electromagnetics Code (NEC), I run the model over
a range of parameters, then look at the output to see if I'm getting what what I want. If not, I change the code (which essentially changes the antenna design), rerun the models, and see if it worked. I'd love an iteration time of "a minute or two" for the
computation, maybe a minute or two to plot the outputs (fiddling with the plot ranges, etc.). For reference, for a radio astronomy array on the far side of the Moon, I was running 144 cases, each at 380 frequencies: to run 1 case takes 30 seconds, so farming
it out to 12 processors gave me a 6 minute run time, which is in the right range. Another model of interaction of antnenas on a spacecraft runs about 15 seconds/case; and a third is about 120 seconds/case.
<br>
<br>
To get "interactive development", then, I want the "cycle time" to be 10 minutes - 30 minutes of thinking about how to change the design and altering the code to generate the new design, make a couple test runs to find the equivalent of "syntax errors", and
then turn it loose - get a cup of coffee, answer a few emails, come back and see the results. I could iterate maybe a half dozen shots a day, which is pretty productive. (Compared to straight up sequential - 144 runs at 30 seconds is more than an hour -
and that triggers a different working cadence that devolves to sort of one shot a day) - The "10 minute" turnaround is also compatible with my job, which, unfortunately, has things other than computing - meetings, budgets, schedules. At 10 minute runs, I
can carve out a few hours and get into that "flow state" on the technical problem, before being disrupted by "a person from Porlock."<br>
<br>
So this is, I think, a classic example of "I want local control" - sure, you might have access to a 1000 or more node cluster, but you're going to have to figure out how to use its batch management system (SLURM and PBS are two I've used) - and that's a bit
different than "self managed 100% access". Or, AWS kinds of solutions for EP problems. There's something very satisfying about getting an idea and not having to "ok, now I have to log in to the remote cluster with TFA, set up the tunnel, move my data, get
the job spun up, get the data back" - especially for iterative development. I did do that using JPLs and TACCs clusters, and "moving data" proved to be a barrier - the other thing was the "iterative code development" in between runs - Most institutional clusters
discourage interactive development on the cluster (even if you're only sucking up one core). If the tools were a bit more "transparent" and there were "shared disk" capabilities, this might be more attractive, and while everyone is exceedingly helpful, there
are still barriers to making it "run it on my desktop"<br>
<br>
Another use case that I wind up designing for is the "HPC in places without good communications and limited infrastructure" - The notional use case might be an archaeological expedition wanting to use HPC to process ground penetrating radar data or something
like that. (or, given that I work at JPL, you have a need for HPC on the surface of Mars) - So sending your data to a remote cluster isn't really an option. And here, the "speedup" you need might well be a factor of 10-20 over a single computer, something
doable in a "portable" configuration (check it as luggage, for instance). Just as for my antenna modeling problems, turning an "overnight" computation into a "10-20 minute" computation would change the workflow dramatically.
<br>
<br>
<br>
Another market is "learn how to cluster" - for which the RPi clusters work (or "packs" of Beagleboards) - they're fun, and in a classroom environment, I think they are an excellent cost effective solution to learning all the facets of "bringing up a cluster
from scratch", but I'm not convinced they provide a good "MIPS/Watt" or "MIPS/liter" metric - in terms of convenience. That is, rather than a cluster of 10 RPis, you might be better off just buying a faster desktop machine.<br>
<br>
Let's talk design desirements/constraints<br>
<br>
I've had a chance to use some "clusters in a box" over the last decades, and I'd suggest that while power is one constraint, another is noise. Just the other day, I was in a lab and someone commented that "those computers are amazingly fast, but you really
need to put them in another room". Yes, all those 1U and 2U rack mounted boxes with tiny fans screaming is just not "office compatible" And that kind of brings up another interesting constraint for "deskside" computing - heat. Sure you can plug in 1500W
of computers (or even 3000W if you have two circuits), but can you live in your office with a 1500W space heater? Interestingly, for *my* workflow, that's probably ok - *my* computation has a 10-30% duty cycle - think for 30 minutes, compute for 5-10. But
still, your office mate will appreciate if you keep the sound level down to 50dBA.<br>
<br>
GPUs - some codes can use them, some can't. They tend, though, to be noisy (all that air flow for cooling). I don't know that GPU manufacturers spend a lot of time on this. Sure, I've seen charts and specs that claim <50 dBA. But I think they're gaming the
measurement, counting on the user to be a gamer wearing headphones or with a big sound system. I will say, for instance, that the PS/4 positively roars when spun up unless you’ve got external forced ventilation to keep the inlet air temp low.<br>
<br>
Looking at GSA guidelines for office space - if it's "deskside" it's got to fit in the 50-80 square foot cubicle or your shared part of a 120 square foot office.<br>
<br>
Then one needs to figure out the "refresh cycle time" for buying hardware - This has been a topic on this list forever - you have 2 years of computation to do: do you buy N nodes today at speed X, or do you wait a year, buy N/2 nodes at speed 4X, and finish
your computation at the same time. <br>
<br>
Fancy desktop PCs with monitors, etc. come in at under $5k, including burdens and installation, but not including monthly service charges (in an institutional environment). If you look at "purchase limits" there's some thresholds (usually around $10k, then
increasing in factors of 10 or 100 steps) for approvals. So a $100k deskside box is going to be a tough sell.<br>
<br>
<br>
<br>
On 8/24/21, 6:07 AM, "Beowulf on behalf of Douglas Eadline" <<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a> on behalf of
<a href="mailto:deadline@eadline.org" target="_blank">deadline@eadline.org</a>> wrote:<br>
<br>
Jonathan<br>
<br>
It is a real cluster, available in 4 and 8 node versions.<br>
The design if for non-data center use. That is, local<br>
office, lab, home where power, cooling, and noise<br>
are important. More info here:<br>
<br>
<a href="https://urldefense.us/v3/__https:/www.limulus-computing.com__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDpKU4fOA$" target="_blank">
https://urldefense.us/v3/__https://www.limulus-computing.com__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDpKU4fOA$</a>
<br>
<a href="https://urldefense.us/v3/__https:/www.limulus-computing.com/Limulus-Manual__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XD7eWwVuM$" target="_blank">
https://urldefense.us/v3/__https://www.limulus-computing.com/Limulus-Manual__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XD7eWwVuM$</a>
<br>
<br>
--<br>
Doug<br>
<br>
<br>
<br>
> Hi Doug,<br>
><br>
> Not to derail the discussion, but a quick question you say desk side<br>
> cluster is it a single machine that will run a vm cluster?<br>
><br>
> Regards,<br>
> Jonathan<br>
><br>
> -----Original Message-----<br>
> From: Beowulf <<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a>> On Behalf Of Douglas Eadline<br>
> Sent: 23 August 2021 23:12<br>
> To: John Hearns <<a href="mailto:hearnsj@gmail.com" target="_blank">hearnsj@gmail.com</a>><br>
> Cc: Beowulf Mailing List <<a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a>><br>
> Subject: Re: [Beowulf] List archives<br>
><br>
> John,<br>
><br>
> I think that was on twitter.<br>
><br>
> In any case, I'm working with these processors right now.<br>
><br>
> On the new Ryzens, the power usage is actually quite tunable.<br>
> There are three settings.<br>
><br>
> 1) Package Power Tracking: The PPT threshold is the allowed socket power<br>
> consumption permitted across the voltage rails supplying the socket.<br>
><br>
> 2) Thermal Design Current: The maximum current (TDC) (amps) that can be<br>
> delivered by a specific motherboard's voltage regulator configuration in<br>
> thermally-constrained scenarios.<br>
><br>
> 3) Electrical Design Current: The maximum current (EDC) (amps) that can be<br>
> delivered by a specific motherboard's voltage regulator configuration in a<br>
> peak ("spike") condition for a short period of time.<br>
><br>
> My goal is to tweak the 105W TDP R7-5800X so it draws power like the<br>
> 65W-TDP R5-5600X<br>
><br>
> This is desk-side cluster low power stuff.<br>
> I am using extension cable-plug for Limulus blades that have an in-line<br>
> current meter (normally used for solar panels).<br>
> Now I can load them up and watch exactly how much current is being pulled<br>
> across the 12V rails.<br>
><br>
> If you need more info, let me know<br>
><br>
> --<br>
> Doug<br>
><br>
>> The Beowulf list archives seem to end in July 2021.<br>
>> I was looking for Doug Eadline's post on limiting AMD power and the<br>
>> results on performance.<br>
>><br>
>> John H<br>
>> _______________________________________________<br>
>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">
Beowulf@beowulf.org</a> sponsored by Penguin<br>
>> Computing To change your subscription (digest mode or unsubscribe)<br>
>> visit<br>
>> <a href="https://urldefense.us/v3/__https:/link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:*__;Lw!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDvUGSdHI$" target="_blank">
https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:*__;Lw!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDvUGSdHI$</a>
<br>
>> /<a href="https://urldefense.us/v3/__http:/beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!YZBC5Z_E3miRCi6CM_2NjqRVSAI_2CqoV9vJcI5wn2UX-chIfhuIvrbqsv-y_FroUsGg53U$" target="_blank">beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
>><br>
><br>
><br>
> --<br>
> Doug<br>
><br>
> _______________________________________________<br>
> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">
Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
> To change your subscription (digest mode or unsubscribe) visit<br>
> <a href="https://urldefense.us/v3/__https:/link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:**Abeowulf.org*cgi-bin*mailman*listinfo*beowulf__;Ly8vLy8v!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDUP8JZUc$" target="_blank">
https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:**Abeowulf.org*cgi-bin*mailman*listinfo*beowulf__;Ly8vLy8v!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDUP8JZUc$</a>
<br>
><br>
<br>
<br>
-- <br>
Doug<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://urldefense.us/v3/__https:/beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDv6c1nNc$" target="_blank">
https://urldefense.us/v3/__https://beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDv6c1nNc$</a>
<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://urldefense.us/v3/__https:/beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!YZBC5Z_E3miRCi6CM_2NjqRVSAI_2CqoV9vJcI5wn2UX-chIfhuIvrbqsv-y_FroL_bZyeE$" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>