<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Not anymore, at least not in the HPC realm. We recently
purchased quad-socket systems with a total of 96 Intel cores/node,
and dual socket systems with 128 AMD cores/node. <br>
</p>
<p>With Intel now marking their "highly scalable" (or something like
that) line of processors, and AMD, who was always pushing highr
core-counts, back in the game, I think numbers like that will be
common in HPC clusters puchased in the next year or so. <br>
</p>
<p>But, yeah, I guess 28 physical cores is more than the average
desktop has these days. <br>
</p>
<p><br>
</p>
<pre class="moz-signature" cols="72">Prentice </pre>
<div class="moz-cite-prefix">On 8/24/21 6:42 PM, Jonathan Engwall
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAP=T6ZOEX+RNXB3epZJ9bAXfT-SoiYUpYqQpr0oCsCTe0oSiJg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">EMC offers dual socket 28 physical core
processors. That's a lot of computer.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Aug 24, 2021, 1:33 PM
Lux, Jim (US 7140) via Beowulf <<a
href="mailto:beowulf@beowulf.org" moz-do-not-send="true">beowulf@beowulf.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Yes,
indeed.. I didn't call out Limulus, because it was mentioned
earlier in the thread.<br>
<br>
And another reason why you might want your own.<br>
Every so often, the notice from JPL's HPC goes out to the
users - "Halo/Gattaca/clustername will not be available
because it is reserved for Mars {Year}" While Mars landings
at JPL are a *big deal*, not everyone is working on them (in
fact, by that time, most of the Martians are now working on
something else), and you want to get your work done. I
suspect other institutional clusters have similar "the 800
pound (363 kg) gorilla has requested" scenarios.<br>
<br>
<br>
On 8/24/21, 11:34 AM, "Douglas Eadline" <<a
href="mailto:deadline@eadline.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">deadline@eadline.org</a>>
wrote:<br>
<br>
<br>
Jim,<br>
<br>
You are describing a lot of the design pathway for Limulus<br>
clusters. The local (non-data center) power, heat, noise
are all<br>
minimized while performance is maximized.<br>
<br>
A well decked out system is often less than $10K and<br>
are on par with a fat multi-core workstations.<br>
(and there are reasons a clustered approach performs
better)<br>
<br>
Another use case is where there is no available research
data center<br>
hardware because there is no specialized
sysadmins/space/budget.<br>
(Many smaller colleges and universities fall into this<br>
group). Plus, often times, dropping something into a data
center<br>
means an additional cost to the researchers budget.<br>
<br>
--<br>
Doug<br>
<br>
<br>
> I've been looking at "small scale" clusters for a
long time (2000?) and<br>
> talked a lot with the folks from Orion, as well as on
this list.<br>
> They fit in a "hard to market to" niche.<br>
><br>
> My own workflow tends to have use cases that are a
big "off-nominal" - one<br>
> is the rapid iteration of a computational model while
experimenting - That<br>
> is, I have a python code that generates input to
Numerical<br>
> Electromagnetics Code (NEC), I run the model over a
range of parameters,<br>
> then look at the output to see if I'm getting what
what I want. If not, I<br>
> change the code (which essentially changes the
antenna design), rerun the<br>
> models, and see if it worked. I'd love an iteration
time of "a minute or<br>
> two" for the computation, maybe a minute or two to
plot the outputs<br>
> (fiddling with the plot ranges, etc.). For
reference, for a radio<br>
> astronomy array on the far side of the Moon, I was
running 144 cases, each<br>
> at 380 frequencies: to run 1 case takes 30 seconds,
so farming it out to<br>
> 12 processors gave me a 6 minute run time, which is
in the right range.<br>
> Another model of interaction of antnenas on a
spacecraft runs about 15<br>
> seconds/case; and a third is about 120 seconds/case.<br>
><br>
> To get "interactive development", then, I want the
"cycle time" to be 10<br>
> minutes - 30 minutes of thinking about how to change
the design and<br>
> altering the code to generate the new design, make a
couple test runs to<br>
> find the equivalent of "syntax errors", and then turn
it loose - get a cup<br>
> of coffee, answer a few emails, come back and see the
results. I could<br>
> iterate maybe a half dozen shots a day, which is
pretty productive.<br>
> (Compared to straight up sequential - 144 runs at 30
seconds is more than<br>
> an hour - and that triggers a different working
cadence that devolves to<br>
> sort of one shot a day) - The "10 minute" turnaround
is also compatible<br>
> with my job, which, unfortunately, has things other
than computing -<br>
> meetings, budgets, schedules. At 10 minute runs, I
can carve out a few<br>
> hours and get into that "flow state" on the technical
problem, before<br>
> being disrupted by "a person from Porlock."<br>
><br>
> So this is, I think, a classic example of "I want
local control" - sure,<br>
> you might have access to a 1000 or more node cluster,
but you're going to<br>
> have to figure out how to use its batch management
system (SLURM and PBS<br>
> are two I've used) - and that's a bit different than
"self managed 100%<br>
> access". Or, AWS kinds of solutions for EP problems.
There's something<br>
> very satisfying about getting an idea and not having
to "ok, now I have to<br>
> log in to the remote cluster with TFA, set up the
tunnel, move my data,<br>
> get the job spun up, get the data back" - especially
for iterative<br>
> development. I did do that using JPLs and TACCs
clusters, and "moving<br>
> data" proved to be a barrier - the other thing was
the "iterative code<br>
> development" in between runs - Most institutional
clusters discourage<br>
> interactive development on the cluster (even if
you're only sucking up one<br>
> core). If the tools were a bit more "transparent"
and there were "shared<br>
> disk" capabilities, this might be more attractive,
and while everyone is<br>
> exceedingly helpful, there are still barriers to
making it "run it on my<br>
> desktop"<br>
><br>
> Another use case that I wind up designing for is the
"HPC in places<br>
> without good communications and limited
infrastructure" - The notional<br>
> use case might be an archaeological expedition
wanting to use HPC to<br>
> process ground penetrating radar data or something
like that. (or, given<br>
> that I work at JPL, you have a need for HPC on the
surface of Mars) - So<br>
> sending your data to a remote cluster isn't really an
option. And here,<br>
> the "speedup" you need might well be a factor of
10-20 over a single<br>
> computer, something doable in a "portable"
configuration (check it as<br>
> luggage, for instance). Just as for my antenna
modeling problems, turning<br>
> an "overnight" computation into a "10-20 minute"
computation would change<br>
> the workflow dramatically.<br>
><br>
><br>
> Another market is "learn how to cluster" - for which
the RPi clusters work<br>
> (or "packs" of Beagleboards) - they're fun, and in a
classroom<br>
> environment, I think they are an excellent cost
effective solution to<br>
> learning all the facets of "bringing up a cluster
from scratch", but I'm<br>
> not convinced they provide a good "MIPS/Watt" or
"MIPS/liter" metric - in<br>
> terms of convenience. That is, rather than a cluster
of 10 RPis, you<br>
> might be better off just buying a faster desktop
machine.<br>
><br>
> Let's talk design desirements/constraints<br>
><br>
> I've had a chance to use some "clusters in a box"
over the last decades,<br>
> and I'd suggest that while power is one constraint,
another is noise.<br>
> Just the other day, I was in a lab and someone
commented that "those<br>
> computers are amazingly fast, but you really need to
put them in another<br>
> room". Yes, all those 1U and 2U rack mounted boxes
with tiny fans<br>
> screaming is just not "office compatible" And that
kind of brings up<br>
> another interesting constraint for "deskside"
computing - heat. Sure you<br>
> can plug in 1500W of computers (or even 3000W if you
have two circuits),<br>
> but can you live in your office with a 1500W space
heater?<br>
> Interestingly, for *my* workflow, that's probably ok
- *my* computation<br>
> has a 10-30% duty cycle - think for 30 minutes,
compute for 5-10. But<br>
> still, your office mate will appreciate if you keep
the sound level down<br>
> to 50dBA.<br>
><br>
> GPUs - some codes can use them, some can't. They
tend, though, to be<br>
> noisy (all that air flow for cooling). I don't know
that GPU manufacturers<br>
> spend a lot of time on this. Sure, I've seen charts
and specs that claim<br>
> <50 dBA. But I think they're gaming the
measurement, counting on the user<br>
> to be a gamer wearing headphones or with a big sound
system. I will say,<br>
> for instance, that the PS/4 positively roars when
spun up unless you’ve<br>
> got external forced ventilation to keep the inlet air
temp low.<br>
><br>
> Looking at GSA guidelines for office space - if it's
"deskside" it's got<br>
> to fit in the 50-80 square foot cubicle or your
shared part of a 120<br>
> square foot office.<br>
><br>
> Then one needs to figure out the "refresh cycle time"
for buying hardware<br>
> - This has been a topic on this list forever - you
have 2 years of<br>
> computation to do: do you buy N nodes today at speed
X, or do you wait a<br>
> year, buy N/2 nodes at speed 4X, and finish your
computation at the same<br>
> time.<br>
><br>
> Fancy desktop PCs with monitors, etc. come in at
under $5k, including<br>
> burdens and installation, but not including monthly
service charges (in an<br>
> institutional environment). If you look at "purchase
limits" there's some<br>
> thresholds (usually around $10k, then increasing in
factors of 10 or 100<br>
> steps) for approvals. So a $100k deskside box is
going to be a tough<br>
> sell.<br>
><br>
><br>
><br>
> On 8/24/21, 6:07 AM, "Beowulf on behalf of Douglas
Eadline"<br>
> <<a href="mailto:beowulf-bounces@beowulf.org"
target="_blank" rel="noreferrer" moz-do-not-send="true">beowulf-bounces@beowulf.org</a>
on behalf of <a href="mailto:deadline@eadline.org"
target="_blank" rel="noreferrer" moz-do-not-send="true">deadline@eadline.org</a>>
wrote:<br>
><br>
> Jonathan<br>
><br>
> It is a real cluster, available in 4 and 8 node
versions.<br>
> The design if for non-data center use. That is,
local<br>
> office, lab, home where power, cooling, and noise<br>
> are important. More info here:<br>
><br>
> <a
href="https://urldefense.us/v3/__https://www.limulus-computing.com__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDpKU4fOA$"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://urldefense.us/v3/__https://www.limulus-computing.com__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDpKU4fOA$</a><br>
> <a
href="https://urldefense.us/v3/__https://www.limulus-computing.com/Limulus-Manual__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XD7eWwVuM$"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://urldefense.us/v3/__https://www.limulus-computing.com/Limulus-Manual__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XD7eWwVuM$</a><br>
><br>
> --<br>
> Doug<br>
><br>
><br>
><br>
> > Hi Doug,<br>
> ><br>
> > Not to derail the discussion, but a quick
question you say desk<br>
> side<br>
> > cluster is it a single machine that will run
a vm cluster?<br>
> ><br>
> > Regards,<br>
> > Jonathan<br>
> ><br>
> > -----Original Message-----<br>
> > From: Beowulf <<a
href="mailto:beowulf-bounces@beowulf.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">beowulf-bounces@beowulf.org</a>>
On Behalf Of Douglas<br>
> Eadline<br>
> > Sent: 23 August 2021 23:12<br>
> > To: John Hearns <<a
href="mailto:hearnsj@gmail.com" target="_blank"
rel="noreferrer" moz-do-not-send="true">hearnsj@gmail.com</a>><br>
> > Cc: Beowulf Mailing List <<a
href="mailto:beowulf@beowulf.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">beowulf@beowulf.org</a>><br>
> > Subject: Re: [Beowulf] List archives<br>
> ><br>
> > John,<br>
> ><br>
> > I think that was on twitter.<br>
> ><br>
> > In any case, I'm working with these
processors right now.<br>
> ><br>
> > On the new Ryzens, the power usage is
actually quite tunable.<br>
> > There are three settings.<br>
> ><br>
> > 1) Package Power Tracking: The PPT threshold
is the allowed socket<br>
> power<br>
> > consumption permitted across the voltage
rails supplying the<br>
> socket.<br>
> ><br>
> > 2) Thermal Design Current: The maximum
current (TDC) (amps) that can<br>
> be<br>
> > delivered by a specific motherboard's
voltage regulator<br>
> configuration in<br>
> > thermally-constrained scenarios.<br>
> ><br>
> > 3) Electrical Design Current: The maximum
current (EDC) (amps) that<br>
> can be<br>
> > delivered by a specific motherboard's
voltage regulator<br>
> configuration in a<br>
> > peak ("spike") condition for a short period
of time.<br>
> ><br>
> > My goal is to tweak the 105W TDP R7-5800X so
it draws power like<br>
> the<br>
> > 65W-TDP R5-5600X<br>
> ><br>
> > This is desk-side cluster low power stuff.<br>
> > I am using extension cable-plug for Limulus
blades that have an<br>
> in-line<br>
> > current meter (normally used for solar
panels).<br>
> > Now I can load them up and watch exactly how
much current is being<br>
> pulled<br>
> > across the 12V rails.<br>
> ><br>
> > If you need more info, let me know<br>
> ><br>
> > --<br>
> > Doug<br>
> ><br>
> >> The Beowulf list archives seem to end in
July 2021.<br>
> >> I was looking for Doug Eadline's post on
limiting AMD power and<br>
> the<br>
> >> results on performance.<br>
> >><br>
> >> John H<br>
> >>
_______________________________________________<br>
> >> Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin<br>
> >> Computing To change your subscription
(digest mode or unsubscribe)<br>
> >> visit<br>
> >> <a
href="https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:*__;Lw!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDvUGSdHI$"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:*__;Lw!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDvUGSdHI$</a><br>
> >> /<a
href="http://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
> >><br>
> ><br>
> ><br>
> > --<br>
> > Doug<br>
> ><br>
> >
_______________________________________________<br>
> > Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin<br>
> Computing<br>
> > To change your subscription (digest mode or
unsubscribe) visit<br>
> > <a
href="https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:**Abeowulf.org*cgi-bin*mailman*listinfo*beowulf__;Ly8vLy8v!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDUP8JZUc$"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://urldefense.us/v3/__https://link.edgepilot.com/s/9c656d83/pBaaRl2iV0OmLHAXqkoDZQ?u=https:**Abeowulf.org*cgi-bin*mailman*listinfo*beowulf__;Ly8vLy8v!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDUP8JZUc$</a><br>
> ><br>
><br>
><br>
> --<br>
> Doug<br>
><br>
> _______________________________________________<br>
> Beowulf mailing list, <a
href="mailto:Beowulf@beowulf.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin<br>
> Computing<br>
> To change your subscription (digest mode or
unsubscribe) visit<br>
> <a
href="https://urldefense.us/v3/__https://beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDv6c1nNc$"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://urldefense.us/v3/__https://beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!f3kkkCuq3GKO288fxeGGHi3i-bsSY5P83PKu_svOVUISu7dkNygQtSvIpxHkE0XDv6c1nNc$</a><br>
><br>
><br>
<br>
<br>
-- <br>
Doug<br>
<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
target="_blank" rel="noreferrer" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit
<a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
</body>
</html>