[Beowulf] Beowulf Digest, Vol 125, Issue 13
Larry Lesser
larry at pssclabs.com
Mon Jul 28 12:48:47 PDT 2014
Prentice:
What cpu or cpus, amount of memory and how many hard drives are on a single node. I may have the power draw for that configuration.
Larry
-----Original Message-----
From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of beowulf-request at beowulf.org
Sent: Monday, July 28, 2014 12:00 PM
To: beowulf at beowulf.org
Subject: Beowulf Digest, Vol 125, Issue 13
Send Beowulf mailing list submissions to
beowulf at beowulf.org
To subscribe or unsubscribe via the World Wide Web, visit
http://www.beowulf.org/mailman/listinfo/beowulf
or, via email, send a message with subject or body 'help' to
beowulf-request at beowulf.org
You can reach the person managing the list at
beowulf-owner at beowulf.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Beowulf digest..."
Today's Topics:
1. Power draw of cluster nodes under heavy load (Prentice Bisbal)
2. Re: Power draw of cluster nodes under heavy load (Jeff White)
3. Re: Power draw of cluster nodes under heavy load
(Michael Di Domenico)
4. Re: Power draw of cluster nodes under heavy load (Mark Hahn)
5. Re: Power draw of cluster nodes under heavy load (Prentice Bisbal)
6. Re: Power draw of cluster nodes under heavy load (Prentice Bisbal)
----------------------------------------------------------------------
Message: 1
Date: Mon, 28 Jul 2014 10:51:12 -0400
From: Prentice Bisbal <prentice.bisbal at rutgers.edu>
To: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID: <53D66360.1050609 at rutgers.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Beowulfers,
Are any of you monitoring the power draw on your clusters? If so, can any of you provide me with some statistics on your power draw under heavy load? Ideally, I'm looking for the power load for a worst-case scenario, such as running HPL, on a per-rack basis. If you can provide me with the power draw and a description of the hardware, that would be great.
I have some numbers from a friend who lurks on this list, but the more data points I have, the better I can justify my power requirements for a new cluster purchase I'm working on.
--
Prentice
------------------------------
Message: 2
Date: Mon, 28 Jul 2014 13:29:35 -0400
From: Jeff White <jaw171 at pitt.edu>
To: <beowulf at beowulf.org>
Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID: <53D6887F.60303 at pitt.edu>
Content-Type: text/plain; format=flowed; charset="ISO-8859-1"
Power draw will vary greatly depending on many factors. Where I am at
we currently have 16 racks of HPC equipment (compute nodes, storage,
network gear, etc.) using about 140kVA but can use up to 160 kVA. A
single rack with 26 compute nodes each with 64 cores worth of AMD 6276
(Supermicro boxes) is using about 18 kW across the PDUs, 3 phase at 240
volts, with most of the nodes at 100% CPU usage.
Jeff White - GNU+Linux Systems Administrator
University of Pittsburgh - CSSD
On 07/28/2014 10:51 AM, Prentice Bisbal wrote:
> Beowulfers,
>
> Are any of you monitoring the power draw on your clusters? If so, can
> any of you provide me with some statistics on your power draw under
> heavy load? Ideally, I'm looking for the power load for a worst-case
> scenario, such as running HPL, on a per-rack basis. If you can provide
> me with the power draw and a description of the hardware, that would be
> great.
>
> I have some numbers from a friend who lurks on this list, but the more
> data points I have, the better I can justify my power requirements for a
> new cluster purchase I'm working on.
>
------------------------------
Message: 3
Date: Mon, 28 Jul 2014 13:38:21 -0400
From: Michael Di Domenico <mdidomenico4 at gmail.com>
Cc: Beowulf Mailing List <beowulf at beowulf.org>
Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID:
<CABOsP2O7qNNHxC0nvUs=s22BKQFVbNFGRhahMht_eKP87OYSpA at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
I can't go into specifics. with HPL i usually can't top 80-85% of
full power draw per cabinet. we have cabinets ranging from 17kw,
25kw, and 35kw. some of our user codes can push the machines to 90%
of full load. this is shown on both amd 4-socket nodes and intel
dual-socket with gpu's. we're running 208v three phase to the racks
and 208v single phase to each server
On Mon, Jul 28, 2014 at 1:29 PM, Jeff White <jaw171 at pitt.edu> wrote:
> Power draw will vary greatly depending on many factors. Where I am at we
> currently have 16 racks of HPC equipment (compute nodes, storage, network
> gear, etc.) using about 140kVA but can use up to 160 kVA. A single rack
> with 26 compute nodes each with 64 cores worth of AMD 6276 (Supermicro
> boxes) is using about 18 kW across the PDUs, 3 phase at 240 volts, with most
> of the nodes at 100% CPU usage.
>
> Jeff White - GNU+Linux Systems Administrator
> University of Pittsburgh - CSSD
>
>
> On 07/28/2014 10:51 AM, Prentice Bisbal wrote:
>>
>> Beowulfers,
>>
>> Are any of you monitoring the power draw on your clusters? If so, can
>> any of you provide me with some statistics on your power draw under
>> heavy load? Ideally, I'm looking for the power load for a worst-case
>> scenario, such as running HPL, on a per-rack basis. If you can provide
>> me with the power draw and a description of the hardware, that would be
>> great.
>>
>> I have some numbers from a friend who lurks on this list, but the more
>> data points I have, the better I can justify my power requirements for a
>> new cluster purchase I'm working on.
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
------------------------------
Message: 4
Date: Mon, 28 Jul 2014 14:13:48 -0400 (EDT)
From: Mark Hahn <hahn at mcmaster.ca>
To: Prentice Bisbal <prentice.bisbal at rutgers.edu>
Cc: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID:
<alpine.LFD.2.02.1407281220230.30037 at coffee.psychology.mcmaster.ca>
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
> Are any of you monitoring the power draw on your clusters? If so, can any of
> you provide me with some statistics on your power draw under heavy load?
good question; it's something that deserves more attention and coverage.
ATM, I can only provide one non-answer:
http://www.sharcnet.ca/~hahn/saw-power-by-node.png
this is active mixed-user load (45 unrelated users, approximately 85%
CPU utilization due to memory scheduling and job layout constraints).
this an older cluster, HP dual-socket E5440 (2.833G) whose IPMI happens to
return nice power measures.
> Ideally, I'm looking for the power load for a worst-case scenario, such as
> running HPL, on a per-rack basis.
I don't understand the "per-rack" part - aren't you interested in per-node?
> I have some numbers from a friend who lurks on this list, but the more data
> points I have, the better I can justify my power requirements for a new
> cluster purchase I'm working on.
my experience is that vendors are useless in this regard: they always want
to quote the PSU max rating, and then often don't even use the number right.
(ie, put all the low-dissipation stuff like networking together, etc.)
has anyone tried to rate the accuracy of vendor power calculators?
at least a few years ago, they were absurdly inflated.
regards, mark hahn.
------------------------------
Message: 5
Date: Mon, 28 Jul 2014 14:53:05 -0400
From: Prentice Bisbal <prentice.bisbal at rutgers.edu>
To: Mark Hahn <hahn at mcmaster.ca>
Cc: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID: <53D69C11.6000109 at rutgers.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 07/28/2014 02:13 PM, Mark Hahn wrote:
>> Are any of you monitoring the power draw on your clusters? If so, can
>> any of you provide me with some statistics on your power draw under
>> heavy load?
>
> good question; it's something that deserves more attention and coverage.
>
> ATM, I can only provide one non-answer:
>
> http://www.sharcnet.ca/~hahn/saw-power-by-node.png
>
> this is active mixed-user load (45 unrelated users, approximately 85%
> CPU utilization due to memory scheduling and job layout constraints).
> this an older cluster, HP dual-socket E5440 (2.833G) whose IPMI
> happens to
> return nice power measures.
Thanks. That image is more helpful than you think - I didn't even think
of using IPMI to report power consumption. Using that, I could run HPL
on some nodes here and get measurements.
>
>
>> Ideally, I'm looking for the power load for a worst-case scenario,
>> such as running HPL, on a per-rack basis.
>
> I don't understand the "per-rack" part - aren't you interested in
> per-node?
Ideally, per-node is even better, but I figured most measurements would
be at the PDU or circuit level, with one or two PDUs/Circuits per rack.
I figured this is the granularity most people are measuring at, which is
why I asked that way.
>
>
>> I have some numbers from a friend who lurks on this list, but the
>> more data points I have, the better I can justify my power
>> requirements for a new cluster purchase I'm working on.
>
> my experience is that vendors are useless in this regard: they always
> want
> to quote the PSU max rating, and then often don't even use the number
> right.
> (ie, put all the low-dissipation stuff like networking together, etc.)
>
> has anyone tried to rate the accuracy of vendor power calculators?
> at least a few years ago, they were absurdly inflated.
This is why I'm asking for actual, measured numbers. I read a whitepaper
by APC or Raritan that said that if you go with the nameplate on a PDU,
you can oversize your power requirements by a factor of 2x. For HPC, I
imagine it wouldn't be that extreme, since cluster nodes tend to be at
100% more of the time and therefore use more power. One vendor said they
assume 60% - 90% of nameplate ratings when estimating power needs, which
is still a pretty broad range.
>
> regards, mark hahn.
------------------------------
Message: 6
Date: Mon, 28 Jul 2014 14:55:38 -0400
From: Prentice Bisbal <prentice.bisbal at rutgers.edu>
To: beowulf at beowulf.org
Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load
Message-ID: <53D69CAA.20609 at rutgers.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 07/28/2014 01:29 PM, Jeff White wrote:
> Power draw will vary greatly depending on many factors. Where I am at
> we currently have 16 racks of HPC equipment (compute nodes, storage,
> network gear, etc.) using about 140kVA but can use up to 160 kVA. A
> single rack with 26 compute nodes each with 64 cores worth of AMD 6276
> (Supermicro boxes) is using about 18 kW across the PDUs, 3 phase at
> 240 volts, with most of the nodes at 100% CPU usage.
Agreed there's a lot of variability. Since I don't exactly what's going
in my new space yet, I'm looking for everyone's input to come up with an
average, or ballpark amount. the 5 - 10 kW one vendor specified seems
waaaay too low for a rack of high-density HPC nodes running at or near
100% utilization.
>
> Jeff White - GNU+Linux Systems Administrator
> University of Pittsburgh - CSSD
>
> On 07/28/2014 10:51 AM, Prentice Bisbal wrote:
>> Beowulfers,
>>
>> Are any of you monitoring the power draw on your clusters? If so, can
>> any of you provide me with some statistics on your power draw under
>> heavy load? Ideally, I'm looking for the power load for a worst-case
>> scenario, such as running HPL, on a per-rack basis. If you can provide
>> me with the power draw and a description of the hardware, that would be
>> great.
>>
>> I have some numbers from a friend who lurks on this list, but the more
>> data points I have, the better I can justify my power requirements for a
>> new cluster purchase I'm working on.
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
------------------------------
Subject: Digest Footer
_______________________________________________
Beowulf mailing list
Beowulf at beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf
------------------------------
End of Beowulf Digest, Vol 125, Issue 13
****************************************
More information about the Beowulf
mailing list