[Beowulf] Re: Estimating cluster power consumption (r reynoldson)
edkarns at firewirestuff.com
Thu Dec 15 14:56:06 PST 2005
On Mr. Reynoldson's questions below and Mr. Lux' responses.
It is advised to run four power line branches into your system room
with quad "four hole" outlets on each branch (total of 16 power holes
in 8 outlet sets in 4 utility boxes) and with individual circuit
breakers on each branch ... Two runs on one phase (2 branches) and
two runs on the other using 12 or 14 gauge wiring. There are several
peculiarities about running multiple switching power supplies on a
single 120 / 220 VAC branch (each system has one). It turns out that
switching power supplies (as found in standard PC cases) can actually
"sync up" and cause mysterious crashes, the switching power supplies
placing a 400 Hertz demand wave on top of the 50 / 60 cycle power
wave. By connecting half of your load to one phase (one of two
branches) and half to the other phase on separate circuit breakers,
the 400 Hertz switching demand load is "spread" over the two phases
and your overall load is "balanced".
" ... What's the least expensive way of getting more electricity to
the room? I can't do it myself (at least I think I can't). ..." ...
Note: Canadian household power may be 220 VAC, split into two phases
(in some provinces). Check before you buy anything or "plumb up" your
power lines to the service panel.
Total operating load = 16 nodes X average 200 watts each = 3200+
watts. (This is a "running load", not a start up load which could
easily exceed 5000 watts, even without hard drives and monitors.)
Using four wiring branches, figure 1000 watts per branch = #14 AWG
wiring. I would use 14 gauge "3-wire w/ground Romex" household wire
(Red, Black, White & ground) making two home runs, two branches in
each run, back to the main service panel ... connected to four, 15
Amp circuit breakers (one for each branch, two red wires, two black
wires). Plug each pair of breakers into opposite phases in the main
service panel to balancing the load as per above. (This will give you
a modest expansion allowance up to maybe 20 to 24 nodes.) Note that 3-
wire w/ground, 14 gauge wiring is commonly available and relatively
inexpensive = here in California it is less than US$1.00 per foot and
non-GFI, 15 Amp breakers are less than US$20.00 each.
Ground loops? Connect all white (common wires) and all ground wires
in the main service panel to the central bus bar and make sure you
have a very good "local ground" in the main service panel. (Watch
out! ... ZZZZZZTTT = tag, you've been hit! and are now headed for the
hospital ... unless you turned off the main breaker before you opened
the service panel.)
" ... Is there any chance of a fire hazzard if the breaker is
overloaded or will the breaker just trip? ..." Yes! ... but if you
have a qualified local electrician examine your work before you crank
it up, there should be no problem found.
" ... Would running the cluster diskless help much with power
consumption? What about no CD and floppy drive? ..." ... Yes, but
only to a limited extent. Trying to micro manage your power usage by
removing drive mechanisms will be counter productive in the long run.
Go ahead and leave the hard drives, etc., in the systems. The amount
of power saved per node is less than 5% ... and being able to boot
and run each system individually from its own hard drive will save
you a world of time and possible frustration.
" ... What are some good references to answer such
questions? ..." ... make good friends with a qualified electrician.
Mr. Lux: " ... so, you can run all 5 boxes on the 15A
circuit ..." ... Yes, of course. This may however stretch the limits
of a many breakers. Five nodes X 200 watts per = 1000 watts total,
minimum ... and assumes that the internal switching power supplies
will never "sync up" making momentary power demands of 2000 or more
watts every few cycles. I would not recommend more than four boxes
per 15 Amp circuit breaker.
Mr. Lux" " ... Figure 200W/node (for lack of a better estimate) =
3200W. 3200W/110V = 29Amps 29A * 1.2 (design capacity factor) =
branch circuit capacity of 34 A. ..." ... In Canada with 220 VAC
power, a can of worms opens up ... dealing strictly with Watts and
ignoring voltage considerations, you should not try to stress the
circuit breakers beyond 50 to 60 % of rating = use a 2.00 "design
capacity factor" and everything will run cool, system nodes, wiring
and circuit breakers = greatly reduced fire hazards all around =
greatly improved reliability of the whole system.
Mr. Lux: " ... 6) What's the least expensive way of getting more
electricity to the room? ... Extension cords from another branch
circuit <grin>. ..." ... Ah, yes, and a great big grin it is, too ...
Kids, don't try this at home. Multiple extensions cords are
absolutely forbidden in every server farm I have ever seen. Imaging
that you are close to the end of your program run and you invite a
few friends over to celebrate ... and one of them decides to play a
little prank and pull an extension cord plug to make room for the wet
bar ... Ah, yes ... big grins all around. Besided, I was lead to
believe that fire hazards were of some modest concern = no
extensions cords allowed.
(A member of IBEW, Journeyman & Commercial Electrical Contractor
On Dec 15, 2005, at 7:16 AM, beowulf-request at beowulf.org wrote:
> 7. Estimating cluster power consumption (r reynoldson)
> 8. Re: Estimating cluster power consumption (Jim Lux)
> Message: 7
> Date: Wed, 14 Dec 2005 17:28:21 -0800 (PST)
> From: r reynoldson <rreynoldson_ng_mail at yahoo.com>
> Subject: [Beowulf] Estimating cluster power consumption
> To: beowulf at beowulf.org
> Message-ID: <20051215012821.15219.qmail at web54108.mail.yahoo.com>
> Content-Type: text/plain; charset=iso-8859-1
> ... First I should say that I just have a little
> experimental cluster (5 nodes including master) right
> now, but have an opportunity to more than double this
> size very cheaply. The compute nodes are a mixture of
> 366 - 450 Mhz boxes, 128 - 256 Mb ram. Currently they
> have hard drives, but I'm seriously considering going
> diskless (for the learning experience, amoung other
> things). Also, the compute nodes are just boxes (no
> monitor, keyboard, or mouse). The master node has an
> 1800 AMD cpu, 512 ram, and 60 Gb hard drive. ... A typical cluster
> run could last anywhere from a day to several weeks
> (possibly months?).
> My house, which contains my cluster room, is rather
> old. Looking at my breaker panel, it looks like my
> cluster room shares a 15 amp breaker with an adjoining
> room (sad, I know).
> 1) About how many of the above mentioned nodes would I
> be able to safely run?
> 2) Is there any chance of a fire hazzard if the
> breaker is overloaded or will the breaker just trip?
> 3) How does one estimate how many nodes they can
> safely run?
> 4) Would running the cluster disk-less help much with
> power consumption? What about no CD and floppy drive?
> 5) How much juice should be flowing to the room to run
> a cluster of 16 such nodes?
> 6) What's the least expensive way of getting more
> electricity to the room? I can't do it myself (at
> least I think I can't). Which is the more desirable --
> increasing the existing breaker amperage (if possible)
> or getting a second breaker box (or replacing the
> existing box with a bigger and better box)? The house
> has copper wiring in the basement, which may affect
> the cost (I also live in Canada too, eh!).
> 7) What are some good references to answer such
> questions? I'd even spend money on a book, if it were
> recommended highly enough.
> Thanks in advance
> At 05:28 PM 12/14/2005, r reynoldson wrote: ( James Lux replying)
>> 1) About how many of the above mentioned nodes would I
>> be able to safely run?
> It's the rare single box that draws more than 200W (unless you've
> got quad
> xeons or something) without the monitor. You generally don't want
> to load a
> branch circuit to more than 80% of the rated capacity (0.8*15 =
> 12A, here)
> 12A is about 1200W (actually more like 1400, but fudging on the low
> here is a "good thing").. so, you can run all 5 boxes on the 15A
> circuit. (on a more practical note, I used to do this very thing,
> run 5
> 386,486, and Pentium 1 boxes on a single branch circuit in my
> apartmen, it
> was the LaserJet III that caused the problems, especially if I
> decided to
>> 2) Is there any chance of a fire hazzard if the
>> breaker is overloaded or will the breaker just trip?
> The breaker "should" just trip on an overcurrent. However, if the
> is a gazillion years old it might not. Also, if the wall
> receptacle is old
> and decrepit (most particularly, the wires on the back of the
> then you could be drawing a normal amount of current (i.e. 10A),
> and the
> receptacle will still get hot enough to cause problems. Unusual,
> but it
>> 3) How does one estimate how many nodes they can
>> safely run?
> Buy yourself a Kill-A-Watt for $30 and actually measure the power
> (http://www.efi.org/ is one source, there are others)
>> 4) Would running the cluster diskless help much with
>> power consumption? What about no CD and floppy drive?
> Every little bit helps
>> 5) How much juice should be flowing to the room to run
>> a cluster of 16 such nodes?
> Figure 200W/node (for lack of a better estimate) = 3200W. 3200W/
> 110V =
> 29Amps 29A * 1.2 (design capacity factor) = branch circuit
> capacity of 34 A.
>> 6) What's the least expensive way of getting more
>> electricity to the room?
> Extension cords from another branch circuit <grin>.
>> I can't do it myself (at
>> least I think I can't). Which is the more desirable --
>> increasing the existing breaker amperage (if possible)
>> or getting a second breaker box (or replacing the
>> existing box with a bigger and better box)? The house
>> has copper wiring in the basement, which may affect
>> the cost (I also live in Canada too, eh!).
> A lot depends on what else in the house needs to be fixed when they
> get in
> to make the changes. Call your local electrician for an estimate,
> and they
> can tell you what the tradeoffs are.
>> 7) What are some good references to answer such
>> questions? I'd even spend money on a book, if it were
>> recommended highly enough.
> Read RGBs online beowulf book at the duke brahma site. He has a whole
> chapter on infrastructure (power, HVAC, etc) issues.
>> Thanks in advance
> James Lux, P.E.
> Spacecraft Radio Frequency Subsystems Group
> Flight Communications Systems Section
> Jet Propulsion Laboratory, Mail Stop 161-213
> 4800 Oak Grove Drive
> Pasadena CA 91109
> tel: (818)354-2075
> fax: (818)393-6875
> Beowulf mailing list
> Beowulf at beowulf.org
> End of Beowulf Digest, Vol 22, Issue 19
USBStuff / FireWireStuff / WireLessStuf / FiberStuf ... and much more
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf