high physical density cluster design - power/heat/rf questions

[Jim Lux]

>> For powersupplies and HA, we were going to use "lab" power supplies
>> and run a diode array to keep them from fighting too much.
>
>Saw someone post that most diodes will steal too much voltage to be
>able to maintain a steady ~+3V supply to the finicky CPUs unless you
>are very careful. I do have an electrical engineering friend tho... ;)


Power supply needs to be regulated to 5%... A typical Schottky diode might
have a forward voltage drop of 0.5 Volt (10%), and don't forget the loss in
the wiring, as well.  If you are using lab supplies, you could carefully
boost the voltage to compensate, or use the sense input. (paralleling gets
tricky though).

Your real problem is finding a big enough supply. 100W (nominal power
consumption for an ATX mobo), is 20A @ 5V.  Granted, the power is actually
spread across several voltages, and the supplies aren't all that efficient
(say 70%), but you're probably still looking at 10A on each board.  Gang up
10 boards and you're at 100A.. Gang up 24, and you're looking at 240A.  I'll
bet that the lab supply that puts out 240A @ 5V isn't going to be very
cheap.  Even if you divide it up among 4 supplies (@ 60A each) it is still a
chore.


>
>> Instead of x-smaller supplies, you can use 4-5 larger supplies and run
them
>> into a common harness to supply power.  You'll need 3.3v, 5v, 12v
supplies,
>> but it beats running 24 serparate supplies (IMHO) and if one dies, you
don't
>> lose the board, you just take a drop in supply until you replace it.
>>
>> For heat dissapation, we're in a CoLo facility.  Since getting to/from
the
>> individual video/network/mouse/keyboard/etc stuff is very rare
(hopefully)
>> once it's up, we were going to put a pair of box-fans (wind tunnel style)
>> in front and behind the box.  =)  In a CoLo, noise is not an issue.
>> Depending
>> on exact design, you might even get away with dropping the fans off of
the
>> individual boards and letting the windtunnel do that part, but that's got
>> problems if the tunnel dies and affects every processor in the box.


Better check your airflow requirements..  If a typical PC needs 30-40 CFM to
keep the temperatures reasonable (21 CFM for a micro ATX), a gang of 24 will
need 600-1000 CFM.  This is substantially more than the typical "box fan"
can push at any reasonable pressure drop.  A typical 1/4HP, 18"diameter fan
has a free air flow of about 2500CFM, dropping down to 1000 cfm at 0.3"
water column pressure drop.

Your idea of hooking right to the chiller might be a good one.  Typical HVAC
systems are designed to push against substantial pressure drop, and, of
course, the air will be coldest at that point.

>
>
>Like I said we may just take some accordion airduct hose from the
>ceiling and latch it onto the whole array. The Liebert is far more
>reliable than any box fan and is alarmed up the yingyang. Its possible



>> > Problem 1
>> > """""""""
>> > The problem is in the diagram above, the upside down board has another
>> board
>> > .5" above it - are these two boards going to leak RF like mad and
>> interefere
>> > with eachothers' operations? I assume there's not much to do there but
to
>> put
>> > a layer of grounded (to the cabinet) metal in between.  This will drive
up
>> the
>> > cabinet construction costs. I'd rather avoid this if possible.
>> >


I don't think that "interboard" EMI would be a big problem.


>> > Problem 3 - Power
>> > """""""""""""""""
>> > The power density here is going to be high. I need to mount 64 power
>> supplies
>> > in close proximity to the boards, another reason I might need to
maintain
>> > the 2x1 instead of 2x2 design. (2x1 allows easier access too).
>> >
>> >
>> >
>> > I am wondering if someone has a reliable way of wiring together
multiple
>> > boards per power supply? Whats the max density per supply? Can we
>> > go with redundant power supplies, like N+1? We dont need that much
>> > reliability (jobs are short, run on one machine and can be restarted
>> > elsewhere), but I am really looking for something thats going to
>> > reduce the cabling.

Standard PC power supplies may not parallel well, even with diodes (it is
something you would definitely want to test).  It will almost certainly be
model specific (i.e. what works with Brand X PS may not work with Brand Y).

However, running two boards off a 300W PS might be reasonably feasible.