[Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath

Vincent Diepeveen diep at xs4all.nl
Tue Sep 4 16:46:04 PDT 2012

On Sep 4, 2012, at 11:04 PM, Ellis H. Wilson III wrote:

> On 09/04/2012 04:59 PM, Joe Landman wrote:
>> On 09/04/2012 12:54 PM, Lux, Jim (337C) wrote:
>>> Hydrogen is cheaper than He and works even better.  Just make  
>>> sure you don't have any air leaks in (i.e. keep a bit of positive  
>>> pressure).  For the "server farm in a container" model, this  
>>> would work just fine.. leaks would just float up into the  
>>> atmosphere.
>>> @ 300K
>>> Air 26.2 mW/m*K
>>> He 156.7
>>> H2 186.9  !!!
>> [singing to the tune of "you light up my life"]
> I think you guys (Joe and RGB) missed the critical caveat Jim  
> mentioned
> in his first sentence about the "server farms in a container."  I
> understand this to mean the outdoor trailer container setup that  
> Google
> released in 2009, similar to the diagram in the following link:
> http://asset2.cbsistatic.com/cnwk.1d/i/bto/20090401/ 
> Google_data_centers-4.jpg
> Yes, Google does house these containers in a fairly basic building,  
> but
> there is no reason I can think of why it couldn't put them out in the
> open and run all wires, etc, into the ground instead.  I think they  
> just
> put them in a building for convenience to the maintainers, rather than
> for some property of the building itself that would enable the
> containers to work better.
> I still agree (see my email 3 back now) that there are major concerns
> with using hydrogen this way, but at massive scale and with massive
> precautions you might be able to extract some nice power savings with
> it.  Maybe even enough to tolerate a few going sky high.  Alas, my
> physical sciences skills are way to weak to get a bound on what those
> benefits might be.
> Best,
> ellis

Well maybe some google guy wants to comment, but if we speak about  
their plans and analyze it financial on paper,
then they're supposed to have a 100k servers here in Netherlands.  
It's located next to some very huge energy centrals,
so delivering power to 100k servers is not gonna cost them much.

What will it eat 100k commodity servers?

Honestely i've got no idea how many harddrives each machine gets  
equipped with in google.
Let's say they do it extremely cheap and just fill it up with the  
maximum amount of drives a motherboard can handle
with a single raid  card.

Let's say 24 drives.

that'll be 6-8 drives. Let's say losing 4 out of each 12 effectively  
to redundancy that'll mean roughly 16 drives
on each array for effective storage.

What's standard by now, 3 TB, yet this is a few years old so say a  
2TB or so. 15 * 2 = 30 TB.

So we look at a potential of a 30TB * 100k = 3000+ petabyte. as a  
careful guess for this datacenter from 2010.

This for a single datacenter theirs and they'll have i'd blindfolded  
guess another 500k servers elsewhere.

The actual total power the datacenter eats, i would guess it's a tad  
more than 20 megawatt.

That's not so much compared to its economic value.

As for power costs we're looking at something far beyond market price  
in that area. They don't know where to transport
the power to anyway, most power they intend to transport soon to  
Germany i'd guess and they're praying a tad too much
(regrettably doing more than praying) about electric cars (just  
calculate how inefficient electric cars are if you compare the
  entire PRACTICAL chain with a diesel car, also realizing that they  
won't recycle lithium-ion - i get to around 10% efficiency
for electric driven cars entire chain versus factors more efficiency  
for diesel cars - that's a total different discussion).

Marketprice if you consume above 10k kilowatthour a year is around a  
4.4 cents a kilowatthour here right now.
Let's say careful Google didn't negotiate too much about power price  
as we'll see that's not their main concern:

As 20 megawatt still isn't "big power consumption" where they'd build  
a central for, probably we're looking at a 1 cent a kilowatthour or so.

10 euro a megawatthour times 20 megawatt = 200 euro an hour.
8760 hours a year times 200 euro = 1.75 million euro

That's if google did *not* negotiate too much, as a tad older table i  
have here from that same company that's there,
they offer for factories factor 10 underneath market price compared  
to the 10 kilowatthour guys, provided you eat
in the gigawatthours a year and probably conditions such as a perfect  
from energy companies viewpoint; google is at such location.

More important i'd guess is being close to the ocean at a spot where  
big internet cables from entire europe can easily
get together, that's probably more of a concern.

What's price of 100k servers  * 24 disks of 2 TB?

Well, they're 97 euro in the shop here now. Assuming nearly no margin  
that would mean a production price of 40 dollar.

$40 * 24 * 100k = 4M * 24 = 96 million dollar worth of harddrives.  
Equipment that outdates rapidly and needs to get upgraded nonstop.
Fails nonstop.

Servicing a few million drives and a 100k computers, i'd guess you  
would want to robotize replacing failing equipment, especially
the replacement of drives, it means big personnel and a huge cost in  

A generic robot that replaces drives quickly, that's rather easy to  

Making all this 100x more complex with liquid cooling, that would  
make servicing costs exponential more expensive,
probably every component from everything gets more expensive.

In fact i already assumed a big power savings. If all drives would be  
operational at the same time you're looking at a massive power usage
a factor 2 more or so, nearly 40 megawatt.

By having majority of drives used for long term storage, only now and  
then waking them up, in fact by steering the redundancy in a clever  
like delaying some forms of redundancy updating the information at a  
later stage, one can obtain massive power savings i'd guess.

Probably the drives eat a watt or 8 when writing and asleep just a  
part of that.

Of course i assumed now cheap hardware, not dedicated HPC type  
equipment, which undoubtfully also are part of the datacenters  
hardware set.

Symbolically googling for information there, one sees actually a  
pretty limited truth and only guesstimates that outsiders did do. The  
is just another of such guesstimate, in this case mine.

My point is however that the price of the power is just a part of the  
total costs of such datacenter.

Just the storage costs of paying for harddrives already is  
overwhelming more expensive than power costs, and i didn't even use
actually costs i had on a paper here for companies using maybe a tad  
more power, yet not even located in fact next to some big energy

>> And you ...
>>     ... light up my server room
>>     ... You make it hot
>> And when sparks fly ...
>>     ... so do my servers in
>>     ... the ensuing explosion ...
>> (ok, I'll keep the day job ... sheesh!)
>>> Ar  17.9
>>> CH4 34.1
>>> He also is hard to come by..
>>> Jim Lux
>>> -----Original Message-----
>>> From: Robert G. Brown [mailto:rgb at phy.duke.edu]
>>> Sent: Tuesday, September 04, 2012 6:16 AM
>>> To: Lux, Jim (337C)
>>> Cc: Eugen Leitl; beowulf at beowulf.org
>>> Subject: Re: [Beowulf] Servers Too Hot? Intel Recommends a  
>>> Luxurious Oil Bath
>>> On Mon, 3 Sep 2012, Lux, Jim (337C) wrote:
>>>> I'll bet they have to change it more often than that.  This isnt
>>>> something like a pole transformer.
>>> Absolutely. Think of what you can do with a big vat of hot oil  
>>> handy in the workspace.  Buffalo Wings.  French Fries.  Chicken.   
>>> Fish.  The reason nobody does this is because OSHA prohibits it  
>>> -- it is a huge health hazard.  Not even Jolt Cola can keep you  
>>> thin in a sedentary profession with your own personal deep frier  
>>> as close as your server room.  Although you do have to change the  
>>> oil pretty often, as otherwise shrimp tails and bits of  
>>> overcooked tempura crust gunk up the memory and CPU.  Systems  
>>> people were dying like pudgy little flies of advanced  
>>> cardiovascular disease before the practice of using computers to  
>>> heat deep fat was banned.
>>> On a more serious note, one wonders why nobody has tried helium  
>>> instead.
>>> No, silly, not liquid helium, helium gas.  The reason they fill  
>>> windows with argon is that it has around 2/3 the thermal  
>>> conductivity of air, and hence is a better insulator.  This, in  
>>> turn, is because it is more massive -- conductivity is tightly  
>>> tied to mass and hence the speed of the molecules when they have  
>>> kT sorts of energies.
>>> Helium, OTOH, has six times the thermal conductivity of air, and  
>>> is relatively inexpensive.  The biggest downside I can think of  
>>> is that it requires a pretty good seal and thick walls to keep  
>>> the slippery little atoms from sliding right through to the  
>>> outside, and of course the fact that systems techs would always  
>>> be hitting up the helium tanks so that they could talk like  
>>> Donald Duck.  And you'd still have to refrigerate the outside of  
>>> the systems units.  But all of these things are still orders of  
>>> magnitude easier than with oil, and even things like cooling fans  
>>> work fine in Helium.  Maybe there are other problems -- lower  
>>> heat capacity to match its higher conductivity -- but it seems  
>>> like it is worth an experiment or two...
>>>       rgb
>>> Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
>>> Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305
>>> Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
>>> Computing
>>> To change your subscription (digest mode or unsubscribe) visit  
>>> http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
> Computing
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list