[Beowulf] A Cluster of Motherboard.

[Vincent Diepeveen]

At 12:29 PM 11/10/2005 -0500, Robert G. Brown wrote:
>On Thu, 10 Nov 2005, H.Vidal, Jr. wrote:
>
>> What's remarkable to consuder is that one of the very largest
>> (if not the largest?) data cluster systems in the world is a bare
>> motherboard system, strapped together with lots of simple
>> screws and Velcro.
>>
>> That's Google, in case you did not know. I was shocked to see
>> this when I saw a presentation recently by one of the Google
>> guns here in NYC (actually, the inventor of Froogle).
>> He showed us pix of a bunch of nodes essentially
>> sitting on some insulating material, screwed to a simple
>> frame-style chassis with careful consideration of grounding
>> and power. His point was to emphasize that google considers
>> lots of very cheap, very simple nodes key to their growth, and cases
>> are 'right out' when you go to this scale (he would not share the
>> exact N of nodes with us, but alluded to something on the order of
>> 100K, at that time, and this is *always* growing).
>>
>> Really quite fascinating, and so low-tech compared to the attendees'
>> expectations.
>
>Very interesting indeed.  Of course they DO have the resources and
>economy of scale required to support a machine shop and testbed
>laboratory where they probably try out different vanilla motherboards
>and customize mounting etc as required.  I've also heard that their swap
>out time on a node crash is pretty extraordinary -- minutes.  They have
>hot and cold running techs with spares read to plug right in.
>
>In comparison, I fairly recently toured one of IBM's webserver farms
>expecting to see neat racks of netfinities or the like.  Instead I saw
>-- heavy duty shelving with custom supertower cases, all lined up just
>like the original beowulf(s) only not quite so neatly.  No rackmounts
>here.  I think that the moral of the story is that when the economy of
>scale reaches a certain point, it really does start to save real money
>going with non-racked solutions.  Depending on all sorts of opportunity
>cost issues, of course -- this presupposes a surplus of labor you're
>paying for anyway so using their spare time to do the additional hand
>work required by naked nodes or tower nodes (if any in the latter case)
>while enabling one to save on rack/case expenses is good.
>
>Another cost savings is potentially in the power supply itself.  IIRC
>larger supplies are more efficient, so that they reduce energy
>consumption overall.  I'm sure this helps motivate Google's solution as
>well -- if they have 100K nodes at 100W each, that is 10 megawatts right
>there (read -- millions of dollars worth of electricity and cooling
>ALONE per year, depending on their discount).  Naked motherboards might
>cut per-node consumption down to only 60-70 watts.  That's likely a
>million-dollar-plus saving right there.

If you consume 10 megawatt at 1 spot, like a plastic factory here is
consuming 10 megawatt, you get power roughly factor 20 cheaper than cost of
the power in your house. Many electricity companies will fight for your
contract in such case.

For the power bill, the thing that counts most is your peak usage, not your
*average* use.

>    rgb
>
>>
>> hv
>>
>> Robert G. Brown wrote:
>>> On Thu, 10 Nov 2005, Josip Loncaric wrote:
>>> 
>>>> A cheap bare case can cost less than $20 -- and can save you a lot of 
>>>> work, although perhaps not shelf space.  For safety, ease of
installation, 
>>>> and FCC EM reasons, cheap metal cases should be considered.
>>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
>-- 
>Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
>Duke University Dept. of Physics, Box 90305
>Durham, N.C. 27708-0305
>Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
>
>