[Beowulf] A Cluster of Motherboard.

[Robert G. Brown]

On Thu, 10 Nov 2005, H.Vidal, Jr. wrote:

> What's remarkable to consuder is that one of the very largest
> (if not the largest?) data cluster systems in the world is a bare
> motherboard system, strapped together with lots of simple
> screws and Velcro.
>
> That's Google, in case you did not know. I was shocked to see
> this when I saw a presentation recently by one of the Google
> guns here in NYC (actually, the inventor of Froogle).
> He showed us pix of a bunch of nodes essentially
> sitting on some insulating material, screwed to a simple
> frame-style chassis with careful consideration of grounding
> and power. His point was to emphasize that google considers
> lots of very cheap, very simple nodes key to their growth, and cases
> are 'right out' when you go to this scale (he would not share the
> exact N of nodes with us, but alluded to something on the order of
> 100K, at that time, and this is *always* growing).
>
> Really quite fascinating, and so low-tech compared to the attendees'
> expectations.

Very interesting indeed.  Of course they DO have the resources and
economy of scale required to support a machine shop and testbed
laboratory where they probably try out different vanilla motherboards
and customize mounting etc as required.  I've also heard that their swap
out time on a node crash is pretty extraordinary -- minutes.  They have
hot and cold running techs with spares read to plug right in.

In comparison, I fairly recently toured one of IBM's webserver farms
expecting to see neat racks of netfinities or the like.  Instead I saw
-- heavy duty shelving with custom supertower cases, all lined up just
like the original beowulf(s) only not quite so neatly.  No rackmounts
here.  I think that the moral of the story is that when the economy of
scale reaches a certain point, it really does start to save real money
going with non-racked solutions.  Depending on all sorts of opportunity
cost issues, of course -- this presupposes a surplus of labor you're
paying for anyway so using their spare time to do the additional hand
work required by naked nodes or tower nodes (if any in the latter case)
while enabling one to save on rack/case expenses is good.

Another cost savings is potentially in the power supply itself.  IIRC
larger supplies are more efficient, so that they reduce energy
consumption overall.  I'm sure this helps motivate Google's solution as
well -- if they have 100K nodes at 100W each, that is 10 megawatts right
there (read -- millions of dollars worth of electricity and cooling
ALONE per year, depending on their discount).  Naked motherboards might
cut per-node consumption down to only 60-70 watts.  That's likely a
million-dollar-plus saving right there.

    rgb

>
> hv
>
> Robert G. Brown wrote:
>> On Thu, 10 Nov 2005, Josip Loncaric wrote:
>> 
>>> A cheap bare case can cost less than $20 -- and can save you a lot of 
>>> work, although perhaps not shelf space.  For safety, ease of installation, 
>>> and FCC EM reasons, cheap metal cases should be considered.
>> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu