[Beowulf] A Cluster of Motherboard.

[Jim Lux]

At 06:58 AM 11/11/2005, Robert G. Brown wrote:
>On Fri, 11 Nov 2005, Jim Lux wrote:
>
>>If you have an unusual packaging concern.. maybe it needs to be in a 
>>sealed box? maybe it has to fit into that antique grandfather clock?, 
>>etc. then you've got a case for custom design.
>
>Or if the commercial packaging available is still being sold at a hefty
>premium, opening up a potential window of CBA advantage.
>
>Let us recall that all of these arguments might have been applied to the
>creation of beowulfs in the first place, when perfectly good big iron
>supercomputers that were de facto clusters were available.  After all,
>it took lots of human time to set up that stack of PC's, buy shelving,
>screw around with wiring them together, and then (from a certain amount
>of bitter experience) wrestle with Linux (especially e.g. the SMP
>kernels and device driver issues) to get the whole thing to "work".


You raise an excellent point.  Of course, the early beowulfs were 
essentially hobby clusters, in that the builder was willing to spend a fair 
amount of time for the thrill of creation.  When you're doing it for the 
first time, spending valuable research grant money (or equally valuable 
uncompensated "free time") for labor to do it might be justifiable, if for 
no other reason than you are helping to bring the per unit cost for those 
who follow down the "learning curve".

It's the classic capital poor, time rich scenario that has driven artists 
in garretts, inventors in their garages, and scientists in wherever poor 
scientists work (a shack out back, a la Mme Curie?) to be creative.  You 
have a burning desire to answer some question, you can't afford a Cray to 
answer it, so you improvise, and thereby develop something useful for the ages.


But.. there's a limit to the scale of such things.  Have there been any 
100+node hobby clusters (i.e. ones where there wasn't funding specifically 
provided to get the cluster built), except perhaps the StoneSouperComputer 
at ORNL?  (having shot my figurative mouth off about this before, I expect 
no less than 3 emails with links to specific examples of people building 
dreadnought class 1000 processor clusters in their garage in their spare time).

Home/self built packaging is, I'd venture to guess, a way to do "proof of 
concept" on a limited budget.  Or, it's a somewhat unique artistic medium.

>The reasons that it >>was<< economically efficient to build beowulfish
>
>(order of magnitude ten, not even unity).  Opportunity cost labor is
>trickier, but in most cases one would argue that it increases the
>cost-efficiency of clusters still further, at least in certain ranges of
>cluster size and with certain degrees of OC expertise available.

Exactly.. I'd say the size of 4-16 (just to keep it powers of 2) nodes is 
where the "sweet spot" is for proof of concept/hobby/self packaged.  You're 
still at a reasonable overall power consumption (e.g. you can plug it into 
one circuit and not start a fire, nor cause your a/c bill to skyrocket). 
And, it's going to be in the "fit under or next to a desk" sort of size.

I'd argue that for >100 nodes, the job is bigger than one person can do, so 
the issue of finding other "volunteers" comes in.



>That appears to be less true as blades are moving more towards the mass
>market and some low rent "micro board on cheap plate blades" appear --
>pretty much what you'd make yourself but done professionally and at an
>efficient scale, sans bells, whistles, and hefty markup.

I think, and others have also pointed out, there might be a market for 
someone who makes some sort of cheap and cheerful vertical racking 
hardware, adapted from existing "card cage" technology. That is, the "box" 
is available already from some other source, and someone makes a flat plate 
upon which a standard mobo and peripherals are mounted that slides into the 
cage.

My previous experience with getting moderate quantities of sheet metal 
enclosures made makes me think that the basic per unit cost of such a thing 
might be in the $15-20 range, for quantities in the tens or low 
hundreds.  A few hundred dollars for NRE setup, then the rest is machine 
time and handling (the raw material is steel, and exceedingly cheap)