cookie tray cluster Re: [Beowulf] Cluster newbie, power recommendations

Wed Mar 22 06:27:05 PST 2006

On Tue, Mar 21, 2006 at 06:58:05AM -0500, Robert G. Brown wrote:
Subject: Re: [Beowulf] Cluster newbie, power recommendations
> On Mon, 20 Mar 2006, Charlie Peck wrote:

> >signal back to the power supply.  I believe the site that RGB is referring 
> >to is http://joule.bu.edu/~hazen/LinuxCluster.  If this isn't enough get 

> Yeah, that looks like it.  Kinda cute in its own way.  And you don't
> "have" to buy stock aluminum or steel and cut and bend it into shelf
> trays for the motherboards, BTW -- there was a design somewhere that
> mounted the motherboards onto cookie sheets and "racked" them in a
> standard cookie-sheet rack, or a rack like the ones they use in
> cafeterias.  Slide in, slide out.

I believe I originated the cookie tray cluster idea, and then later
first mentioned it on the list here:

  http://www.beowulf.org/archive/2005-November/014108.html

Unfortunately, I have no website on it, as I was reluctant to write it
up until I had an actual finished working cluster to take pictures of,
etc.  (Which I still don't have - funny how real work and other
distractions tend to interrupt one's hobbies like that...)

A hugely useful addition to the basic cookie tray idea was Jim Lux's
suggestion here:

  http://www.beowulf.org/archive/2005-November/014088.html
  http://www.beowulf.org/archive/2005-November/014091.html

So far the double sided foam tape seems to work very well!  I've tried
it on a SMALL scale - 3 motherboards.  They run memtest86+ just fine,
but I have not yet tried network booting Linux on them with Warewulf
and actually using them heavily in a cluster.

I exchanged some private emails back and forth with other interested
experimenters about that, but I don't think it's been mentioned on the
list yet, so here are further details:

I purchased a roll of "3M Double Coated Urethane Foam Tape 4004
Off-White, 1 in x 18 yd", here:

  http://www.hillas.com/Products/3M_Double_Coated_Foam_Tapes/3M_4004_1_X_18.asp

Normal motherboard standoffs raise the board 0.25" off the case.  This
3M 4004 tape is nominally 0.25" thick, but once mounted it's actually
a bit thinner.

The foam tape is very firm, you have to push HARD to compress it much,
and then it gradually springs back.  Thus, the tape alone seems to do
a good job keeping the knobs of solder and other stuff on the back of
the motherboard away from the aluminum cookie tray, even when you
press down hard while inserting memory sticks or PCI cards.  This is
good, you do NOT need any additional rigid standoffs on the back of
the motherboard.

Note, I have NOT yet tried removing one of those taped down
motherboards...

The tape is somewhat expensive, $46.44 + shipping per roll.  For each
of my microATX motherboards, I used 3 strips about 9" long each, so I
should be able to do about 18*3*12 /27 = 24 motherboards from that one
roll.  For ATX boards I suppose 1 roll would let you do 18 or 19
motherboards.  (And you could probably get away with only 2 strips of
tape rather than my 3, if you cared.)

It's still a bit time consuming to place the tape and mount the
boards, but it is EASY.  Drilling holes for standoffs both takes
longer and is much easier to get wrong.  The double-sided tape is
certainly convenient, and I plan to try a thinner version for mounting
power supplies, hard drives, etc. as well.

In addition to the foam tape, 3M's velcro-like "Reclosable Fastener"
tape materials (e.g., SJ3540) might also be interesting for mounting
motherboards:

  http://products3.3m.com/catalog/us/en001/manufacturing_industry/engineered_adhesives/node_GSTHK7XYXKbe/root_GST1T4S9TCgv/vroot_WHG0MWH6QPge/gvel_QM7L6GDVFXgl/theme_us_adhesivetape_3_0/command_AbcPageHandler/output_html

As far as installing stuff on the cookie trays themselves goes, I
mostly talked about that in my post back in November.  But here are
some additional bits:

Figuring out what fits is most conveniently done in person with an
actual cookie tray and actual tray rack, as some of the dimensions are
sloppy.  The tray itself has angled lips, you lose some space on the
sides due to the rack rails extending over the edge of each tray, the
tray slides side-to-side a bit in the rack, etc.  The nice thing is
that none of this mounting hardware is too expensive, so just buying
it and fooling around isn't much problem.

Remember that the tiny fans in 1U power supplies have an annoying
high-pitched whine which tends to carry.  So if you use 1U supplies,
you definitely want the cluster behind some kind of door, NOT sitting
right next to your desk.

The motherboards I'm using are microATX (9.5" x 9.5").  2 of those
plus 2 1U power supplies will easily fit on 1 cookie tray.  2 ATX
motherboards will also definitely fit on a single cookie tray, and
should fit even with 2 1U power supplies as well.  With any
motherboard substantially larger than ATX, you're virtually guaranteed
to fit only 1 board per tray at most.

Since my microATX boards are 9.5" square, after mounting them flush up
against the long front edge of the cookie tray, I have free space
about 6.75" deep at the back of the tray, all the way across.

A standard 3.5" hard drive is thin enough that if you mount it flush
to the cookie try, it will NOT stick up over the 1" tall lip of the
tray at all.  This means that you can mount it all the way out to the
edge of the tray, even underneath where the tray holding rails stick
in, saving a bit of space.

There ARE 1U power supplies like the SPI FSP250-50PBL that are only
3.25" wide, so I could easily fit two of them side by side in that
6.75" space.  However, my Emacs 1U power supplies are 4" wide so I
can't do that.  To mount 2 supplies, I need to put them tail to tail.
That works fine but takes up most of the free space on the tray, I
could MAYBE squeeze 1 hard drive flat onto the tray next to the power
supplies but definitely not 2.

I could mount 2 1U power supplies on top of each other, but that would
require a 1 tray every 3 slots spacing.  A better idea, if you want
disks, is to mount a 3.5" hard drisk on TOP of a 1U power supply -
that fits fine with a 1 every 2 slot tray spacing.  This is basically
free space for disks, as nothing else is going to fit on top of those
power supplies.

So if I want to, on each tray spaced 1 every 2 slots, I can definitely
fit 2 microATX motherboards, 2 of my 4" wide 1U power supplies, and 2
hard drives (1 on top of each power supply).  ATX motherboards are
about 12" x 9.5" rather than 9.5 x 9.5, so you should still be able to
use the exact same arrangment even with ATX rather than microATX
boards.

And of course, to save even more space, you could experiment with
Y-cables to see if you can reliably power 2 motherboards with only 1
power supply, as discussed elsewhere on the Beowulf list.  They used
to be cheaper, c. $8.50 each for 10+ shipped, rather than the $15.25
they want now, but the power supply Y-cables I found were:

  CB-Z-Y2020-EXT "ZIPPY Power Cable Splitter: ATX 20 pin to Two ATX 20
  pin for ATX Power Supplies"
  http://www.ateck.com/viewItem.asp?idProduct=453056250

  CB-Z-Y2424-SPLITTER "ZIPPY Power Cable Splitter: One EPS12V 24 pin to
  Two EPS12V 24 pin for ATX EPS12V Power Supplies"
  http://www.ateck.com/viewItem.asp?idProduct=453056251

-- 
Andrew Piskorski <atp at piskorski.com>
http://www.piskorski.com/