[Beowulf] Newbie Question: Racks versus boxes and good rack solutions for commodity hardware

[Robert G. Brown]

On Thu, 11 Dec 2008, arjuna wrote:

> I am in NewDelhi India. However I would prefer to put the cluster together
> myself, because

Ya, that's where I lived for seven years growing up.

> 1) I am a good python programmer and like programming and playing with
> computers
> 2) I will be using the cluster for animation (art + computers) and may have
> to bend it and tinker with it...therefore it makes sense for me to know it
> inside out.
> 3) If I set it up then I can grow it, and i envision it growing, outsourcing
> the whole thing would be expensive
> 4) I have been using linux for several years and am comfortable in the
> environment
> 5) I have a bunch of old computers lying about which are not so old and run
> basic versions of linux fast.

All excellent and traditional reasons, although you'll want to learn a
compiler, either C, C++ or Fortran.  Which one is most appropriate
depends a little bit on the application space you want to work in, a
little bit on your personality.  None are terribly like python.

> What is 1u?

A rack comes in "U"nits of height with prespecified/standard layouts for
screws and so on.  A "1U" rack chassis occupies 1.75" of vertical rack
space in a rack that is typically anywhere from 20U to 42U in height
(the latter is basically the height of a person; MUCH higher and you
start having difficult working on the upper slots and can interfere with
the ceiling or overhead cable trays, etc).  See:

   http://www.webopedia.com/TERM/1/1U.html

While we're on this question, remember Google Is Your Friend (GIYF).
These days, so are the various online references, especially e.g.
wikipedia.  So while we're always HAPPY to answer questions, you should
(as a good student:-) always try to answer them yourself first,
especially the easy ones.

> What is a blade system?

Here's another example. I google a second, pick the wikipedia article
and:

   http://en.wikipedia.org/wiki/Blade_server

complete with pictures.  The Google return also has dozens of links to
Tier 1 builders of bladed systems and vendors that resell them.  From
the latter you can actually look over specific blade servers and maybe
even get prices.

> I would be putting it in a room with air conditioning.

Sure, but ESPECIALLY in New Delhi, with post-monsoon summertime
temperatures in the 40C range outdoors (and with monsoon humidity AND
heat before that) you will need to take special care with your
environment.  The AC will needed to dehumidify and keep the room cooled
down to (ideally) 20C or lower all year long, summer and winter.

To help you estimate the cooling capacity:  AC is usually sold in
"tons".  1 ton of AC can remove 3500 joules of heat per second (3500
watts).  It needs some of this capacity to maintain a temperature
DIFFERENTIAL between inside and outside; a 20+ C differential will use
an easy 10% of the capacity, maybe more.  So you can look at your AC
unit and figure out how many systems you can put into the space before
it starts to get too warm -- most systems draw between 100 and 300 watts
loaded (sorry about the large variation, but there is everything from
single core UP to dual quad core out there with lots of combinations of
memory and accessory hardware).  If you have a half-ton of AC (say),
your body and the electric lights are probably 200W, heat infiltration
through the walls another 100W or more depending on where the room is,
so you can run as many as 10 systems or as few as three, depending.

Note that you'll pay for energy twice -- once for the power coming in,
again for the power used by the AC to remove it.

Oh, and New Delhi has one other unique-ish environmental constraint,
unless things have changed a lot since I lived there.  Post-monsoon,
when it dries out again you have dust storms.  I don't think most list
members can really imagine them, but I can (I used to climb a tree
outside of our house and feel the dust stinging my cheeks and erasing
the buildings all from sight).  You will need to be able to keep the
dust that infiltrates EVERYWHERE in the houses at that time out of the
computer room, as computers (especially the cooling fans) don't like
dust.  After a big one, you may need to shut down and vaccuum out the
insides of your systems.

> At this time I am trying to figure out the racks. Am meeting the hardware
> guy on Saturday and we were thinking of opening up the PCS i have lying
> around and taking measurements of how the mother boards fit into the
> cases,with the intention of creating a rack from scratch. Any ideas of what
> goes into a good rack in terms of size and matieral (assuming it has to be
> insulated)

Let's talk terminology.  What you are calling "a rack" we call
"shelving".  A rack is the thing described in the article up above -- a
completely standardized computer/telecom equipment holding arrangement.
When somebody talks about "rackmount equipment" they refer to stuff
boxed up to "slide into a rack" -- made a precise size and with screws
and/or rails in just the right places to accomplish this.

What you're talking about is a form of interesting homebrew cluster, I
think.

Periodically people talk about this sort of racking up of motherboards
in a homemade (cheap but still effective) way on list.  Search back
through the archives and you'll find some great discussions.  "Recipes"
that I can recall include:

   a) Mounting motherboards on cookie sheets and using a baking rack for
a cluster.

   b) Mounting motherboards on cookie sheets and using heavy duty steel
shelving with wooden shelves to make a sort of "vertically bladed"
cluster, sliding the cookie sheets in and out of slots cut into the
wood.

   c) Clusters built into standard file cabinets.

and several others.  In the discussions were some suggestions concerning
safety (fire and otherwise) and electomagnetic isolation and noise.
Links to pictures, as well, let's see:

   http://www.beowulf.org/archive/2006-March/015209.html

and as you can see, nearly everything is in the beowulf archives
somewhere if you search for it cleverly.  I think Andrew is still around
and may be listening in case the links have been moved in the meantime.

> Also again, what might be some upto date books on the subject and any
> experiences regarding the actual creation of the rack and the physical
> hardware.

People don't build racks.  People buy racks.  However, if you have a
machine shop and access to steel and know how to bend and tap it and
weld it, you could probably, from the link up above and perhaps some
more stuff like it gleaned from the web, build a simple four poster that
would "work" to hold standard rackmount chassis.

Heck, even building rackmount cases has been discussed on list.  Sheet
aluminum or steel, cut to spec, fold and weld, and so on.  Here it isn't
worth the time any more -- rackmount boxes and racks aren't THAT
expensive compared to the time needed to DIY -- but I suppose it is
possible.

> I am starting with 3 nodes to be expanded to n nodes....The 3 nodes will
> allow me to keep complexity down while learning and then i can expand to n
> nodes once i have it down to increase speed.

Sure.  Good plan.  Get yourself an 8 port (or better) gigabit ethernet
switch to use as your first network, too.

> Am planning to run animation software (like blender) on it. Since animation
> software requires large processing power i am assuming they have already
> worked on parrallelizing the code...

Assume nothing, unfortunately.  However, even if they haven't, if you
can partition up the tasks and just run it N times in a batch mode on N
systems, that's pretty good parallel speed up right there, and likely
doable for a task that is basically embarrassingly parallel.

> Anyone using clusters for animation on this list?

Don't know.  I doubt it.  Not QUITE HPC, although I do know physicists
who have e.g. animated simulations and so on on clusters.  However, the
animation itself wasn't done in parallel, only the generation of data to
animate.

    rgb

> 
>  
>        
> Two pieces of advice
>  
> a) let us know where you are physically. Talk to a clustering company
> in your country, or area.
> You will be surprised - they will put the whole thing together for you
> as a 'turnkey' cluster AND what's more important support it. OK, you
> don't get the learning experience which you are after.
>  
> b) if this thing is to sit in your office, think about noise, cooling
> and how many amps you can draw from a wall socket.
> 1U servers have lots of little high speed fans and the noise gets
> very, very annoying.
> Think of putting this thing in a separate room, with some air
> conditioning. Even a small room with a portable wheeled unit, venting
> to the outside may be adequate for you.
>  
>  
> Have you thought about a blade system for your particular situation?
> Might be the ideal solution.
>  
>  
>  
>  
> 
> 
> 
> 
> --
> Best regards,
> arjuna
> http://www.brahmaforces.com
> 
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu