<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: high physical density cluster design - power/heat/rf questions</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>I took a look at the website. These boards are full size PC boards, and might not perform well with the compact space, due to the problems you've outlined. </FONT></P>
<P><FONT SIZE=2>But, on the other hand, FreeBSD should work fine on these. I'm using FreeBSD for clustering right now, and the operating system is pretty stable.</FONT></P>
<P><FONT SIZE=2>Check out <A HREF="http://www.emjembedded.com/products/products.html" TARGET="_blank">http://www.emjembedded.com/products/products.html</A> for single board computers that may give you much more of a dense setup than with these boards.</FONT></P>
<P><FONT SIZE=2>Richard Schilling</FONT>
<BR><FONT SIZE=2>Mount Vernon, WA</FONT>
</P>
<BR>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Velocet [<A HREF="mailto:mathboy@velocet.ca">mailto:mathboy@velocet.ca</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Monday, March 05, 2001 9:36 PM</FONT>
<BR><FONT SIZE=2>> To: beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>> Subject: high physical density cluster design - power/heat/rf </FONT>
<BR><FONT SIZE=2>> questions</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I have some questions about a cluster we're designing. We really need</FONT>
<BR><FONT SIZE=2>> a relatively high density configuration here, in terms of floor space.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> To be able to do this I have found out pricing on some socket </FONT>
<BR><FONT SIZE=2>> A boards with</FONT>
<BR><FONT SIZE=2>> onboard NICs and video (dont need video though). We arent </FONT>
<BR><FONT SIZE=2>> doing anything</FONT>
<BR><FONT SIZE=2>> massively parallel right now (just running </FONT>
<BR><FONT SIZE=2>> Gaussian/Jaguar/MPQC calculations)</FONT>
<BR><FONT SIZE=2>> so we dont need major bandwidth.* We're booting with root </FONT>
<BR><FONT SIZE=2>> filesystem over</FONT>
<BR><FONT SIZE=2>> NFS on these boards. Havent decided on FreeBSD or Linux yet. </FONT>
<BR><FONT SIZE=2>> (This email</FONT>
<BR><FONT SIZE=2>> isnt about software config, but feel free to ask questions).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> (* even with NFS disk we're looking at using MFS on freebsd </FONT>
<BR><FONT SIZE=2>> (or possibly</FONT>
<BR><FONT SIZE=2>> the new md system) or the new nbd on linux or equivalent for </FONT>
<BR><FONT SIZE=2>> gaussian's</FONT>
<BR><FONT SIZE=2>> scratch files - oodles faster than disk, and in our case, with no</FONT>
<BR><FONT SIZE=2>> disk, it writes across the network only when required. Various tricks</FONT>
<BR><FONT SIZE=2>> we can do here.)</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The boards we're using are PC Chip M810 boards </FONT>
<BR><FONT SIZE=2>> (www.pcchips.com). Linux seems</FONT>
<BR><FONT SIZE=2>> fine with the NIC on board (SiS chip of some kind - Ben </FONT>
<BR><FONT SIZE=2>> LaHaise of redhat is</FONT>
<BR><FONT SIZE=2>> working with me on some of the design and has been testing it </FONT>
<BR><FONT SIZE=2>> for Linux, I</FONT>
<BR><FONT SIZE=2>> have yet to play with freebsd on it).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The configuration we're looking at to achieve high physical density is</FONT>
<BR><FONT SIZE=2>> something like this:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> NIC and Video connectors</FONT>
<BR><FONT SIZE=2>> /</FONT>
<BR><FONT SIZE=2>> ------------=-------------- board upside down</FONT>
<BR><FONT SIZE=2>> | cpu | = | RAM |</FONT>
<BR><FONT SIZE=2>> |-----| |_________|</FONT>
<BR><FONT SIZE=2>> |hsync|</FONT>
<BR><FONT SIZE=2>> | | --fan--</FONT>
<BR><FONT SIZE=2>> --fan-- | | </FONT>
<BR><FONT SIZE=2>> _________ |hsync|</FONT>
<BR><FONT SIZE=2>> | | |-----|</FONT>
<BR><FONT SIZE=2>> | RAM | = | cpu |</FONT>
<BR><FONT SIZE=2>> -------------=------------- board right side up</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> as you can see the boards kind of mesh together to take up </FONT>
<BR><FONT SIZE=2>> less space. At</FONT>
<BR><FONT SIZE=2>> micro ATX factor (9.25" I think per side) and about 2.5 or 3" </FONT>
<BR><FONT SIZE=2>> high for the</FONT>
<BR><FONT SIZE=2>> CPU+Sync+fan (tallest) and 1" tall for the ram or less, I can </FONT>
<BR><FONT SIZE=2>> stack two of</FONT>
<BR><FONT SIZE=2>> these into 7" (4U). At 9.25" per side, 2 wide inside a </FONT>
<BR><FONT SIZE=2>> cabinet gives me 4</FONT>
<BR><FONT SIZE=2>> boards per 4U in a standard 24" rack footprint. If I go 2 </FONT>
<BR><FONT SIZE=2>> deep as well (ie 2x2</FONT>
<BR><FONT SIZE=2>> config), then for every 4U I can get 16 boards in.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The cost for this is amazing, some $405 CDN right now for </FONT>
<BR><FONT SIZE=2>> Duron 800s with</FONT>
<BR><FONT SIZE=2>> 128Mb of RAM each without the power supply (see below; </FONT>
<BR><FONT SIZE=2>> standard ATX power is</FONT>
<BR><FONT SIZE=2>> $30 CDN/machine). For $30000 you can get a large ass-load of </FONT>
<BR><FONT SIZE=2>> machines ;)</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Obviously this is pretty ambitious. I heard talk of some people doing</FONT>
<BR><FONT SIZE=2>> something like this, with the same physical confirguration and cabinet</FONT>
<BR><FONT SIZE=2>> construction, on the list. Wondering what your experiences have been.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Problem 1</FONT>
<BR><FONT SIZE=2>> """""""""</FONT>
<BR><FONT SIZE=2>> The problem is in the diagram above, the upside down board </FONT>
<BR><FONT SIZE=2>> has another board</FONT>
<BR><FONT SIZE=2>> .5" above it - are these two boards going to leak RF like mad </FONT>
<BR><FONT SIZE=2>> and interefere</FONT>
<BR><FONT SIZE=2>> with eachothers' operations? I assume there's not much to do </FONT>
<BR><FONT SIZE=2>> there but to put</FONT>
<BR><FONT SIZE=2>> a layer of grounded (to the cabinet) metal in between. This </FONT>
<BR><FONT SIZE=2>> will drive up the</FONT>
<BR><FONT SIZE=2>> cabinet construction costs. I'd rather avoid this if possible.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Our original construction was going to be copper pipe and </FONT>
<BR><FONT SIZE=2>> plexiglass sheeting,</FONT>
<BR><FONT SIZE=2>> but we're not sure that this will be viable for something </FONT>
<BR><FONT SIZE=2>> that could be rather</FONT>
<BR><FONT SIZE=2>> tall in our future revisions of our model. Then again, copper </FONT>
<BR><FONT SIZE=2>> pipe can be</FONT>
<BR><FONT SIZE=2>> bolted to our (cement) ceiling and floor for support.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> For a small model that Ben LaHaise built, check the pix at</FONT>
<BR><FONT SIZE=2>> <A HREF="http://trooper.velocet.ca/~mathboy/giocomms/images" TARGET="_blank">http://trooper.velocet.ca/~mathboy/giocomms/images</A></FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Its quick a hack, try not to laugh. It does engender the 'do </FONT>
<BR><FONT SIZE=2>> it damn cheap'</FONT>
<BR><FONT SIZE=2>> mentality we're operating with here.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The boards are designed to slide out the front once the power </FONT>
<BR><FONT SIZE=2>> and network</FONT>
<BR><FONT SIZE=2>> are disconnected.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> An alternate construction we're considering is sheet metal cutting and</FONT>
<BR><FONT SIZE=2>> folding, but at much higher cost.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Problem 2 - Heat Dissipation</FONT>
<BR><FONT SIZE=2>> """""""""""""""""""""""""""" </FONT>
<BR><FONT SIZE=2>> The other problem we're going to have is heat. We're going to </FONT>
<BR><FONT SIZE=2>> need to build</FONT>
<BR><FONT SIZE=2>> our cabinet such that its relatively sealed, except at front, </FONT>
<BR><FONT SIZE=2>> so we can get</FONT>
<BR><FONT SIZE=2>> some coherent airflow in between boards. I am thinking we're </FONT>
<BR><FONT SIZE=2>> going to need to</FONT>
<BR><FONT SIZE=2>> mount extra fans on the back (this is going to make the 2x2 </FONT>
<BR><FONT SIZE=2>> design a bit more</FONT>
<BR><FONT SIZE=2>> tricky, but at only 64 odd machines we can go with 2x1 config </FONT>
<BR><FONT SIZE=2>> instead, 2</FONT>
<BR><FONT SIZE=2>> stacks of 32, just 16U high). I dont know what you can </FONT>
<BR><FONT SIZE=2>> suggest here, its all</FONT>
<BR><FONT SIZE=2>> going to depend on physical configuration. The machine is </FONT>
<BR><FONT SIZE=2>> housed in a proper</FONT>
<BR><FONT SIZE=2>> environment (Datavaults.com's facilities, where I work :) </FONT>
<BR><FONT SIZE=2>> thats climate</FONT>
<BR><FONT SIZE=2>> controlled, but the inside of the cabinet will still need </FONT>
<BR><FONT SIZE=2>> massive airflow,</FONT>
<BR><FONT SIZE=2>> even with the room at 68F.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Problem 3 - Power</FONT>
<BR><FONT SIZE=2>> """""""""""""""""</FONT>
<BR><FONT SIZE=2>> The power density here is going to be high. I need to mount </FONT>
<BR><FONT SIZE=2>> 64 power supplies</FONT>
<BR><FONT SIZE=2>> in close proximity to the boards, another reason I might need </FONT>
<BR><FONT SIZE=2>> to maintain</FONT>
<BR><FONT SIZE=2>> the 2x1 instead of 2x2 design. (2x1 allows easier access too). </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> We dont really wanna pull that many power outlets into the </FONT>
<BR><FONT SIZE=2>> room - I dont know</FONT>
<BR><FONT SIZE=2>> what a diskless Duron800 board with 256Mb or 512Mb ram will </FONT>
<BR><FONT SIZE=2>> use, though I</FONT>
<BR><FONT SIZE=2>> guess around .75 to 1 A. Im gonna need 3 or 4 full circuits </FONT>
<BR><FONT SIZE=2>> in the room (not</FONT>
<BR><FONT SIZE=2>> too bad actually). However thats alot of weight on the </FONT>
<BR><FONT SIZE=2>> cabinet to hold 60 odd</FONT>
<BR><FONT SIZE=2>> power supplies, not to mention the weight of the cables </FONT>
<BR><FONT SIZE=2>> themselves weighing</FONT>
<BR><FONT SIZE=2>> down on it, and a huge mess of them to boot.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I am wondering if someone has a reliable way of wiring </FONT>
<BR><FONT SIZE=2>> together multiple</FONT>
<BR><FONT SIZE=2>> boards per power supply? Whats the max density per supply? Can we</FONT>
<BR><FONT SIZE=2>> go with redundant power supplies, like N+1? We dont need that much</FONT>
<BR><FONT SIZE=2>> reliability (jobs are short, run on one machine and can be restarted</FONT>
<BR><FONT SIZE=2>> elsewhere), but I am really looking for something thats going to</FONT>
<BR><FONT SIZE=2>> reduce the cabling.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> As well, I am hoping there is some economy of power converted here -</FONT>
<BR><FONT SIZE=2>> a big supply will hopefully convert power for multiple boards more</FONT>
<BR><FONT SIZE=2>> efficiently than a single supply per board. However, as always, the</FONT>
<BR><FONT SIZE=2>> main concern is cost.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Any help or ideas is appreciated.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> /kc</FONT>
<BR><FONT SIZE=2>> -- </FONT>
<BR><FONT SIZE=2>> Ken Chase, math@velocet.ca * Velocet Communications Inc. * </FONT>
<BR><FONT SIZE=2>> Toronto, CANADA </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> _______________________________________________</FONT>
<BR><FONT SIZE=2>> Beowulf mailing list, Beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>> To change your subscription (digest mode or unsubscribe) </FONT>
<BR><FONT SIZE=2>> visit <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf" TARGET="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</A></FONT>
<BR><FONT SIZE=2>> </FONT>
</P>
</BODY>
</HTML>