Racks vs. pile of PCs

[Ben DeLong]

David,

Our experience was that we did get a lower ratio when getting a cluster
with more expensive guts.  In our case we got a 32 node dual athlon
system with Myrinet as the interconnect and 2 gig of RAM per node.  We
spec'd this system in both a rack (4U cases) and pile-o-PCs
configuration and the cost difference was about 10% (~$12k on ~$118k for
pile-o-PCs).  

We ended up going with the rack solution for a number of reasons
(aesthetics being among them).  Looking back on it, I believe I would
make the same decision again.  We spent less time putting the rack
together than we would have setting up the pile-o-PCs and we have a good
solid configuration with easy access to the nodes (They slide out of the
rack and open from the top - all without having to unhook any cables!) 
Cabling issues were also clean as we could easily tie them off to the
sides of the rack.  

All of this added up to less labor hours used in the setup and
maintenance of the cluster, which offset some of the extra 10% cost. 
This, plus the aesthetics and durability issues, made the racks a better
solution in my opinion.

Of the benefits you brought up with the pile-o-PCs approach, only number
4 applied to our setup as we went with 4U cases.  These cases were as
bigger than a midsize tower and just as versatile.

Ben

P.S.  We did have any space issues with this cluster.  We had more space
than we needed.  :)

> Racks sure look nice and there is no question that they
> are space efficient, but I'm really starting to wonder if
> they are such a great idea for a smallish cluster (<=20 nodes)
> in those situations where there is enough space for a
> classic pile of PCs.   I mean, what other advantages do they
> have besides those two to offset their many disadvantages?
> 
> Racks better than piles:
> 
> 1.  Space efficiency.
> 2.  Aesthetics (racks look cool)
> 
> Piles better than racks (these are not orthogonal):
> 
> 1.  Internal space constraints
> 2.  CPU/motherboard Cooling.  This follows from [1].
> 3.  Motherboard/CPU options.  This follows from [1]. 
>     With a few exceptions most motherboard/CPU combinations
>     will fit into a standard ATX case -  good luck getting
>     a 2.4 Ghz P4 into a 1U.
> 4.  Initial purchase price for equivalent performance.
> 5.  Maintenance costs (rack parts tend to be nonstandard
>     and expensive to replace, for instance, 1U power supplies).
> 
> Other factors?
> 
> I estimate that for a small cluster (<1 rack's worth of equipment)
with
> node guts (mobo,CPU,disk,ram) costing <= $1200 the racked version
> will cost at least 20-30% more than the piled version.  So if a piled
> 20 node cluster costs $24000, the equivalent racked version will
> be at least $30000.   $6000 seems a lot to pay for no extra
performance.
> If the "guts" were much more expensive the additional rack costs
would,
> in theory be a lower percentage.  In practice, it is my impression
that
> the ratio is no lower because the vendors charge even more for the 
> racked versions of high performance nodes.
> 
> Comments? 
> 
> Regards,
> 
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech