[Beowulf] petabyte for $117k

[Bill Broadley]

Eugen Leitl wrote:
> On Tue, Sep 01, 2009 at 04:28:10PM -0700, Bill Broadley wrote:
> 
>> I'm very curious to hear how they are in production.  I've had vibration of
> 
> My thoughts exactly.

The lid screws down to apply pressure to a piece of foam.  Foam presses down
on 45 drives.  5 drives (1.4 lb each) sit on each port multipliers.  6 nylon
mounts support each multiplier supporting 7 pounds of drives.  Seems like the
damping from the nylon mounts would be minimal under that much pressure.  So
on the bottom of the case you have 63 pounds of drives, a significant fraction
of which is rotating mass.

I wonder if the port multipliers are really designed to actually support the
drives.  Doesn't seem like the SATA power/data connects I've seen are designed
for that.

Not to mention it's hard to imagine the typical flexible thin sheet metal lid
applying even pressure to 45 drives through the foam.  Seems like most of the
pressure would be on the drives on the outside edge, leaving the inside drives
relatively undamped.

My experience is that nylon mounts don't help much.  Sure poor manufacturing
tolerances, and near zero load/tension often prevent things like fans from
tightly coupling with a chassis.  But to decouple drive vibration from a
chassis seems to require something much more aggressive.  Something like a
very soft/gooey rubber with a fair bit of travel and give under minimal
pressure (4-6 ounces).

>> large sets of drives basically render the consumer drives useless.  Timeouts,
>> highly variable performance, drives constantly dropping out of raids.  It
>> became especially fun when the heavy I/O of a rebuild knocks additional drives
>> out of the array.
> 
> Also my experience down to a T.

Strangely their design works out to 0.11 per GB.  I tweaked their design to my
liking.  I upgraded to the $200 baracuda 2TB drive, 6GB of DDR3-1333 ECC
memory (from 4GB ddr2), ECC capable motherboard (with dual gigE), and a
Nehalem based xeon (ECC capable).

The result was $0.13 per GB.  Seems like a rather worthwhile investment from a
reliability perspective, let alone performance.

Anyone familiar with what the sun thumper does to minimize vibration?

>> I'd also worry that running the consumer drives well out of spec (both in duty
>> cycle and vibration) might significantly shorten their lives.
>>
>> Are the 7200.11 1.5TB seagate's particularly vibration resistant?
> 
> No. They're awful, as the entire 7200.11 line (I've had failures
> in 750 GByte, 1 TByte, 1.5 TByte, everywhere, even with reasonably
> small drive populations). 

A rather scary 26% of over 1500 reviews on newegg give that seagate 1 star out
of 5.  The consumer 1TB WD drive has 1100 reviews and 74% are 5 stars and only
8% are 1 star.