[Beowulf] storage server hardware considerations

[Mark Hahn]

>> Rather than looking at the solution, could you describe the problem you
>> need to solve?
>
> I've been building cheap, big, slow storage servers for our use. However, 
> some of our users are starting to need higher performance IO. We still don't 
> have a lot of money, but I'd like to provide something with a modest amount 
> of storage (at least 500GB) that can at least handle full GigE, perhaps up to 
> DDR Infiniband for future expansion, without breaking the bank.

if you can't saturate gigabit with very modest raids,
you're doing something wrong.  a single ultra-cheap disk
these days (seagate 7200.12 500G, $60 or so) will hit 
close to 135 MB/s on outer tracks and average > 105 over
the whole disk.  I'm guessing that you're losing performance
due to either bad controllers (avoid HW raid on anything that's 
not fairly recent) or a combination of raid6 and a write-heavy workload...

bandwidth is easy; latency is hard.  the only time I'd consider flash
for storage is if the workload was astonishingly write-dominated.  or 
for a pretty large cluster (for instance, the metadata write load for 
a a few thousand jobs can _flatten_ an indifferently configured Lustre 
system, even though the bandwidth is fairly low.