[Beowulf] hardware RAID versus mdadm versus LVM-striping

[Bill Broadley]

Rahul Nabar wrote:
> If I have a option between doing Hardware RAID versus having software
> raid via mdadm is there a clear winner in terms of performance?

No.

> Or is
> the answer only resolvable by actual testing? I have a fairly fast
> machine (Nehalem 2.26 GHz 8 cores) and 48 gigs of RAM.
> 
> Should I be using the vendor's hardware RAID or mdadm? 

Depends.  Are you performance limited?  If not I'd go with the one that makes
the admin happier.  I prefer software raid because I'm familiar with the
interface, monitoring, and recovery.  Not to mention the flexibility of
setting RAID per partition and the ability to easily move RAIDs between
machines is valuable to me.  With hardware RAID I'd want a spare controller
around in case of failure.

Be warned that various hardware RAID companies seem to be making it harder to
do software RAID.  I've seen controllers do nasty things like not boot with 16
JBOD devices.

> In case a
> generic answer is not possible, what might be a good way to test the
> two options?

Ideally you would use your performance limited application as a benchmark.
Sure there are plenty of micro benchmarks to help.  Things like bonnie++,
iozone, or postmark for a variety of random and sequential workloads.
Postmark in particular can simulate a wide range of workloads.  The ultimate
benchmark is your particular usage.

> Any other implications that I should be thinking about?

Learning GUI, bugs, interface, of hardware requires time and energy.  I'd
look at the failure modes you are trying protect against.  As far as
reliability go I like for smartctl, lmsensors, RAID scrubbing and the like to
work to I can keep an eye on the disks health.

> Finally, there;s always hybrid approaches. I could have several small
> RAID5's  at the hardware level (RIAD5 seems ok since I have smaller
> disks ~300 GB so not really in the domain where the RAID6 arguments
> kick in, I think) Then using LVM I can integrate storage while asking
> LVM to stripe across these RAID5's. Thus I'd get striping at two
> levels: LVM (software) and RAID5 (hardware).

For a given level of reliability size does effect the maximum number of disks
in a RAID.  Double disk failures do happen.  I'd hesitate to spread a file
system across multiple RAIDs just because of recovery performance issues.
After all a file system spread across multiple RAIDs effectively reduces the
entire group of disks to a single head.  For 16 disks I often use three 5 disk
RAID5s and one global spare.  If I run 3 workloads, one each per 5 disks I get
dramatically better performance than 3 workloads running on 15 disks.  So most
issues (slow performance during backups, failures, disk full, migration, etc)
only effects 5 disks at a time and could even be migrated to a different
machines.