[Beowulf] hardware RAID versus mdadm versus LVM-striping

[Geoff Jacobs]

Tony Travis wrote:
> Rahul Nabar wrote:
>> If I have a option between doing Hardware RAID versus having software
>> raid via mdadm is there a clear winner in terms of performance? Or is
>> the answer only resolvable by actual testing? I have a fairly fast
>> machine (Nehalem 2.26 GHz 8 cores) and 48 gigs of RAM.
> 
> Hello, Rahul.
> 
> It depends which level of RAID you want to use, and if you want hot-swap
> capability. I use inexpensive 3ware 8006-2 RAID1 controllers and stripe
> them using "md" software RAID0 to make RAID10 arrays. This gives me good
> performance and hot-swap capability (the production md RAID driver does
> not support hot-swap). However, where "md" really scores is portability.
> My RAID's can only be read by 3ware controllers - I made a considered
> descision about this: The 3ware controllers are well-supported by Linux
> kernels, but it makes me uneasy using a proprietary RAID format. I do
> also use "md" RAID5 which is more space efficient, but read this:
> 
>   http://www.baarf.com/

Hot swap is at least partially dependent on the controller. Even most of
the built-in controllers now support hot swap. I'm not aware of md
hardcoding anything on boot which would prevent a change on the fly,
except that you would manually have to initiate a rebuild after swapping.

Could you be more specific about what wasn't working as far as hot
swapping? Was this with current controllers? Which ones?

>> Should I be using the vendor's hardware RAID or mdadm? In case a
>> generic answer is not possible, what might be a good way to test the
>> two options? Any other implications that I should be thinking about?
> 
> In fact, "mdadm" is just the user-space command for controlling the "md"
> driver. The problem with using an on-board RAID controller is that many
> of these are 'host' RAID (i.e. need a Windows driver to do the RAID) in
> which case you are using the CPU anyway, and they also use proprietary
> formats. Generally, I just use SATA mode on the on-board RAID controller
> and create an "md" RAID. This means that I can replace a motherboard
> withour worrying if it has the same type of RAID controller on-board.

Yes, it's pedigreed software instead of mystery meat firmware.

>> Finally, there;s always hybrid approaches. I could have several small
>> RAID5's  at the hardware level (RIAD5 seems ok since I have smaller
>> disks ~300 GB so not really in the domain where the RAID6 arguments
>> kick in, I think) Then using LVM I can integrate storage while asking
>> LVM to stripe across these RAID5's. Thus I'd get striping at two
>> levels: LVM (software) and RAID5 (hardware).
> 
> Yes, I think a hybrid approach is good because that's what I use ;-)
> 
> However, I would avoid relying on LVM mirroring for data protection. It
> is much safer to stripe a set of RAID1's using LVM. I don't think LVM is
> useful unless you are managing a disk farm. The commonest issue in disk
> perfomance is decoupling seeks between different spindles, so I put the
> system files on a different RAID1-set to /export (or /home) filesystems.

LVM is there as a management convenience. It allows you to grow your
disk pool more-or-less on demand. Where it's really beautiful, though,
is when you want to migrate data

-- 
Geoffrey D. Jacobs