[Beowulf] Software Raid

[Michael T. Prinkey]

Honestly, I am wondering if the Software/Hardware RAID argument has
devolved to the state of SCSI versus ATA or (heaven forfend) emacs versus
vi.  8)

My experiences with hardware raid have been consistently lack luster over
7 years and several generations of hardware.  My experiences with software
RAID servers (specifically built for the task) have been largely positive.  
When I read comments extolling the virtues of hardware RAID solutions, I
find myself constantly wondering if I could be missing something after
some many years and many dozens of deployed units.

To provide numbers, I am really only concerned if the raid array can
saturate the gigabit line feeding it.  On-server performance is pretty
useless as no work is ever done directly on the RAID servers.  For reading
data, the server could certainly saturate gigabit...Bonnie on the NFS
mount gave roughly 85 MB/sec for software RAID5.  When we deployed an
8-drive RAID5 array using hardware RAID on the SATA 3ware card,
performance was on the order of 15 MB/sec.  We had initially deployed
these raid servers using the hardware RAID5 setup, but we had several user
complaints about poor storage performance.  So we retooled with software
RAID and the rest is history.

Clearly, YMMV.

Mike

On Tue, 13 Dec 2005, Joe Landman wrote:

> 
> 
> Vincent Diepeveen wrote:
> 
> >>The remaining advantage of hardware is still hot-swapping
> >>failed drives without having to shutdown the server.
> > 
> > 
> > Those same nerds of above, they do not take into account that if 
> > something complex like a raid array gets suddenly handled in 
> > software instead of hardware, that even the tiniest 
> > undiscovered bug in a file system, will impact you.
> 
> <scratch />
> <scratch />
> <huh? />
> 
> As the raid device is being created at the block device level, and the 
> file system resides above this, a file system bug will be just as 
> detrimental to a hardware raid system as it would a software raid system.
> 
> Of course, you could have meant a bug in the software raid block device 
> driver.  Yes such things do exist.  So do bugs in the hardware raid 
> controllers.  In *neither* case do you want to touch the buggy code. 
> Best case is completely innocuous behavior.  Worst case, well, lets not 
> get into that.
> 
> Bugs can and do occur in any software.  Whether burned into firmware, 
> written as VHDL/Verilog that creates the ASICs or FPGAs on the hardware 
> raid, or in the software raid block device.
> 
> [...]
> 
> > And be sure that there is bugs. So doing a hardware XOR (or whatever) in
> > RAM of the raid controller instead of in the software, is a huge advantage.
> 
> RAID is *far more* than doing hardware XOR.  Most XOR implementations 
> tend to be bug free given how atomic this operation is.  The code around 
> it however occasionally has bugs.  Firmware and software code.
> 
> > It reduces complexity of what software has to do, so it reduces the
> > chance that a bug will occur in the OS somewhere, causing you to lose
> > all your files.
> 
> No.  Absolutely not.  Software raid simply does in software what the 
> hardware may do in part in hardware.  Any bug, anywhere in this process 
> (in either HW or SW raid) and you can have problems.  Problems and bugs 
> are not just the provenance of SW raid.
> 
> 
>