[Beowulf] Software RAID?

[Vincent Diepeveen]

Wait a minute do i read that correctly,

"you have to rescan the scsi bus".

In short, first you spend really a lot of money to get SCSI
drives in order to then get confronted with all the software raid issues.

If you can afford SCSI drives, why not buy a great raidcard as well?

IMHO the interesting issue with raid is how to get a raid system where
you can hotswap and which supports both raid10 as well as different
types of raid5 (with one extra spare).

Speaking of that, how to save power with raid when it's not currently
streaming. is there hardware cards that let the drives idle when the raid
array hardly gets used for i/o?

I'm about to investigate how to cheap build a huge raid array (with
hotswap) for private purposes (chess EGTB generation and i guess i'll
require a TB or 4+ for that and raid10 as the write load is also very
high during generation).

What solutions are there?

Vincent

On Wed, 21 Nov 2007, Joe Landman wrote:

> Ekechi Nwokah wrote:
> > Hi,
> >
> > Does anyone know of any software RAID solutions that come close to the
> > performance of a commodity RAID card such as LSI/3ware/Areca for
> > direct-attached drives?
>
> For small numbers of drives, yes, the MD driver is superb with two
> (well, really three) caveats.
>
> First:  No hot swap.  You can do a kind-of-cold swap (have to take the
> mount offline, and can execute a few MD raw-disassemble, and then turn
> the device off, swap, force Linux to rescan the scsi bus, mark the drive
> as a hot spare, and force reassembly ... then remount).  This may or may
> not work, depending upon the linux driver for the SATA port.  Some get
> very unhappy if the drive goes away after it found it.
>
> Second (and third):  Context switches (and interrupts) tend to quickly
> swamp even fast systems with lots of processors.  This is because the
> SATA drivers on Linux, while good for basic SATA operations, may have a
> few issues with multiple CSW needed for each transfer.  You can drive a
> fast system to become slow with a simple RAID0 across two drives.  Run
> bonnie++ on it (not IOzone, unless you want to measure memory cache).
> Now imagine that system serving NFS requests.  Additionally, the
> interrupts driven by these hard IO operations also often drive the
> system performance into the ground.  We see 15-20k CSW and 20+k
> interrupts under heavy load for a simple two drive RAID0 serving NFS
> over gigabit.
>
> That is, it is not a bad idea, and it is possible to do it.  But be
> aware that you are going to need a fairly beefy machine (lots of RAM,
> lots of cores) to handle the buffering and the interrupts.  Can't help
> much on the CSW's, you will just have to pay that price.
>
> >
> > With the availability multi-core chips and SSE instruction sets, it
> > would seem to me that this is doable. Would be nice to not have to pay
> > for those RAID cards if I don't have to. Just wondering if anything
> > already exists.
>
> The extra you pay for those RAID cards buys you hot swap, and if you
> choose carefully, reasonable RAID engines.  They aren't perfect, their
> small random IO performance on large files leaves something to be
> desired (as do all RAID controllers from what I can see, unless you want
> to buy Bluearc or other units)
>
> If you do choose to go the MD route, check out which SATA drivers are
> well performing (low CSW/interrupts), and focus upon them.  There are a
> few out there.
>
> Joe
>
> >
> > Thanks,
> > Ekechi
> >
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web  : http://www.scalableinformatics.com
>         http://jackrabbit.scalableinformatics.com
> phone: +1 734 786 8423
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>