[Beowulf] Software RAID?

[Joe Landman]

Ekechi Nwokah wrote:
> Hi,
> 
> Does anyone know of any software RAID solutions that come close to the
> performance of a commodity RAID card such as LSI/3ware/Areca for
> direct-attached drives?

For small numbers of drives, yes, the MD driver is superb with two 
(well, really three) caveats.

First:  No hot swap.  You can do a kind-of-cold swap (have to take the 
mount offline, and can execute a few MD raw-disassemble, and then turn 
the device off, swap, force Linux to rescan the scsi bus, mark the drive 
as a hot spare, and force reassembly ... then remount).  This may or may 
not work, depending upon the linux driver for the SATA port.  Some get 
very unhappy if the drive goes away after it found it.

Second (and third):  Context switches (and interrupts) tend to quickly 
swamp even fast systems with lots of processors.  This is because the 
SATA drivers on Linux, while good for basic SATA operations, may have a 
few issues with multiple CSW needed for each transfer.  You can drive a 
fast system to become slow with a simple RAID0 across two drives.  Run 
bonnie++ on it (not IOzone, unless you want to measure memory cache). 
Now imagine that system serving NFS requests.  Additionally, the 
interrupts driven by these hard IO operations also often drive the 
system performance into the ground.  We see 15-20k CSW and 20+k 
interrupts under heavy load for a simple two drive RAID0 serving NFS 
over gigabit.

That is, it is not a bad idea, and it is possible to do it.  But be 
aware that you are going to need a fairly beefy machine (lots of RAM, 
lots of cores) to handle the buffering and the interrupts.  Can't help 
much on the CSW's, you will just have to pay that price.

> 
> With the availability multi-core chips and SSE instruction sets, it
> would seem to me that this is doable. Would be nice to not have to pay
> for those RAID cards if I don't have to. Just wondering if anything
> already exists.

The extra you pay for those RAID cards buys you hot swap, and if you 
choose carefully, reasonable RAID engines.  They aren't perfect, their 
small random IO performance on large files leaves something to be 
desired (as do all RAID controllers from what I can see, unless you want 
to buy Bluearc or other units)

If you do choose to go the MD route, check out which SATA drivers are 
well performing (low CSW/interrupts), and focus upon them.  There are a 
few out there.

Joe

> 
> Thanks,
> Ekechi
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615