[Beowulf] hardware RAID versus mdadm versus LVM-striping

[Tony Travis]

Rahul Nabar wrote:
> If I have a option between doing Hardware RAID versus having software
> raid via mdadm is there a clear winner in terms of performance? Or is
> the answer only resolvable by actual testing? I have a fairly fast
> machine (Nehalem 2.26 GHz 8 cores) and 48 gigs of RAM.

Hello, Rahul.

It depends which level of RAID you want to use, and if you want hot-swap 
capability. I use inexpensive 3ware 8006-2 RAID1 controllers and stripe 
them using "md" software RAID0 to make RAID10 arrays. This gives me good 
performance and hot-swap capability (the production md RAID driver does 
not support hot-swap). However, where "md" really scores is portability. 
My RAID's can only be read by 3ware controllers - I made a considered 
descision about this: The 3ware controllers are well-supported by Linux 
kernels, but it makes me uneasy using a proprietary RAID format. I do 
also use "md" RAID5 which is more space efficient, but read this:

   http://www.baarf.com/

> Should I be using the vendor's hardware RAID or mdadm? In case a
> generic answer is not possible, what might be a good way to test the
> two options? Any other implications that I should be thinking about?

In fact, "mdadm" is just the user-space command for controlling the "md" 
driver. The problem with using an on-board RAID controller is that many 
of these are 'host' RAID (i.e. need a Windows driver to do the RAID) in 
which case you are using the CPU anyway, and they also use proprietary 
formats. Generally, I just use SATA mode on the on-board RAID controller 
and create an "md" RAID. This means that I can replace a motherboard 
withour worrying if it has the same type of RAID controller on-board.

> Finally, there;s always hybrid approaches. I could have several small
> RAID5's  at the hardware level (RIAD5 seems ok since I have smaller
> disks ~300 GB so not really in the domain where the RAID6 arguments
> kick in, I think) Then using LVM I can integrate storage while asking
> LVM to stripe across these RAID5's. Thus I'd get striping at two
> levels: LVM (software) and RAID5 (hardware).

Yes, I think a hybrid approach is good because that's what I use ;-)

However, I would avoid relying on LVM mirroring for data protection. It 
is much safer to stripe a set of RAID1's using LVM. I don't think LVM is 
useful unless you are managing a disk farm. The commonest issue in disk 
perfomance is decoupling seeks between different spindles, so I put the 
system files on a different RAID1-set to /export (or /home) filesystems.

HTH,

   Tony.
-- 
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt