[Beowulf] hardware RAID versus mdadm versus LVM-striping

[Joe Landman]

Rahul Nabar wrote:
> If I have a option between doing Hardware RAID versus having software
> raid via mdadm is there a clear winner in terms of performance? Or is

Depends upon workload, writes vs reads, streaming vs random IO, number 
of simultaneous readers/writers.  There is no real clear answer.

> the answer only resolvable by actual testing? I have a fairly fast
> machine (Nehalem 2.26 GHz 8 cores) and 48 gigs of RAM.

Testing is a good thing.  Sadly, too many people test *after* they 
purchased something (only to discover what is meant by the term 
"marketing benchmark numbers").


> Should I be using the vendor's hardware RAID or mdadm? In case a

Ohhh ... it depends.  Some of the "vendors" hardware raid ... heck ... 
most of it ... is rebadged LSI gear.  Usually their lower end stuff 
which is sometimes fake-raid.  Use fake-raid only if no other options 
exist.

More in a moment.

> generic answer is not possible, what might be a good way to test the
> two options? Any other implications that I should be thinking about?

Benchmark your load using a load generator like fio.

> 
> Finally, there;s always hybrid approaches. I could have several small
> RAID5's  at the hardware level (RIAD5 seems ok since I have smaller
> disks ~300 GB so not really in the domain where the RAID6 arguments
> kick in, I think) Then using LVM I can integrate storage while asking

RAID6 kicks in purely from the second correlated disk failure scenario. 
  This is size independent.  It happens, and you need to be prepared.

> LVM to stripe across these RAID5's. Thus I'd get striping at two
> levels: LVM (software) and RAID5 (hardware).

LVM is not a performance tool.  Use it to help you manage things, not 
speed things.

Our own testing puts our 24 bay DV4 unit at a bit more than 1GB/s 
sustained read (large block sequential) in RAID6, with writes in the 
400-500 MB/s region (large block sequential).  This is MD RAID based. 
Our "equivalent" JR4 system clocks in at nearly double the read speed, 
and about 3+ x the write speed.  This is a hardware RAID system.

Your mileage will vary ... tremendously ... as a function of your IO 
pattern.

My own suggestion is to test before you buy.  After you buy, well, its a 
bit harder to change your mind.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615