[Beowulf] OT: Software RAID & Multipath

[Guy Coates]

Chris Samuel wrote:
> This is about a storage node for a cluster, so it's partly on topic.. :-)
> 
> Through happy coincidence we now have a box with two FC cards going to a SAN 
> switch and thence into each side of an IBM FAStT 600 (doing H/W RAID5).  The 
> FAStT is partitioned into two 1.6TB lumps and each FC card can see both 
> controllers on the FAStT (for failover).


> Booting a live CD shows me that the multipath-tools package automatically 
> detects it has two paths and sets this up appropriately (very nice).
> 
> Now, if I wanted to stripe accesses to the FAStT down each controller I seem 
> tohave two options:

You should be able to do this entirely within the  dm-multipath layer. It can
deal with multiple controllers as well as multiple fabrics.

In dual controller/dual fabric setups, multipath should create  four paths for
each lun  (2 fabrics * 2 controllers = 4 paths).

However, not all dual controller arrays are active-active, some are active-passive.


If the controller really is active-active, then the dm-multipath should
round-robbin IO across all 4 paths (you can check that with iostat).

(Example multipath output from an HP EVA8000)

[size=3726 GB][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=4][enabled]
 \_ 1:0:2:1 sdab 65:176 [active][ready]          <--- Green Fabric, controller A
 \_ 0:0:2:1 sdaa 65:160 [active][ready]          <----Red fabric, controller A
 \_ 0:0:3:1 sdac 65:192 [active][ready]          <----Red fabric, controller B
 \_ 1:0:3:1 sdz  65:144 [active][ready]          <----Green fabric, controller B


If the controller is active-passive, then you will see 2 active paths, and 2
passive paths. (this example is from an HP EVA5000, where passive paths are
labelled "ghost." The output may be different for other models of controller)

[size=100 GB][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=2][enabled]
 \_ 0:0:0:2 sdb  8:16   [active][ready]    <---green fabric, controller A
 \_ 1:0:1:2 sdk  8:160  [active][ready]    <---red fabric, controller A
\_ round-robin 0 [prio=2][enabled]
 \_ 0:0:1:2 sde  8:64   [active][ghost]    <---green fabric, controller B
 \_ 1:0:0:2 sdh  8:112  [active][ghost]    <----red fabric, controller B


If you are seeing something different, you need to mess with the
path_grouping_policy and path_checker values in the multipath.conf file.  The
exact values depends on the exact model disk-controller you are using.  I don't
have any IBM storage, so can't help you with the exact values I'm afraid.

The people on the dm-devel list should be able to help you with the actual
values to use. http://sources.redhat.com/dm/


Cheers,

Guy

-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802