[Beowulf] Any recommendations for a good JBOD?

[Joe Landman]

Alex Chekholko wrote:

>> Thanks for the pointers! I had never heard of AoE before!
> 
> This is all well and good until you compare the prices of the respective solutions.
> 
> E.g. what's the cheapest 5TB (usable) AoE box you can buy?

I believe somewhat more than a relatively fast iSCSI/SRP/NFS/CIFS box 
with 6.75TB usable (but we are biased).

AoE hasn't really found a niche for many reasons.  Not the least of 
which is the paucity of quality software target implementations, a 
single vendor hardware supplier, and the significant resource consuming 
initiator.  We have lots of experience setting up AoE systems for users 
and customers, lots of experience fixing systems, and altering designs 
so that the AoE initiators don't bring down head nodes, login nodes, etc 
that they are attached to.  It is best, when building an AoE system 
design, to isolate the unit mounting the AoE targets from important 
services that have to be up.

Whether or not the AoE protocol is superior to iSCSI is moot.  The AoE 
initiator for windows isn't terribly stable.  Last I checked there was 
something for Solaris though I don't know the state.  iSCSI is 
everywhere, it is available on pretty much all platforms ... it has 
achieved ubiquity.  Quality targets and initiators exist as software 
stacks you can use.  They interoperate reasonably well, though the 
Windows 2.08 stack doesn't seem to follow the standard terribly well in 
terms of reconnecting to a single target.  These minor nits aside, it 
works, reasonably well, and without significant pain.

This aside, both AoE and iSCSI provide block device services.  Both 
systems can present a block device with a RAID backing store.  Patrick 
and others will talk about the beauty of the standards, but this is 
unfortunately irrelevant in the market.  The market isn't a meritocracy.

Actually, with the advent of USB3 and related devices, I'd expect AoE 
and lower end raid to be effectively completely subsumed by this.  USB3 
has ample bandwidth to connect a low end RAID unit, more than GbE.  The 
comments on the Atom based micro itx MBs are quite relevant there. 
Especially if you can get one with multiple SATA and USB3 ports 
(especially if they can be USB3 targets).

If you want the lowest end RAID right now, get an eSATA device and 
enclosure (not cheap but doable).  It will work, albeit not as fast as 
you might like.  Use mdadm, and be done with it.

Remember, if you want to use mdadm, you do need block devices to work 
with.  Whether these block devices are gigabit, USB, or eSATA attached 
is irrelevant.  Better/more expensive systems will put the RAID on the 
unit and export you either a block device, a file system, or both.  Even 
better systems will give you thin provisioning, snapshotting, and all 
these other very nice features.  You make your choices and pay your money.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615