Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

[Joe Landman]

Craig Tierney wrote:

> My concern wouldn't be the stability of the xfs filesystem.  We have 
> used it for almost 5 years now in the configuration discussed above.
> The filesystems weren't as large as 16TB (no more than 2TB), but
> that is so we could divide performance over several servers.
> 
> My concern with this setup isn't xfs, it would be the stability of
> the storage.  Also, if there is a disk hiccup  (which will happen) that
> repairing a 16 TB filesystem takes a long time.  A distributed 
> filesystem (PVFS2, Ibrix, etc) you would only have to fix the one 
> volume, not the entire filesystem.  There may be some filesystem 
> consistency checks after repair, but not to the extent of a full 
> filesystem check.

With AoE based Coraid or the iSCSI based IBrix and others, you localize 
hardware raid to a single disk tray.  Then you mirror trays.  Disk 
hiccups should not impact the fs.

The Coraid driver is FOSS, and we make RPMs of it available to our 
customers from our download site.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615