Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

[Craig Tierney]

Warren Turkal wrote:
> On Thursday 04 May 2006 12:16, Craig Tierney wrote:
>> Why FOSS (not to start a flame war)?  What if AcmeFS was reasonably
>> priced and did what you needed it to do?
> 
> Well, the one issue here is future upgradability. If you tie yourself to some 
> version of a proprietary product, (a) they may not release versions that are 
> compatible with whatever distro you might prefer or (b) they may not make 
> upgrades available without additional costs for when you upgrade your system. 
> There is also a doomsday scenario if the vendor stopped shipping the product. 
> Also, they obviously won't be shipping with the kernel, which is where the 
> best quality drivers tend to be shipped.
> 
> wt

I agree with all of this.  It is a challenge to work with vendors with
closed source code.  I don't let vendors put specifics on my kernel or
system, except when it comes to filesystems.  When filesystems 'just 
work' and I have choice of products then I can start demanding more 
flexibility from the kernel support.  At this point, most all vendors
have been flexible with me and my kernel versions.  Servers aren't 
generally a problem (Terrascale, Ibrix, StorNext) because I have been
able to treat them as black-box systems as they serve that one purpose.
For clients, they all have been very flexible.  Yes, I can't change my
kernel every day, but I wouldn't do that on production systems anyway.

(Follow-up: Now Terrascale sells their servers with the hardware so
that isn't an issue any more).

 From your reasons, it all comes down to risk.  What is the risk
of the issues that you brought up happening?  What is the risk of
finding bugs in your FOSS solution and how do you risk plan to insure
those get fixed (ie. commercial support)?  What about the doomsday 
scenario where your FOSS filesystem is no longer supported?  Yes you
can fix it yourself but filesystems are hard and it is going to take
a long time to ramp up to be able to support that code.  What 
reliability numbers to do you have to guarantee for your customer or 
your users? What is the risk of losing or corrupting data and your users 
getting angry and pounding on your door carrying torches and heavy metal 
objects?

I like PVFS2.  It is a good solution if it fits your needs.  It was 
designed for parallel IO and may not be the right fit for your 
applications.  GFS has been around long enough that I assume it mostly 
works, but it is good for certain uses. However, it was designed for 
parallel IO and may not be the right fit for you.

Craig