[Beowulf] and the winner of "Best 2007 parallel file system" is...

[Robert Latham]

On Fri, Nov 23, 2007 at 06:41:52PM +0000, andrew holway wrote:
> Scores for redundancy, reliability and value. Nice features etc. Id
> like to build a decent comparison of all the options available cos
> honestly, I don't have a clue :) and all the data online seems a
> little stale.

Do you have infinite money?  Well, that might mean you pick Foo-FS.
Are you primarily worried about parallel applications making heavy use
of MPI-IO?  Bar-FS is your winner. Lots of small files?  Uncordinated
access?  go Baz-FS. 

The reason the online data is a little stale is that the interesting
systems with lots of disk and large number of compute nodes have the
pesky little problem of being used for production.  The admins don't
take too kindly to turing over the system for a week or more of I/O
benchmarking (and that's what it would take -- at least -- what with
updating kernel and OS images for all the various options out there) 

To get the best parallel I/O performance, you need more than just the
file system anyway.  You need a whole I/O software stack from the
application, to libraries like parallel-NetCDF and parallel-HDF5, to
efficient MPI-IO implementations, and then at the foundation a good
parallel file system. 

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B