[Beowulf] 512 nodes Myrinet cluster Challanges

[Robert Latham]

On Fri, Apr 28, 2006 at 09:27:19AM +0200, Jaime Perea wrote:
> >From my point of view, the big problem there is the IO, we installed
> on our small cluster the pvfs2 system, it works well, using the
> myrinet gm for the passing mechanism, the pvfs2 is only a solution
> for parallel IO, since mpi can use it. On the other hand it can not
> be used for the normal user stuff, so you have to take that into
> account and think carefully on how to install a good poweful nfs server
> machine which has to be on an alternative standard network. On the
> "other" architecture IBM's gpfs is really a nice alternative.  

Hi

I'm one of the PVFS2 developers, so I'm biased, but good choice of
parallel file system!

when you say "can not be used for the normal user stuff" what do you
mean exactly?  It's a pretty common setup on many clusters to have a
slower home directory file system and a faster scratch file system for
applications.   

Just to clarify, you can use PVFS2 for just about any workload,
though yes, we are a lot slower for some typical "home directory"
workloads than the other file systems out there.  PVFS2 performs best
when coupled with an MPI-IO application, but we have a lot of users
who just use plain old serial unix read() and write() (or some variant
of those commands) and have satisfactory experiences.  

Hope you are having good success with PVFS2.  If you run into any
problems or questions, please don't hesitate to ask on
pvfs2-users at beowulf-underground.org

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B