[Beowulf] High performance storage with GbE?

[Steve Cousins]

We are currently looking to upgrade storage on a 256 node cluster for both 
performance and size (around 100 TB).  The current thread on distributed 
file storage talks about a number of issues we are dealing with.

We are trying to spec out storage that would give us somewhere in the 
neighborhood of 5 to 10 MB/sec at each node which translates to around 1.2 
to 2.5 GB/sec when it gets to the storage. Of course, this is only if all 
nodes are trying to access/write data at once but for some models this is 
a possibility.  We are trying to do this with GbE since we can't afford to 
add FC type cards to each node.

Some vendors use NAS gateways to consolidate a number of FC DAS RAID 
arrays together and then serve out NFS via multiple GbE pipes.  Others 
(Panasas, Isilon) do distributed storage that scales up as you add storage 
shelves.  Then there are distributed file systems like PVFS and Lustre 
that would allow us to use node disks as scratch space.  Since we are on a 
fairly tight budget I don't think we can afford to do all Panasas/Isilon 
type storage.  We are probably going to need to do a tiered system with 
faster and slower storage with scratch and permanent space.  Although a 
tape silo is a possibility for part of the storage, for now we want to go 
with all disk storage.

My question is: what are other people doing to be able to provide fast 
GbE storage to the nodes?

Thanks,

Steve
______________________________________________________________________
  Steve Cousins, Ocean Modeling Group    Email: cousins at umit.maine.edu
  Marine Sciences, 452 Aubert Hall       http://rocky.umeoce.maine.edu
  Univ. of Maine, Orono, ME 04469        Phone: (207) 581-4302