[Beowulf] Re: High performance storage with GbE?

[Steve Cousins]

Bill Broadley wrote:

> Sorry all, my math was a bit off (thanks Mike and others).
>
> To be clearer:
> * $24k or so per pair of servers and a pair of 15*750GB arrays
> * The pair has 16.4TB usable (without the normal 5% reserved by the
>   filesystem)
> * The pair has 20.5TB raw  (counting disks use for spare and redundancy)
> * Each pair has 22 TB marketing (Using TB = 10^9 instead of 2^30)
>
> So 7 servers could manage to provide the mentioned >= 100TB.  Total cost of
> 7*$24k=$168k.  Total storage cost does not include the GigE switches, 10G
> uplinks, nor 10G nics.  Usable capacity of 7*16 = 112TB.

Ah, I see.  That brings it down quite a bit. Thanks for clarifying.

> The above config assumes that software RAID performance = hardware RAID
> performance or that the Dell MD-1000 can allow 2 masters access to
> all 15 disks (not at the same time).  I'm not sure that either is true.
>
> From the scaling numbers I've seen published with PVFS2 and Lustre seems like
> it should still be possible to manage the mentioned goal of 1.5-2.5GB/sec
> (which assumes 220MB/sec or 440MB/sec per file server pair for 7 pairs.)

Again, if anyone can provide a real-world layout of what they are using, 
along with real-world speeds at the node, that would help out 
tremendously.

Thanks,

Steve