[Beowulf] how large of an installation have people used NFS with? would 300 mounts kill performance?

[Joe Landman]

Rahul Nabar wrote:

> I'm thinking of having multiple 10GigE uplinks between the switch and
> the NFS server. The actual storage is planned to reside on a box of
> SAS disks. Approx 15 disks. THe NFS server is planned with at least
> two RAID cards with multiple SAS connections to the box.

ugh ... Why are you designing it ahead of time?  Why not take your 
requirements and needs and use that to dictate the design?

> But that's just my planning. The question is do people have numbers.
> What I/O throughputs are your NFS devices giving?  I want to get a

Depending upon workload, you can get performance ranging from 100MB/s 
through GB+/s.

> feel for what my I/O performance envelope should be like. What kind of
> I/O gurrantees are available? Any vendors around want to comment?

You want a guarantee of I/O performance?  For an arbitrary I/O pattern 
and load?  So if you suddenly start random seeking with 4kB reads, you 
still want to hit 1+GB/s with these 4kB random seek and reads?

Not sure if anyone would be willing to guarantee a particular rate for 
any workload.  We have found well known benchmark codes (bonnie++ 1.0x 
and some of 1.9x) doing not so good I/O (long OS based pauses) where 
other codes seem fine.

We use our io-bm code, fio, and a few others to bang on our systems. 
fio lets us model per unit workloads fairly nicely, io-bm lets us create 
a system/cluster-wide I/O hammer.

> On the other hand just multiplying NFS clients by their peak bandwidth
> (300 x 1 GB) is an overkill. THat is a very unlikely situation. What

Each 1Gb interface can move about 120MB/s best case.  So 300x 120MB/s => 
3.6E+4 MB/s .  This is likely to be overkill, as you report your highest 
IO utilization is about 10% of CPU (need to get what that translates to 
in MB/s, I'd suggest installing iftop on that machine and measuring when 
it is doing its 10% time in IO).


> are typical workloads like? Given x NFS mounts in a computational
> environment with a y GB uplink each what's the factor on the net
> loading of the central storage? Any back of the envelope numbers?

In the distant past, we used 8 nodes per GbE port for a port on the NFS 
server.  This allowed us to serve up to 32 nodes with 4GbE ports, and 
the NFS servers weren't badly loaded.

This ratio is a function of utilization of the links, the I/O duty 
cycle, etc.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615