[Beowulf] Help for terrible NFS write performance

[Orion Poplawski]

I'm trying to improve the terrible NFS (write in particular) performance 
I'm seeing.  Pure network performance does not appear to be an issue as 
I can hit 120MB/s reading which should be about the limit for gigE. 
Perhaps the local disk performance is not what it should be.  Any help 
would be greatly appreciated.  Using bonnie++ for benchmarks.

Server:

Dual proc dual core opteron 2GHz
8GB RAM
CentOS 4.7
kernel  2.6.9-78.0.22.plus.c4smp
3 8-port Marvell MV88SX6081 SATAII controllers
sata_mv 3.6.2 driver
Ethernet controller: nVidia Corporation MCP55 Ethernet (rev a3)
MTU 8982

Arrays are linux md arrays of 6 disks with 2 on each controller.  64k 
cunks.  ext3 filesystem.

"working" - raid0 ST31000340AS 1TB drives
local perf: 224-240MB/s write, 135MB/s rewrite, 390-400MB/s read
"cora6" - raid5  ST31500341AS 1.5TB drives
local perf: 84MB/s write, 42MB/s rewrite, 161-166MB/s read

/etc/exports:
/export          *.cora.nwra.com(rw,sync,fsid=0)
/export/cora6    *.cora.nwra.com(rw,sync,nohide)
/export/working  *.cora.nwra.com(rw,sync,nohide)

/etc/sysconfig/nfs:
# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 
(auth=krb5,krb5i, or krb5p)
SECURE_NFS="no"
RPCNFSDCOUNT=32
MOUNTD_NFS_V2=no


client:
Dual Opteron 246
3GB RAM
CentOS 5.3
kernel 2.6.18-144.el5
Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit 
Ethernet (rev 03)

mounts:
NFSv4 rsize=32768,wsize=32768
"working" - 60MB/s write, 12MB/s rewrite, 120MB/s read
"cora6" - 21MB/s write, 7.7MB/s rewrite, 120MB/s read

Thanks again!

-- 
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA/CoRA Division                    FAX: 303-415-9702
3380 Mitchell Lane                  orion at cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com