[Beowulf] Help for terrible NFS write performance

[Joe Landman]

Orion Poplawski wrote:
> I'm trying to improve the terrible NFS (write in particular) performance 
> I'm seeing.  Pure network performance does not appear to be an issue as 
> I can hit 120MB/s reading which should be about the limit for gigE. 
> Perhaps the local disk performance is not what it should be.  Any help 
> would be greatly appreciated.  Using bonnie++ for benchmarks.
> 
> Server:
> 
> Dual proc dual core opteron 2GHz
> 8GB RAM
> CentOS 4.7
> kernel  2.6.9-78.0.22.plus.c4smp
> 3 8-port Marvell MV88SX6081 SATAII controllers
> sata_mv 3.6.2 driver
> Ethernet controller: nVidia Corporation MCP55 Ethernet (rev a3)
> MTU 8982
> 
> Arrays are linux md arrays of 6 disks with 2 on each controller.  64k 
> cunks.  ext3 filesystem.

If I had to bet, ext3 would have much to do with this ... though, 
honestly, md RAID write performance over NFS is nothing to write home 
about.  We can get ~350MB/s on our DeltaV's, but this takes lots of work.

> 
> "working" - raid0 ST31000340AS 1TB drives
> local perf: 224-240MB/s write, 135MB/s rewrite, 390-400MB/s read
> "cora6" - raid5  ST31500341AS 1.5TB drives
> local perf: 84MB/s write, 42MB/s rewrite, 161-166MB/s read
> 
> /etc/exports:
> /export          *.cora.nwra.com(rw,sync,fsid=0)
> /export/cora6    *.cora.nwra.com(rw,sync,nohide)
> /export/working  *.cora.nwra.com(rw,sync,nohide)

Ok.  There it is...  Sync.

Don't need to see anything else.

That is it.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615