NFS 100:1 performance loss

[Josip Loncaric]

Jakob Østergaard wrote:
> 
> On Fri, 18 Aug 2000, Josip Loncaric wrote:
> 
> >  We are running
> > 16 kernel nfsd threads on each machine.  Here is how long it takes to
> > copy a 28,955,860 byte file from machine 1 to machine 2:
> >
> > rcp:       1.04 seconds (27.8 MB/s, where 1MB=10^6B)
> > ftp:       1.12 seconds (25.8 MB/s)
> > NFS 1KB:  12.35 seconds ( 2.3 MB/s, using rsize=wsize=1024)
> > NFS 8KB: 129.42 seconds ( 0.2 MB/s, using rsize=wsize=8192)
> 
> You do use the kernel nfs server right ?  knfsd, not the old
> userspace nfs daemon...

We *do* use kernel nfsd -- but here is a hint of what may be wrong with
Linux NFS:

We did not have a serious NFS problem while we used jumbo frames
(MTU=9000) on our Gigabit Ethernet interfaces.  Recently, we switched to
regular size frames (MTU=1500) so that the servers could communicate
with machines which cannot handle jumbo frames.  I assume that
rsize=wsize=8192 means that NFS will send bursts of UDP packets to carry
an 8KB message.  Our MTU change means that an 8KB NFS message cannot be
sent as a single UDP packet, but takes at least 6 UDP packets.  On the
theory that kernel nfsd cannot handle very high data rates and that some
UDP packets were not being received correctly, I've changed the
interrupt settings on our interface card and got much better results:

30 microseconds / 2 frames  ------> NFS 8KB test takes 129.42 seconds
75 microseconds / 6 frames  ------> NFS 8KB test takes   4.92 seconds

Before, the NIC would interupt within 30 microseconds or on receiving 2
frames (whichever came first), thus delivering an incomplete NFS message
in all cases.  Now, it interrupts within 75 microseconds or on receiving
6 frames, which has a good chance of delivering the full NFS message.

It's nice that we got back a factor of 25, but this NFS performance is
still 4+ times slower than the plain 'rcp'.  I'll experiment with this
further and report what I learn.

Sincerely,
Josip



-- 
Dr. Josip Loncaric, Senior Staff Scientist        mailto:josip at icase.edu
ICASE, Mail Stop 132C           PGP key at http://www.icase.edu./~josip/
NASA Langley Research Center             mailto:j.loncaric at larc.nasa.gov
Hampton, VA 23681-2199, USA    Tel. +1 757 864-2192  Fax +1 757 864-6134