[Beowulf] What are people seeing performance-wise for NFS over 10GbE

[Joe Landman]

Hi folks:

   Have a few simple setups that I am trying to figure out if I have a 
problem, or if what I am seeing is normal.

   Two 10 GbE cards, connected with a CX4 cable.  Server is one of our 
JackRabbits, with 750+ MB/s direct IO and 500-650 MB/s buffered IO 
(read-write), for IO about 10x system ram (~100x RAID cache).

   Getting ok iperf numbers, about 7 Gb/s single thread.  Running NFS, 
and seeing ~200-300 MB/s best case between the systems.  Is this what 
others have seen?  Latest drivers from vendor, sent a note to them to 
see if we can figure this out.  Figured I would tap into the 'wulf 
collective memory to learn what other people see.

   2.6.23.14 kernel on both sides, jumbo frames enabled. No switch, just 
a CX4 cable.

   My expectations are that we would be able to see 80-90% of the 
JackRabbits speed, with the rest being eaten by stack issues.  This is 
the case over channel bonded GbE.

   Do you see 500 MB/s or more?  Less?

   Thanks.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615