[Beowulf] Followup on the ConnectX EN 10 GbE driver

[Joe Landman]

Thought a good followup was in order.

Turned out to be a firmware flash issue.  I gave access to the 
JackRabbit to Mellanox and they flashed the firmware (wish I had known 
this was the issue ... I must have misread the firmware download page 
revisions).

After the update, driver loads fine.  Check your firmware.

Performance is quite good.  First time I have seen NFS (not over RDMA) 
give 485 MB/s over plain old ethernet.

[root at jackrabbit ~]# ls -alF /data2/big.file*
-rw-r--r-- 1 root root 34896609280 Feb 13 14:48 /data2/big.file
-rw-r--r-- 1 root root  7918845952 Feb 13 16:31 /data2/big.file2

[root at jackrabbit ~]# mount | grep data2
192.168.10.157:/data on /data2 type nfs (rw,intr,tcp,bg,addr=192.168.10.157)

[root at jackrabbit ~]# dd if=/data2/big.file of=/dev/null bs=8M
4160+0 records in
4160+0 records out
34896609280 bytes (35 GB) copied, 72.3984 seconds, 482 MB/s

I think we should be able to go faster.  Writes are slower (NFS over 
RDMA also has slower writes, but not nearly as different as I saw here).

The Mellanox team was very helpful in getting this resolved.  Can't say 
enough good things about their work to get this going.
-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615