[Beowulf] Followup on the ConnectX EN 10 GbE driver

Joe Landman landman at scalableinformatics.com
Fri Feb 13 20:55:54 PST 2009

Thought a good followup was in order.

Turned out to be a firmware flash issue.  I gave access to the 
JackRabbit to Mellanox and they flashed the firmware (wish I had known 
this was the issue ... I must have misread the firmware download page 

After the update, driver loads fine.  Check your firmware.

Performance is quite good.  First time I have seen NFS (not over RDMA) 
give 485 MB/s over plain old ethernet.

[root at jackrabbit ~]# ls -alF /data2/big.file*
-rw-r--r-- 1 root root 34896609280 Feb 13 14:48 /data2/big.file
-rw-r--r-- 1 root root  7918845952 Feb 13 16:31 /data2/big.file2

[root at jackrabbit ~]# mount | grep data2 on /data2 type nfs (rw,intr,tcp,bg,addr=

[root at jackrabbit ~]# dd if=/data2/big.file of=/dev/null bs=8M
4160+0 records in
4160+0 records out
34896609280 bytes (35 GB) copied, 72.3984 seconds, 482 MB/s

I think we should be able to go faster.  Writes are slower (NFS over 
RDMA also has slower writes, but not nearly as different as I saw here).

The Mellanox team was very helpful in getting this resolved.  Can't say 
enough good things about their work to get this going.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

More information about the Beowulf mailing list