[Beowulf] Followup on the ConnectX EN 10 GbE driver
landman at scalableinformatics.com
Fri Feb 13 20:55:54 PST 2009
Thought a good followup was in order.
Turned out to be a firmware flash issue. I gave access to the
JackRabbit to Mellanox and they flashed the firmware (wish I had known
this was the issue ... I must have misread the firmware download page
After the update, driver loads fine. Check your firmware.
Performance is quite good. First time I have seen NFS (not over RDMA)
give 485 MB/s over plain old ethernet.
[root at jackrabbit ~]# ls -alF /data2/big.file*
-rw-r--r-- 1 root root 34896609280 Feb 13 14:48 /data2/big.file
-rw-r--r-- 1 root root 7918845952 Feb 13 16:31 /data2/big.file2
[root at jackrabbit ~]# mount | grep data2
192.168.10.157:/data on /data2 type nfs (rw,intr,tcp,bg,addr=192.168.10.157)
[root at jackrabbit ~]# dd if=/data2/big.file of=/dev/null bs=8M
4160+0 records in
4160+0 records out
34896609280 bytes (35 GB) copied, 72.3984 seconds, 482 MB/s
I think we should be able to go faster. Writes are slower (NFS over
RDMA also has slower writes, but not nearly as different as I saw here).
The Mellanox team was very helpful in getting this resolved. Can't say
enough good things about their work to get this going.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Beowulf