NFS 100:1 performance loss
siegert at sfu.ca
Fri Aug 18 15:03:10 PDT 2000
On Fri, 18 Aug 2000, Josip Loncaric wrote:
> Has anyone seen this NFS performance problem?
> Our Beowulf has two servers w/RAID-0 arrays which deliver 45MB/s,
> connected via Gigabit Ethernet. The machines are dual PIII/500 systems
> with 512MB of RAM each. We are running Red Hat 6.2 updated to kernel
> 2.2.16-3 and recently we updated to nfs-utils-0.1.9.1-1. We are running
> 16 kernel nfsd threads on each machine. Here is how long it takes to
> copy a 28,955,860 byte file from machine 1 to machine 2:
> rcp: 1.04 seconds (27.8 MB/s, where 1MB=10^6B)
> ftp: 1.12 seconds (25.8 MB/s)
> NFS 1KB: 12.35 seconds ( 2.3 MB/s, using rsize=wsize=1024)
> NFS 8KB: 129.42 seconds ( 0.2 MB/s, using rsize=wsize=8192)
> Clearly, there is something very wrong with NFS, particularly with 8KB
> rsize/wsize (which should have improved performance!). Our system
> manager tells me that reduced Linux NFS performance with 8KB rsize/wsize
> is a known problem, but even at 1KB our NFS is getting less than 10% of
> the rcp or ftp performance.
> Any ideas?
No ideas. Sorry. But here are the numbers from our system:
dual PII/400, 256MB RAM
No RAID, etc., just IDE drives, /home and /usr/local NFS exported from the
Contents of /etc/fstab:
b01:/usr/local /usr/local nfs ro,hard,intr,bg,rsize=8192 0 0
b01:/home /home nfs rw,hard,intr,bg,rsize=8192,wsize=8192 0 0
# cd /usr/local/src/tar-files
# ls -l linux-2.2.16bugaboo.tar
-rw-r--r-- 1 root root 81674240 Jun 28 15:13 linux-2.2.16bugaboo.tar
# time cp linux-2.2.16bugaboo.tar /tmp
That's about 63.59 Mbit/s.
Second test (after "gzip linux-2.2.16bugaboo.tar" on master node):
# ls -l linux-2.2.16bugaboo.tar.gz
-rw-r--r-- 1 root root 20673033 Jun 28 15:13 linux-2.2.16bugaboo.tar.gz
# time cp linux-2.2.16bugaboo.tar.gz /tmp
That's about 66.19 Mbit/s.
In the other direction:
# cd /tmp
# time cp linux-2.2.16bugaboo.tar.gz /home/siegert
That's about 85.07 Mbit/s.
This is with 100baseT and 3C905B NICs.
So I doubt that it is NFS that is at fault here ...
Academic Computing Services phone: (604) 291-4691
Simon Fraser University fax: (604) 291-4242
Burnaby, British Columbia email: siegert at sfu.ca
Canada V5A 1S6
More information about the Beowulf