NFS 100:1 performance loss

[Martin Siegert]

On Fri, 18 Aug 2000, Josip Loncaric wrote:

> Has anyone seen this NFS performance problem?
> 
> Our Beowulf has two servers w/RAID-0 arrays which deliver 45MB/s,
> connected via Gigabit Ethernet.  The machines are dual PIII/500 systems
> with 512MB of RAM each.  We are running Red Hat 6.2 updated to kernel
> 2.2.16-3 and recently we updated to nfs-utils-0.1.9.1-1.  We are running
> 16 kernel nfsd threads on each machine.  Here is how long it takes to
> copy a 28,955,860 byte file from machine 1 to machine 2:
> 
> rcp:       1.04 seconds (27.8 MB/s, where 1MB=10^6B)
> ftp:       1.12 seconds (25.8 MB/s)
> NFS 1KB:  12.35 seconds ( 2.3 MB/s, using rsize=wsize=1024)
> NFS 8KB: 129.42 seconds ( 0.2 MB/s, using rsize=wsize=8192)
> 
> Clearly, there is something very wrong with NFS, particularly with 8KB
> rsize/wsize (which should have improved performance!).  Our system
> manager tells me that reduced Linux NFS performance with 8KB rsize/wsize
> is a known problem, but even at 1KB our NFS is getting less than 10% of
> the rcp or ftp performance.
> 
> Any ideas?

No ideas. Sorry. But here are the numbers from our system:
dual PII/400, 256MB RAM
kernel-2.2.16-3
nfs-utils-0.1.9.1-1
No RAID, etc., just IDE drives, /home and /usr/local NFS exported from the
master node.

Contents of /etc/fstab:
...
b01:/usr/local          /usr/local              nfs     ro,hard,intr,bg,rsize=8192 0 0
b01:/home               /home                   nfs     rw,hard,intr,bg,rsize=8192,wsize=8192 0 0

# cd /usr/local/src/tar-files
# ls -l linux-2.2.16bugaboo.tar
-rw-r--r--   1 root     root     81674240 Jun 28 15:13 linux-2.2.16bugaboo.tar

# time cp linux-2.2.16bugaboo.tar /tmp

real    0m9.799s
user    0m0.040s
sys     0m2.150s

That's about 63.59 Mbit/s.
Second test (after "gzip linux-2.2.16bugaboo.tar" on master node):
# ls -l linux-2.2.16bugaboo.tar.gz
-rw-r--r--   1 root     root     20673033 Jun 28 15:13 linux-2.2.16bugaboo.tar.gz
# time cp linux-2.2.16bugaboo.tar.gz /tmp

real    0m2.383s
user    0m0.000s
sys     0m0.280s
That's about 66.19 Mbit/s.

In the other direction:
# cd /tmp
# time cp linux-2.2.16bugaboo.tar.gz /home/siegert

real    0m1.854s
user    0m0.000s
sys     0m0.270s
That's about 85.07 Mbit/s.

This is with 100baseT and 3C905B NICs.
So I doubt that it is NFS that is at fault here ...

Cheers,
Martin

========================================================================
Martin Siegert
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: siegert at sfu.ca
Canada  V5A 1S6
========================================================================