NFS file server performance

[Josip Loncaric]

Dan Yocum wrote:
> 
> Josip Loncaric wrote:
> >
> > Linux NFS is a bottleneck in itself, even with Gigabit Ethernet.  You
> > can speed up the network, but the current Linux NFS implementation has
> > limitations which make it five times slower than other forms of file
> 
> Well, I think that statement requires a little more qualification: Linux
> <-> Linux NFSv2 performance is quite good - up to about 8MB/s on fast
> ethernet.

Linux <-> Linux NFSv2 over Gigabit Ethernet performs worse than over
Fast Ethernet, but rcp and ftp improve by about a factor of 2.5-3.  

While rcp/ftp use TCP, NFSv2 uses UDP and each of its 8KB blocks is
split into 6 UDP packets.  If any of the six is lost, all six have to be
resent.  To minimize this unhappy situation, our Gigabit Ethernet cards
interrupt the CPU only on every 6th packet received, but the fundamental
problem is that faster networks increase the probablilty that NFSv2 will
drop some packets, so after retransmits the performance gets worse than
what you'd see on a slower network.  Unfortunately, Linux NFSv3 is not
out of its development phase yet.

To answer Art's questions, MPI communication uses TCP (not NFS), except
of course when the code wants to access NFS mounted filesystems.  Faster
networks can improve global performance -- if communication is a
bottleneck (when this can be avoided, faster network won't help much). 
While the PCI bus can limit performance of any high-end network card,
this limit is 133 MB/s or higher.  With the right hardware, you can
typically reach about an order of magnitude higher bandwidth than Fast
Ethernet.  Is it worth the cost?  Would you rather have more CPUs or a
faster network?  You have to benchmark your code and then decide.  Most
of our codes are written to avoid the communication bottleneck.  Only
when this cannot be done do faster networks become attractive.

Sincerely,
Josip


-- 
Dr. Josip Loncaric, Research Fellow               mailto:josip at icase.edu
ICASE, Mail Stop 132C           PGP key at http://www.icase.edu./~josip/
NASA Langley Research Center             mailto:j.loncaric at larc.nasa.gov
Hampton, VA 23681-2199, USA    Tel. +1 757 864-2192  Fax +1 757 864-6134