[Beowulf] RE: Network Filesystems performance (Glen Dosey)

[David Mathog]

> Glen Dosey <doseyg at r-networks.net> wrote
> On Fri, 2007-08-24 at 08:45 -0400, Lawrence Sorrillo wrote:
> > I am uncertain what this step does....
> > 
> > now we unmount the NFS share, recreate the file on the server, and
remount
> > it to clear the client cache but leave it cached on the server
> > 
> > [root at wopr1 ~]# dd if=/mnt/array3/file.dd of=/dev/null bs=4k
> > 524287+0 records in
> > 524287+0 records out
> > 2147479552 bytes (2.1 GB) copied, 18.5161 seconds, 116 MB/s
> > 
> > 
> > What does it accomplish?
> 
> I was using this step to illustrate that NFS itself is capable of
> transferring data across the GigE network at near wires speeds. 
> 
> By unmounting the NFS share we clear the file from being cached in RAM
> on the client, ensuring that it must be gotten from the server again via
> NFS as opposed to being grabbed from RAM on the client. 

I ran into a nasty bug involving "dump" on recent kernels,
the long and still not fully resolved story is here:

http://bugzilla.kernel.org/show_bug.cgi?id=8636

In a nutshell, dump on recent kernels runs hideously slow on (P)ATA
disks (by a factor of 6!) because of this change in the .config file:

 -CONFIG_DEFAULT_IOSCHED="cfq"
+CONFIG_DEFAULT_IOSCHED="anticipatory"

(cfq in the new kernels, anticipatory in the old ones). The "fix" was:

echo "anticipatory" > /sys/block/hda/queue/scheduler

Now this may be completely irrelevant, for instance, there was no
problem with the SCSI disks I tested, but there might be with your raid
controller.  If NFS can move the data across the network at full speed,
but can't seem to read nearly as fast as the controller, it's possible
that the scheduler you're using is involved.  There are 4 to choose
from, perhaps give them all a test and see if any of them improve matters?  

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech