[Beowulf] Looking for NFS Server performance comparison

[Stuart Midgley]

Hi Bill

Try having multiple servers and then put 1/2 the users on one server  
and 1/2 on the other (or what ever fraction you are able to do).   
Then automount each home directory from /home1 /home2 on the two  
servers into /home on the compute nodes.  That way, your users still  
see /home/username as their home account (even though it is really / 
home1 or /home2).

You can then pull some tricks with login to ensure that a user (upon  
login) gets onto their native server (so that interactive tar/gzip  
etc are fast).  Create a new shell which, upon login, checks which  
server is the native for the user and then do an ssh behind the  
scenes to the appropriate node without them knowing and then run  
their normal shell.  Its good to have a direct back-to-back gig-E  
link to do this across.

Other things I've done to give better performance is to increase the  
number of NFS threads to 64 and set the read/write size to 8192.   
This was the optimal nfs performance on a dual ht cpu.  We see  
roughly 40MB/s over gig-e for large writes (so 2 servers gives you  
80MB/s).

You can setup your dns to round-robin your front end servers, just  
set multiple a records for the 1 dns name and give it the IP  
addresses of your multiple servers.  In this way, users incoming/ 
outgoing network bandwidth will (roughly) get spread over your  
multiple servers.

I run a moderate sized cluster in this way and it works really really  
well.  Most users are not even aware that we have multiple servers.

Stu.



On 08/12/2005, at 6:01, Bill Rankin wrote:

> Hi Gang,
>
> This may be a rehash, but I'm looking for any references to papers  
> that address the issue of NFS server scalability in a cluster  
> environment.  Our current cluster here at Duke is right at 500  
> nodes (and growing) and the single NFS server with attached SCSI  
> drives serving all data to all nodes just doesn't seem to hack it  
> anymore.
>
> I know that there are clustered solutions like PVFS, IBRIX, et. al.  
> as well as the dedicated filer approach (NetApp, EMC, etc.) and we  
> are looking at those. But first we need to convince the powers that  
> be that a Linux-based NFS server just ain't going to cut it as we  
> move forward.
>
> So if anyone knows of good documents (white papers, conference  
> papers, etc.) that will support my assertion, I would appreciate a  
> pointer to them.
>
> Thanks,
>
> -bill


--
Dr Stuart Midgley
sdm900 at gmail.com