[Beowulf] Looking for NFS Server performance comparison
stuart.midgley at ivec.org
Fri Dec 16 16:08:05 PST 2005
Try having multiple servers and then put 1/2 the users on one server
and 1/2 on the other (or what ever fraction you are able to do).
Then automount each home directory from /home1 /home2 on the two
servers into /home on the compute nodes. That way, your users still
see /home/username as their home account (even though it is really /
home1 or /home2).
You can then pull some tricks with login to ensure that a user (upon
login) gets onto their native server (so that interactive tar/gzip
etc are fast). Create a new shell which, upon login, checks which
server is the native for the user and then do an ssh behind the
scenes to the appropriate node without them knowing and then run
their normal shell. Its good to have a direct back-to-back gig-E
link to do this across.
Other things I've done to give better performance is to increase the
number of NFS threads to 64 and set the read/write size to 8192.
This was the optimal nfs performance on a dual ht cpu. We see
roughly 40MB/s over gig-e for large writes (so 2 servers gives you
You can setup your dns to round-robin your front end servers, just
set multiple a records for the 1 dns name and give it the IP
addresses of your multiple servers. In this way, users incoming/
outgoing network bandwidth will (roughly) get spread over your
I run a moderate sized cluster in this way and it works really really
well. Most users are not even aware that we have multiple servers.
On 08/12/2005, at 6:01, Bill Rankin wrote:
> Hi Gang,
> This may be a rehash, but I'm looking for any references to papers
> that address the issue of NFS server scalability in a cluster
> environment. Our current cluster here at Duke is right at 500
> nodes (and growing) and the single NFS server with attached SCSI
> drives serving all data to all nodes just doesn't seem to hack it
> I know that there are clustered solutions like PVFS, IBRIX, et. al.
> as well as the dedicated filer approach (NetApp, EMC, etc.) and we
> are looking at those. But first we need to convince the powers that
> be that a Linux-based NFS server just ain't going to cut it as we
> move forward.
> So if anyone knows of good documents (white papers, conference
> papers, etc.) that will support my assertion, I would appreciate a
> pointer to them.
Dr Stuart Midgley
Industry Uptake Program Leader
iVEC, 'The hub of advanced computing in Western Australia'
26 Dick Perry Avenue, Technology Park
Kensington WA 6151
Phone: +61 8 6436 8545
Fax: +61 8 6436 8555
Email: industry at ivec.org
More information about the Beowulf