[Beowulf] Which do you prefer local disk installed OS or NFS rooted?

[Mark Hahn]

> What benefits and drawbacks have all of you seen by doing either?

using nfsroot means quite a bit of traffic that could be avoided
if those files were installed on each node's disk.  using nfsroot
also makes the nodes less independent, more vulnerable to problems
on the server(s).

using nfsroot is incredibly convenient.  nodes can even tolerate a 
broken disk.

I've built two significant clusters now with nfsroot, and intend to 
use it on a very large one in the coming year.  at least on the two
smaller clusters (~100 duals), there is no clear sign that NFS load
is causing a problem.  in both cases, the single fileserver is nothing
special (a dual, 1G ram, md raid, single gigabit connection).  if there
did appear to be a problem arising, I'd first split across a couple
fileservers, rather than abandoning nfsroot.

many large clusters are now being designed with fileservers sitting
on the fast cluster interconnect.  this trend seems to support the 
nfsroot approach, since going from 60 MB/s over gigabit to 800 MB/s
over quadrics/IB would postpone NFS congestion.

regards, mark hahn.