Mounting from nodes

[Robert G. Brown]

On Sun, 5 Nov 2000, Jon Tegner wrote:

> Hi,
> 
> These days hard disks are cheap and it is not unusual to have more
> space on the nodes than what is needed for the system. And if the
> users require lots of space (and if the number of nodes is not too
> large) I was wondering if it is a bad idea to mount a "home" partition
> from every node to all other nodes in the system? I.e. node i exports
> home_i to every other node, while it mounts (mount -t nfs
> nodej:/home_j home_j (for j .ne. i)) the corresponding partition from
> all other nodes in the cluster?
> 
> Also, now that openafs has been released
> 
> (http://oss.software.ibm.com/developerworks/opensource/afs/?dwzone=opensource)
> 
> it might be better to use that instead of nfs?

AFS has some very good features, but it also has some very evil ones.

One evil one is that (at least as of the last time I tested it a couple
of years ago) it laughed at the fflush command.  If you try the
following experiment:

  Host A: 
     open file file.a on shared volume
     write to file.a on shared volume
     fflush file.a on shared volume

then on Host A: you can read the contents of file.a (what you just
wrote) before closing file.a.  HOWEVER, if you try reading that file on
Host B: (which also shares the volume) the file is empty and will remain
so until you close it on Host A:.

This is something that is (or possibly was) really broken -- it
obviously doesn't correspond to the specified requirements for fflush
and file systems in general -- but it does mean that if you wish to e.g.
run a long application on one host (say a node) that is writing a stream
of results to a file, and wish to simultaneously read a snapshot of
those results to date on another host (say a master host) to process
statistics or display them, you EITHER have to constantly open/close the
file in question on the node (very expensive in system time) OR you're
out of luck.

Maybe it is fixed.  Maybe now that it is open source it will get fixed
if it's not fixed.  Maybe (in the minds of the AFS developers) it is a
"feature and not a bug" as I'm sure it makes AFS more efficient to not
flush files back to the actual master volume disk in real time, but IMO
even if it is, they need to write a new command e.g. fflush_afs to
permit users to force a writeback when they need it.

Regarding your primary idea, you might look into pvfs.  The biggest
hassle with your idea above is a lot of administrative overhead and lots
of fragmented, NFS exported/mounted spaces.  You can always use the
space as big local /tmp (we call it /usr/xtmp) which can be used as
local scratch space, with users responsible for moving stuff they dump
there into a backed up volume from time to time.  PVFS, however, ought
to let you harness all that space as a single logical volume, possibly
with less administrative hassle, possibly not. I think.  Check out e.g.
the beowulf underground website for links to the PVFS project and
installation kits.

   rgb

> 
> /jon
> 
> _______________________________________________
> Beowulf mailing list
> Beowulf at beowulf.org
> http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu