[Beowulf] best archetecture / tradeoffs
hahn at physics.mcmaster.ca
Mon Aug 29 21:20:42 PDT 2005
> This VNFS resides on the master/boot server, and is used to construct
> the root filesystem for each node in your cluster. You can make changes
> to this template directly using chroot, or indirectly with other scripts/tools
> for example: rpm --root /vnfs/default ..., or yum
> --installroot=/vnfs/default ...
heh, that's exactly what I do, but never thought to give it a name.
or rather, I thought "nfsroot" pretty well covered it (and rejoiced
that rpm/yum have those switches.)
so in what sense is it a virtual node FS? how is it different from
the fairly common practice of an NFS-mounted root filesystem?
> recent versions, we introduced a hybrid NFS/ramdisk scheme that reduces
> the permanent RAM footprint dramatically by using a readonly NFS mount
> of the VNFS for non-critical files. Thus, you can have your full blown text
presumably just most bits of /var, no? are there less obvious bits that
you feel need to be in the ramdisk? incidentally, do you use a ramdisk,
or do you use initrd's cpio format, or do you simply populate a tmpfs
during the boot process? I do the latter - it seemed simplest, once the
initrd is under way, and has the NFS root mounted, to just mount tmpfs
here and there and untar (the tar is in the NFS too...)
> editor, compiler, and X installed in the VNFS, yet still have only a 15 to
> 30 MB ramdisk on the nodes. Which files reside on the ramdisk vs. which
hmm, 15M seems fairly elaborate, or do you not pivot/umount away your
> created. Thus, you can get much of the small-RAM-footprint benefit of
> the NFS-root scheme, yet have dramatically lower NFS traffic to the server
> during normal cluster use.
I'd heard people say that was a problem, but haven't found it so. what files
are inadequately cached by NFS and wind up causing noticable traffic?
it seems like starting a new job would read little more than the user's
shell, some shared libraries, /etc/passwd and friends. I haven't tried to
collect traces, but they seem quite NFS-caching-friendly...
regards, mark hahn.
More information about the Beowulf