[Beowulf] tools for cluster event logging?

[David Mathog]

What would be a good tool for logging cluster specific messages, and
nothing else, on a single server?  The purpose of this is to let
computer nodes send messages like "node XXX hardware failure, shutting
down", or "node xxx, boot sequence completed" messages to a central
repository.  But I do not want any other messages logged to the
repository from the clients.

I suppose syslog could be used for this, but the trick would be to
choose a facility/priority for it such that nothing other than the
desired cluster messages was ever sent.  In other words, something
like:

logger -p cluster.info "this is a cluster message"

Unfortunately there is no "cluster" facility, and I do not know
which one of the 20 or so defined facilities (auth, authpriv... local7)
will never be used by some other part of the client OS.

The main reason I'm looking for this now, after so many years of doing
without it, is that changes in umount and umount.nfs and the NFS umount
section of the distro I use have resulted in the loss of the "unmount
request" messages which used to be logged on the NFS server when a
client shut down normally.  (In brief, "umount -l /mountpoint" used
to send these, but it no longer does.)  In the past I used those
messages, and the corresponding "mount request" messages to determine
what the clients were doing, or if they had crashed or shut down
normally.  Since that isn't possible now, I want to modify the init and
hardware monitor scripts to send specific messages.

I am running ganglia, but that doesn't have this particular capability,
at least as far as I can tell.

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech