[Beowulf] typical protocol for cleanup of /tmp: on reboot? cron job? tmpfs?

[Rahul Nabar]

What's the typical protocol about the cleanup of /tmp folders? Do
people clean them on each reboot or at intervals with a cron (sounds a
bad idea). I was always under the impression that a reboot cleans them
but apparantly not on my CentOS distro, by default.

I was burnt earlier today when ompi-ps acted erratically and I
diagnosed it to be caused by stale state information in the /tmp
folder. The remnant of some old dead jobs that  had somehow crashed.

One other option that I've seen mentioned is mounting /tmp on a tmpfs.
Is that a good idea? The risk of using up too much RAM if a program
gets out of hand writing to /tmp.

On the other hand compute-nodes can go a long time without any
reboots; so a more frequent cleanup cycle on /tmp might be desirable?
I suppose most programs ought to cleanup behind them on /tmp but then
again there are bound to be bad apples.

Any comments?

-- 
Rahul