[Beowulf] Lmod question

[David Mathog]

Calling all Lmod gurus...

I have been migrating packages from one machine to another, adding their 
Lmod lua files as I go using this script:

https://saf.bio.caltech.edu/pub/software/linux_or_unix_tools/module_generate_from_directory.sh

This is done from an account "modules" and normally as soon as that script 
has completed doing this:

module avail 2>&1 | grep newpackagename

will find the new package.  For some reason today that stopped working,
or at least working immediately for the modules account.  But it still worked 
immediately for root.  File protections were all good, Selinux was ruled 
out.  After poking around this appears to be related to 
cache files, because when "modules" does:

module --ignore_cache avail 2>&1 | grep newpackagename

it sees the new entry.   The cache files were indeed located:

#as modules
ls -al ~/.lmod.d/.cache
total 424
drwxrwxr-x. 2 modules modules     80 Jun 18 14:37 .
drwxrwxr-x. 3 modules modules     20 Nov  1  2019 ..
-rw-rw-r--. 1 modules modules 200725 Jun 18 10:55 
spiderT.x86_64_Linux.luac_5.3

and 10:55 is about the time things started to go wrong.  On my other 
machines "~/.lmod.d" does not exist.  On this machine root has these files 
too, in fact two of them,  and it seems to have updated recently:

-rw-r--r--. 1 root root 269155 Jun 18 14:17 spiderT.x86_64_Linux.lua
-rw-r--r--. 1 root root 204618 Jun 18 14:17 spiderT.x86_64_Linux.luac_5.3

On the problem machine and the ones which do not have this problem the
cache configuration is the same:

module --config 2>&1 | grep -i cache
number of cache dirs               0
Ignore Cache                       no
Cached loads                       no
User cache valid time(sec)         86400
Write cache after (sec)            2

Another oddity - if a new "modules" session is started on the problem
machine, suddenly the missing modules are listed in
"module avail", not only in that old session, but also in the new one.
But the time stamps on the .cache files do not change.

I have not done anything explicitly to generate those cache files.  The 
only thing modified in the files Lmod installed where the paths in

/etc/profile.d/00-modulepath.sh (and .csh)

The output of "module --config" is the same on the bad machine as on the 
good ones (other than things like differences in some version numbers, 
TCL's for instance.)

Any idea what might be going on here?  My best guess is that 
some side effect is running the spider process and generated those
cache files, and then "module" checked them, even though it was
configured not to (I think, see above.)  Ideally I would like to prevent 
it from making those cache files again, kind of hard to do not knowing what
made them in the first place!

This is on CentOS 8, Lmod 8.2.7-1.

Thanks,

David Mathog