autofs mount on bproc node?

Wed May 8 08:18:39 PDT 2002

Hi,

any of you great gurus managed to use autofs on bproc nodes?

I am pushing hard, but at this moment it looks like hitting concrete
wall with my head... any help would be more than welcome.

My nodes already run NFS client, NFS server and syslogd/klogd.
Automount seemes to start OK but when I ls automounted directory (node
is client, head is server), ls hangs, automount process hangs (kill -9
needed to kill it) and there is no error message anywhere.

I have syslogd working on node and automount is full of syslog()
calls, but in this case is says nothing, it probably hangs early when
receiving automount request from kernel.

The same automount setup works when head is client and node is server.

(Is there any way to force compiled kernel to give out more debug
messages? E.g. write somewhere to /proc?)

Running daemons on bproc nodes is tricky and probably worth
mini-howto. (Yes, I try to avoid daemons on nodes, but sometimes I
really need them.) If you know any related documents, please let me
know.

Thanks

Vaclav

-------------------------------------------------------

My setup:

RedHat 7.2 with most rpm updates till Nov 2001, Clustermatic (March
2002 version), kernel 2.4.18-lanl.16, automount version 3.1.7 (also
tested 4.0.0).

==> /etc/auto.master <==
# /etc/auto.master
/nfs  /etc/auto.nfs  rw,intr,rsize=8192,wsize=8192

==> /etc/auto.nfs <==
# /etc/auto.nfs
*  -fstype=autofs,-Dhost=&  file:/etc/auto.sub

==> /etc/auto.sub <==
# /etc/auto.sub
*  ${host}:/&

==> /etc/beowulf/syslog.conf <==
# syslog.conf for magi nodes

# log everything on screen:
*.*                                                     /dev/console

# log everything to head (magi):
*.*                                                     @10.0.4.1

==> /etc/beowulf/exports.node <==
# exports for beowulf node (experimentel)
/etc 10.0.4.1(ro)

==> /etc/beowulf/nsswitch.conf <==
passwd: bproc files
hosts: bproc

==> /etc/exports <==
# magi exports
/bin 10.0.4.0/255.255.255.0(ro)
/home noel(rw) 10.0.4.0/255.255.255.0(ro)
/lib 10.0.4.0/255.255.255.0(ro)
/sbin 10.0.4.0/255.255.255.0(ro)
/usr 10.0.4.0/255.255.255.0(ro)
/var 10.0.4.0/255.255.255.0(ro)

==> /etc/sysconfig/syslog <==
# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
#SYSLOGD_OPTIONS="-m 0"
# VH: added -r for log from magi nodes to magi master:
SYSLOGD_OPTIONS="-m 0 -r"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and

== And my main experimental script: ==

#!/bin/bash

# beowulf node startup for magi

# New (RH7.2_Clustermatic/magi) version

echo This is /nfs/noel/home/hanzl/beowulf/startnode

########### CONFIG AREA ############

#nodenum:
N=1

#master IP:
MASTER=10.0.4.1

# directories to NFS-replicate from master (read only):

NFSDIRS='bin sbin usr'

# directories to just create on node:

CREATEDIRS='tmp nfs .autofsck var/lib/nfs/sm var/run var/lock/subsys var/nis'

# files to copy from master to node with identical pathname:

COPYFILES='/etc/auto.master /etc/auto.nfs /etc/auto.sub /etc/services /etc/rpc /etc/protocols /etc/passwd /etc/group'

# modules needed on nodes:

MODULES='sunrpc lockd nfs nfsd autofs'

####################################

echo Master IP: $MASTER, node number: $N

bpstat $N

echo Rebooting node $N...

bpctl -S $N -s reboot

while true; do
 STATUS=`bpstat $N -s`
 echo -n ' '$STATUS
 if [[ "$STATUS" == "up" ]]; then break; fi
 sleep 1
done

echo ''
echo Node $N is up!

####################################

echo "Creating any missing directories on node $N:"

for d in $NFSDIRS $CREATEDIRS; do
  echo -n ' '$d
  bpsh $N mkdir -p /$d
done

echo ''
echo ... done

####################################

echo "Inserting modules on node $N"

for m in $MODULES; do
  echo -n ' '$m
  modprobe --node $N $m
done

echo ''
echo ... done

####################################

echo "Copying config files to node $N"

echo "Files with identical pathname:"

for f in $COPYFILES; do
  echo -n ' '$f
  bpcp $f $N:$f
done

echo ''
echo ... done

echo "Files with different pathname:"

bpcp /etc/beowulf/syslog.conf $N:/etc/syslog.conf
bpcp /etc/beowulf/exports.node $N:/etc/exports
bpcp /etc/beowulf/nsswitch.conf $N:/etc/nsswitch.conf

echo ... done

####################################

####################################
echo "Hello to console"|bpsh $N dd of=/dev/console 2>/dev/null

####################################

echo "Starting basic daemons on node $N"

bpsh $N touch /var/lib/nfs/rmtab 
bpsh $N touch /var/lib/nfs/xtab  
bpsh $N touch /var/lib/nfs/etab

echo "portmap..."
bpsh $N portmap

echo "syslogd..."
bpsh $N syslogd

echo "klogd..."
bpsh $N klogd

bpsh $N logger "Starting basic rpc daemons on node $N"

echo "rpc.statd..."
# statd will run in /var/lib/nfs/statd as rpcuser and needs rw access
bpsh $N mkdir -p /var/lib/nfs/statd
bpsh $N chown rpcuser /var/lib/nfs/statd
bpsh $N chgrp rpcuser /var/lib/nfs/statd
bpsh $N rpc.statd 
bpsh $N touch /var/lock/subsys/nfslock

echo "Expected rpcinfo:" nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`
echo "(maybe they did not start yet...)"
# 'nlockmgr' is provided by kernel module, not daemon

####################################
# (This could have been done even just after portmap, but is probably safer here)
echo "Mounting NFS directories, node $N is client, head is server"
bpsh $N logger "Mounting NFS directories, node $N is client, head is server"
for d in $NFSDIRS; do
  echo -n ' '$d
  bpsh 1 mount 10.0.4.1:/$d /$d
done
echo ''
echo ... done
####################################

echo "Expected rpcinfo:" nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`

echo "Starting NFS server daemons on node $N"
bpsh $N logger "Starting NFS server daemons on node $N"

# We should provide: mountd nfs rquotad

RPCNFSDCOUNT=8

echo 'exportfs (not daemon)...'
bpsh $N exportfs -r

## echo "rpc.rquotad"
## bpsh $N rpc.rquotad
## # hangs (but shows in rpcinfo), avoid it

echo "rpc.mountd..."
# Special treatment needed: rpc.mountd with socket on stdin goes mad
# and therefore cannot be started by bpsh directly. Stdin redirect
# on node helps. You can either NFS-mount /usr/sbin or you can use
# new bproc, which delivers missing executables when absolute path
# is used.
bpsh $N sh -c '/usr/sbin/rpc.mountd </dev/null'

echo "rpc.nfsd (count=$RPCNFSDCOUNT)..."
bpsh $N rpc.nfsd $RPCNFSDCOUNT

bpsh $N touch /var/lock/subsys/nfs

echo "NFS server on node $N should work now"

####################################

echo "Expected rpcinfo:" mountd nfs nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`

echo "Testing automount (head is client, node $N is server):"
echo "ls /nfs/n$N/etc:"
ls /nfs/n$N/etc

####################################

echo "Processes on node $N seen in head PID-space:"

ps aux|bpstat -P|grep ^$N'[^0-9]'

####################################

exit

echo "Starting autofs client on node $N"
bpsh $N logger "Starting autofs client on node $N"

bpsh $N /usr/sbin/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192
bpsh $N touch /var/lock/subsys/autofs

#bpsh $N sh -c '/usr/sbin/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192'

#bpsh $N sh -c '/root/autofs-4.0.0pre10/daemon/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192'

# bpsh $N ls /nfs/n-1/var wil HANG !!! :-(

==================== END =====================