[Beowulf] automount on high ports

[Robert G. Brown]

On Wed, 2 Jul 2008, Bogdan Costescu wrote:

> On Wed, 2 Jul 2008, Robert G. Brown wrote:
>
>> The way TCP daemons that listen on a well-known/privileged port work is 
>> that they accept a connection on that port, then fork a connection on a 
>> higher unprivileged (>1023) port on both ends so that the daemon can listen 
>> once again.
>
> 'man 7 socket' and look up SO_REUSEADDR. I don't quite know what you mean by 
> 'forking a connection'; when the daemon encounters a fork() all open file 
> descriptors (including sockets) are being kept in both the parent and the 
> child. The child (usually the part of the daemon that processes the content 
> that comes on that connection) gets the same 4-tuple as the parent. The 
> parent closes its file handle so that only the child is then active on that 
> connection.

I'm stating it badly and incorrectly, confusing port with socket.  See
the following code.  Server listens, bound to a specific port.  When a
connection is initiated by a (possibly remote) client, it accepts it
(creating a socket with its own FD), leaving the original server socket
FD unaffected.  It then forks and the child CLOSES the original socket
lest there be trouble.  The server/parent similarly closes the client
fd.  The client typically got a "random" (kernel chosen) port on ITS
side from the list of available unprotected ports when it formed its
original socket, and it forms one side of the stream connection, with
the server "accept" socket being the other.

What I was trying to convey remarkably poorly is that once you've
created a daemon and bound it to a port, if you try to start up a second
daemon on that port you'll get a EADDRINUSE on the bind (and fail the
loop that checks below), and so if you DON'T fork off the sockets with
listen/accept you'll usually block the port indefinitely while handling
each connection.  I haven't tried (at least, not deliberately:-) not
going through the asymmetric close so that the two processes both have
all the FDs, but I'd guess bad things would happen if I did, a crap
shoot race condition as to which process gets the data or worse.

OTOH, some applications (esp nfsd and httpd) DO fork several child
processes with the original open socket fd so that if incoming requests
for a connection come while one of them is "busy" with the creation of a
child of its own to handle the connection, another will pick it up round
robin.  Unless I'm misunderstanding how they work this or why.  mail is
even more interesting, as imapd has to stick around to manage each
persistent imap connection, so an imapd server has umpty zillion
instances of imapd.  I don't know exactly what smtp daemons do --
postfix or sendmail.

Anyway, some generic forking daemon code, adopted IIRC from Stevens
originally and hacked around some to avoid TIME_WAIT and so on:

  server_fd = socket(AF_INET,SOCK_STREAM,0);
  if (server_fd < 0){
    fprintf(stderr,"socket: %.100s", strerror(errno));
    exit(1);
  }

  /*
   * Set socket options.  We try to make the port reusable and have it
   * close as fast as possible without waiting in unnecessary wait states
   * on close.
   */
  setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, (void *)&on,
                  sizeof(on));

  linger.l_onoff = 1;    /* Linger for just a bit */
  linger.l_linger = 0;   /* do NOT linger -- exit and discard data. */
  setsockopt(server_fd, SOL_SOCKET, SO_LINGER, (void *)&linger,
                  sizeof(linger));
  serverlen = sizeof(serverINETaddress);
  bzero( (char*) &serverINETaddress,serverlen);  /* clear structure */
  serverINETaddress.sin_family = AF_INET;        /* Internet domain */
  serverINETaddress.sin_addr.s_addr = htonl(INADDR_ANY); /* Accept all */
  serverINETaddress.sin_port = htons(port); /* Server port number */
  serverSockAddrPtr = (struct sockaddr*) &serverINETaddress;

  /*
   * Bind the socket to the desired port.  Try up to six times (30sec) IF the
   * port is in use
   */
  retries = 6;
  errno = 0;     /* To zero any possible garbage value */
  while(retries--){
    if(bind(server_fd,serverSockAddrPtr,serverlen) < 0) {
      if(errno != EADDRINUSE){
        close(server_fd);
        fprintf(stderr,"bind: %.100s\n", strerror(errno));
        fprintf(stderr,"socket bind to port %d failed: %d.\n", port,errno);
        exit(255);
      }
    } else break;
    /* printf("Got no port: %s\n",strerror(errno)); */
    sleep(5);
  }
  if(errno){
    if(errno == EADDRINUSE){
      fprintf(stderr,"Timeout (tried to bind six times five seconds apart)\n");
    }
    close(server_fd);
    fprintf(stderr,"bind to port %d failed: %.100s\n",port,strerror(errno));
    exit(0);
   }

  /*
   * Socket exists.  Service it.  Queue up to n_connxns incoming connections
   * or die. Default 10 matches the limits in the default xinetd.
   */
  if(listen(server_fd,nconnxns) < 0){
    fprintf(stderr,"listen: %.100s", strerror(errno));
    exit(255);
  }

  /* Arrange SIGCHLD to be caught. */
  signal(SIGCHLD, sigchld_handler);

  /*
   * Initialize client structures.
   */
  clientlen = sizeof(clientINETaddress);
  clientSockAddrPtr = (struct sockaddr*) &clientINETaddress;

  /*
   * Loop "forever", or until daemon crashes or is killed with a signal.
   */
  while(1){
    /* Accept a client connection */
    if((verbose == D_ALL) || (verbose == D_DAEMON)){
      printf("D_DAEMON: Accepting Client connection...\n");
    }

    /*
     * Wait in select until there is a connection. Presumably this is
     * more efficient than just blocking on the accept
     */
    FD_ZERO(&fdset);
    FD_SET(server_fd, &fdset);
    ret = select(server_fd + 1, &fdset, NULL, NULL, NULL);
    if (ret < 0 || !FD_ISSET(server_fd, &fdset)) {
      if (errno == EINTR)
        continue;
      fprintf(stderr,"select: %.100s", strerror(errno));
      continue;
    }

    /*
     * A call is waiting.  Accept it.
     */
    client_fd = accept(server_fd,clientSockAddrPtr,&clientlen);
    if (client_fd < 0){
      if (errno == EINTR)
        continue;
      fprintf(stderr,"accept: %.100s", strerror(errno));
      continue;
    }
    if((verbose == D_ALL) || (verbose == D_DAEMON)){
      printf("D_DAEMON:                ...client connection made.\n");
    }

    /*
     * IF I GET HERE...
     * ...I'm a real daemon.  I therefore fork and have the child process
     * the connection.  The parent continues listening and can service
     * multiple connections in parallel.
     */

    /*
     * CHILD.  Close the listening (server) socket, and start using the
     * accepted (client) socket.  We break out of the (infinite) loop to
     * handle the connection.
     */
    if ((pid = fork()) == 0){
      close(server_fd);
      break;
    }

    /*
     * PARENT.  Stay in the loop.  Close the client socket (it's the child's)
     * but leave the server socket open.
     */
    if (pid < 0)
      fprintf(stderr,"fork: %.100s", strerror(errno));
    else
      if((verbose == D_ALL) || (verbose == D_DAEMON)){
        printf("D_DAEMON: Forked child %d to handle socket %d.\n", pid,client_fd);
      }
    close(client_fd);

  }

  /* No need to wait for children -- I'm the child */
  signal(SIGCHLD, SIG_DFL);
  /* Dissociate from calling process group and control terminal */
  setsid();


>
>> You can see this by running e.g. netstat -a.
>
> I seriously doubt that you have seen such a behaviour. Empirical evidence 
> which might pass easier than theoretical one: on the e-mail server that I 
> admin, there is an iptable rule to only allow incoming connections to port 25 
> - if connections would suddenly be migrated to different ports they would be 
> blocked and I would not receive any e-mails from this list. But I do, 
> especially during the past few days... (not that I complain :-))
>
>

-- 
Robert G. Brown                            Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977