Scyld Limitations?

[Art Edwards]

I am using Scyld beowulf on a cluster of single-processor AMD boxes, all with 
relatively large (20 G) local hard drives. I have had help from several people 
on this list to get files written to local disk from MPI jobs and help on 
running jobs completly off the head node. Here are some observations that I 
would like either contradicted or confirmed. 

1. When running a job from the head node so that computation takes place on 
the head node as if it were node 0, one can indeed write files on the local 
disk of the slave nodes in /tmp. However, one cannot access subdirectories 
under /tmp. This is a mild annoyance, but not really crucial. 

2. When attempting to run using one of the slave nodes as node 0, using a 
p4pg file, one cannot actually open and write files on the other slave nodes 
from the running application. One can from a shell script using bpsh. If
this is true, it is a real killer. It means that you have essentially a 
single-user system unless you populate the head node with several, competing, 
computational jobs. 

3. Bash shell scripts can be deeply altered by embedded bpsh calls. In one 
case I had a while-loop that read file names from a file, call it infiles, and 
then attempted to write these to the various /tmp files. After the first file
name, it seemed that infiles had been closed or simply forgotten. This is 
another inconvenience but not a killing deficiency. 

Art Edwards