[Beowulf] Intra-cluster security

[Reuti]

Hi,

Am 11.09.2009 um 21:39 schrieb Stuart Barkley:

> We are working with a couple small clusters (6-8 nodes) and will soon
> be working with some much larger cluster/supercomputer systems.  We
> are currently using SGE 6.2 for job queuing.  We use kerberos for
> authentication and ssh for system access.
>
> What are peoples thoughts about secure communications between the
> nodes of a cluster?  I see a cluster as a single computational
> resource and would like to see flexibility of communications between
> the nodes of the cluster.
>
> There seem to be a couple of approaches:
>
> - Old style rsh/rlogin.  Not acceptable for me.

I wouldn't be concerned by this per se, but I also disabled it (in / 
etc/xinetd.d/rsh) because of another reason:

- I setup ssh hostbased authentication, but limit it to admin staff  
with "AllowGroups admin". The ssh-keysign has to be set suid as you  
mention below, I can also send you a rough outline of the necessary  
steps.

- Users can login to any node by using an interactive job in SGE. For  
this special interactive queue I set h_cpu=60, hence they can't abuse  
it. SGE in turn will either use:

a) traditional rsh, but on a random port selected by SGE (this I have  
right now)

b) a "builtin" method in the newer versions of SGE

c) plain ssh (but this will lack correct accounting), directed to a  
special sshd_config (because of the AllowGroups rule in the default one)

d) ssh with a recompiled SGE with -tight-ssh flag for correct accounting

If you have more than one queue in addition to this interactive queue  
on each system, you can't limit the maximum number of slots any  
longer in the exechost definition (as than the interactive queue to  
peek around would also be taken into account), but set it up in an  
RQS (resource quota set) like:

limit        queues !login.q hosts {@dualquad} to slots=8

-- Reuti


PS: I saw on Debian, that their "su" will not only set the user, but  
also remove any imposed cpu time limit by making an su to oneself.  
For the SGE queue to impose the h_cpu limit this must be disabled  
then. I don't know, whether this is still the Debian default.