[Beowulf] Dealing with masquerade attacks (Was: CLuster - Mpich - tstmachines - Heeelp !!!!!!!!)

[Joe Landman]

Leif Nixon wrote:
> Mark Hahn <hahn at physics.mcmaster.ca> writes:
> 
>> this is wandering pretty far afield.  a cluster, to my way of thinking,
>> is intended to act as a single resource, and as such is a single trust
>> domain.
> 
> I used to think that, as well. However, expensively bought experience
> has taught me otherwise.
> 
> Events the last two years [1] have shown that if you have a cluster

s/cluster/any system whatsoever/

We have logs from 1997 onwards (ok, we purged the old ones) that get 
progressively scarier.

> that is somehow reachable from the Internet there is a non-negligible
> risk that an intruder at some point will log in on it using stolen
> credentials. I know for a fact that a large fraction of Swedish
> academic clusters have had such visits.

As have a fair number of academic sites (and industrial sites).  We have 
seen everything from sniffed ftp/telnet passwords (early days), through 
brute force cracking of passwords, as well as capturing "secure keys" 
from USB fobs which have been a fad of a well meaning group of admins 
recently.

> You cannot trust your users, because that user over there might
> actually be a pimply-faced kid holding a freshly stolen password in
> his sweaty palms.

Or worse, they may do something dumb.  Like give out their passwords.

> I don't see the world doing away with password or private-key-on-disk
> authentication any time soon, so this problem is here to stay, I'm
> afraid. We have to learn to live with it.

Or private key on a USB fob, or key-loggers, or, ...

> In general, we have to acknowledge that our security will always be
> slightly broken. This means that you can't put all your effort and
> trust in a perimeter style defense, because it will never be perfect
> and one day somebody will penetrate it. You need defense-in-depth.
> (That old, worn phrase)

True.  At the same time, your layers need to make sense.  I don't want 
to light off an argument here on password strength, but various 
sniffing/cracking tools won't be affected by passwords of effectively 
random characters and relatively short length.  Even longer passphrases 
are sniffable with keyloggers.

You need tools which can withstand a sniffing/logging/interception 
attack and allow you to recover.  A One-Time-Password-In-Everything 
(http://en.wikipedia.org/wiki/One-time_password) method is probably 
advisable for most internet facing systems.  They are two factor 
methods, and if done right, are quite a bit more secure for 
authentication purposes than passphrase based systems alone.

Fundamentally the statement of depth of security, while cliche, is quite 
correct.

> Which in this case boils down to: Yes, you do need internal security
> barriers in your cluster.

I would phrase it that you need to limit the damage one can do.  Assume 
they will compromise your system.  Don't worry about how, simply assume 
they get in as a normal user.  What can they do?

If you can containerize the users (VM their main login, limit access to 
file systems, have them work in a temporary directory, etc...) you would 
reduce the risk that a malicious login could compromise much.  If on 
each login you force them to use a two factor authentication (with a 
physical timed key device) to get at their files (disallow an automount 
or a FUSE based file system on a different machine), after using a two 
factor authentication to log in, you impose yet another limitation of 
what they can do.

This of course needs to be balanced against the users desire to gain 
easy access to their account and ease of use of the system.

In which case, the best security of the system comes from very good backups.

> As you note, hardening a cluster to untrusted external users "would
> take quite a bit of effort", but even when it would be unrealistic to
> go full-out virtualized and compartmentalized, you should still keep
> these issues in mind when designing a cluster.
> 
> Ask yourself, what happens if an intruder gets access to one of the
> machines in the cluster? It's very hard to totally stop the intrusion
> from spreading across the cluster, but you *can* make life harder for
> the intruder, which might just buy you enough time to detect the
> intrusion in its early stages.

Hmmm.  If the compute nodes are not nodes that can be logged into in the 
first place, simply processing elements to be used, then this is moot 
(ala Scyld).  If the nodes need to be logged into in order to be used 
you are constrained by the need to have automated login systems.

> So, for example, do you really need unlimited passwordless access
> across the entire cluster, or can you limit it in useful ways? Perhaps
> you can hook PAM up to PBS, so users only can access nodes they are
> scheduled on? Pay special attention to how root is allowed to access
> other machines. Export NFS filesystems read-only and mount them
> nosuid, unless you really need rw/suid. And, of course, never leave a

and don't forget root_squash.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615