[Beowulf] Newbie

[Robert G. Brown]

On Thu, 5 Jan 2006, Leif Nixon wrote:

>> However, if any account is compromised by any means whatsoever, you're
>> equally screwed regardless of how you authenticate at the shell level.
>
> Kerberos-style security can give you a certain level of extra
> protection, depending on the circumstances, so there *are* different
> shades of screwedness.

???

Teach me this.

My understanding of kerberos is that it is an ungainly and complex PITA
that was developed historically to do poorly what ssh now does well, at
the expense of annoying the hell out of the user and sysadmin alike.
Most people who do end up using it (because it is required in order to
e.g. access fermilab systems) or managing it, that I know of, end up
hating it just a little bit somewhere along the way, even when they are
in an evironment (e.g. one that requires "kerberized" application
authentication without granting shell access) where it DOES have enough
advantage to make it worth the hassle.  In most places it is used, users
can actually access a remote shell (rlogin) with a kerberos ticket
granted on the basis of entering a (potentially trapped) password in a
shell so that it REALLY has no advantage with respect to ssh (and has
numerous disadvantages).  The only way I know of to avoid shell-based
password traps is to use e.g. a SecureID smartcard or other
one-time/real-time password generating systems.

Is this an incorrect view?

That is (rant aside:-), if things are set up so that a user on system A
has login privileges on system B, AFAIK a compromised shell (on A) under
kerberos provides a clever cracker with access to the user's kinit
password, and hence ability to obtain kerberos credentials and a shell,
on system B.  So it is just as vulnerable to password TRAPS (not snoops,
blocked by bidirectional encryption under both ssh and kerberos although
legend has it that a really good snoop can often obtain enough
information just from the packet-rhythm with which you type a password
to be able to significantly reduce the search space) as ssh, and the
trapped password still provides the cracker with just as much access to
resources on B as the user has.  If this doesn't include a shell on B
then yes you protect B to some extent, but there are MANY ways of
controlling the USER'S access to resources on servers and clients
(kerberos being one of them, others based on ssh and/or ssl and/or sudo
possible as well) but the degree of toplevel control implied by
"kerberization" is expensive -- costly to develop and manage and always
accompanied by a reduction in freedom and productivity for most users
thereafter, certainly in a free (e.g. development) environment.  If your
work requires shell-level access to a grid-style cluster of nodes or
workstations in a NOW configuration (as opposed to e.g. a real
scyld/bproc beowulf where there are no node shells to be had at any
price) then kerberos is precisely as secure as ssh.

So, school me, whomp me upside the head if this is wrong.  I can take
it...:-)

> In general, you need to think long and hard about the trust domains
> within a cluster. Adopting the view "This cluster is a single big
> machine. We don't need no steeking internal security barriers" is a
> bad idea; you want to contain intrusions as much as possible. Limit
> the ways root can ssh within the cluster, export your filesystems
> ro/nosuid as far as possible, disable user login on all machines they
> don't need to login on...

All of which is good advise and standard of practice on either cluster
or standard Unix LAN, really, since architecturally MOST cluster designs
are at heart Unix LANs.

However, I think that just WHAT internal security barriers are
appropriate depends very much on the details of the cluster
architecture.  A scyld/bproc cluster where the "head node" is de-facto a
network firewall and where one cannot GET a shell on the nodes IS "a
single big machine" architecturally, and the question of node-level
resource/shell access is irrelevant. A grid-like cluster design that is
behind some sort of firewall but that does permit node shells would
almost certainly be set up with either passwordless rsh or ssh -- host
authentication is irrelevant as all the nodes can talk to are other
nodes, when a cracker has busted one node they've busted them all but
who cares?  The nodes are likely either automagically (re)installable or
diskless and a reboot away from being pristine once the intrusion is
discovered and the toplevel point of cracker access repaired.

Even a NOW grid design, a cluster that is flat to or made up of a
(firewalled) workstation LAN, would likely be set up with internal ssh
permitted with or without a password as the user desires (most likely
without).  This is because there is no marginal increase in security
provided by requiring the password to by typed in every time you ssh
between your desktop and some other workstation in the LAN including the
nodes (in this architecture rsh is definitely Evil).  The security
barriers are typically between the inside and outside of the LAN, and
between LAN clients and LAN servers.  Note that in most LAN cases your
home directory and various work/project directories are mounted across
all nodes/workstations, so if a user's account is cracked on any system
in the LAN, between traps and owning write access to their home
directory it is cracked on all of them within seconds to a day or two at
most.

So I'm not a big fan of toplevel "requirements" that a user type in
their password every time they access any LAN internal resource, and see
little advantage provided by kinit tickets relative to just the fact
that you were granted a shell in the first place (in both cases on the
basis of entering a shell-trappable password).

ROOT, on the other hand, should be configured just as you suggest -- no
passwordless access across systems, DIFFERENT root passwords on clients
and servers, no user access to servers period, well-protected and
carefully monitored servers (and clients) since if a cracker gets root
on a CLIENT they can su to and trojan YOUR account and very likely get
root on server whether or not NFS is ro and no root access.  Here your
only defense (as noted) is being a raving paranoid that (for example)
actually READS those little "last login at..." notices (caught somebody
cracking MY account that way, many years ago:-), reads all the log
records in real-time of who is given a root shell for any reason
whatsoever on all systems under your control and knowing enough to be
able to detect anomalies, trojans their own account in such a way that a
cracker immediately reveals themselves because they don't know the
secret handshake.

I'm not convinced, mind you, that there is THAT much of a security
advantage to requiring root to type a password every time root ssh's
between server and client systems (as root) or between client systems --
in a strongly firewalled environment it certainly makes things more
convenient.  Mostly what requiring a password to be typed does is slow a
cracker down (possibly by as little as a few seconds, but possibly by a
few days) to give you a bit more time to detect the intrusion by using
your sense of smell before they make root on one of your primary
servers, at the expense of slowing down all the work that you do as
root.

Security is always a cost-benefit problem.  Administrative fascism at
one extreme can be almost as expensive as no control at all at the other
extreme.  In between there is an optimum, one that generally permits
relatively "free" access to resources that permit work to be done while
putting armed guards and dogs between most users and the mission
critical servers upon which the entire environment depends.  Beyond
that, keep good backups, remain vigilant, and use one of the
automagically (re)installable methods to actually install clients.  If
you protect your servers (typically "difficult" to rebuild in the event
of a successful crack) one can handle most simple cracks by just
shutting off WAN access, flashing a new system image to the compromised
client(s), patching the problem (via e.g. yum update, no hands required)
and restoring WAN access.  Truly miraculous compared to the "old days"
of a whole decade ago.

Ubercrackers, of course, are another matter altogether now and always.
A real ubercracker is invisible to all but the MOST sensitive and
paranoid and genius hyperguru sysadmin.  I'm aware of one ubercracker
many years ago that took over systems in our CS department who was so
good that they only way they could tell that he was there was to have a
passive host on the network monitoring traffic directed to the host.
>>Everything<< on the system that otherwise might have revealed the
individual's existence was otherwise compromised -- the logs revealed
nothing, the kernel tools lied, the network tools told funny stories,
checksum tools returned the same old results as always.  Quid custodes
custode with a vengeance.  Some part of the theoretical capacity of the
system went away (it ran a bit slower) and some network/disk traffic was
observable when it shouldn't have been there, but otherwise -- no hint,
perfect invisibility.  At the time, the CS sysadmin was, in fact, an
uberguru who DID notice the anomaly and WAS suspicious and tenacious so
they DID eventually demonstrate the ubercracker's existence, did their
best to track him down (working through several layers of breakout) and
eventually flushed him from their network (still uncaught, IIRC) and
reinstalled clean patched systems, but...

The moral of this particular story is that you can never be quite
CERTAIN that your system(s) haven't already been cracked, even if you
are very good, unless you REALLY work at it.  If they are running a bit
slow, if memory appears to be leaking and you can't figure out why, if
things go bump in the night when the systems are presumably idle, if
nmap shows a mysterious port open on 17832 tcp (but only if you run it
to check between 12 a.m. and 12:30 a.m.) -- well, this is WHY guru level
sysadmins become raving paranoids.  They spend a fair bit of time
running down little anomalies that could be bugs, could be nothing, or
could be the critical piece of evidence that their lunch has been eaten
and their life is about to become "interesting" for a while.  On
numerous occasions (maybe a half dozen or more -- I've led an evil
life:-), this is how I personally have discovered intrusions on systems
we run -- running down things that just "weren't quite right" and
fortunately not encountering an ubercracker with better tools and
knowledge of the system than my own.

This kind of proactive vigilance is one part of the "cost" of security,
as those same sysadmins COULD be playing WoW or helping users or
developing applications, their blood pressure COULD be lower and their
metabolic systems not suffering from the long-run effects of catabolic
oxidants produced by chronic low grade stress.

Sorry, time to go read all of my system logs and nmap all the hosts
again...;-)

     rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu