[Beowulf] Intra-cluster security

[Joe Landman]

Leif Nixon wrote:
> Joe Landman <landman at scalableinformatics.com> writes:
> 
>> I won't fisk this, other than to note most of the exploits we have
>> cleaned up for our customers, have been windows based attack vectors.
>> Contrary to the implication here, the ssh-key attack vector, while a
>> risk, isn't nearly as dangerous as others, in active use, out there.
> 
> I'm really hoping you aren't accusing me of security theatre.

Nope.  I thought I made it clear that I wasn't (and if not, then let me 
re-iterate that I am not accusing you of this).

I am noting that the there may be something of an overhyping of this 
vulnerability from where we sit.  YMMV.

> This may be a case of differences between user communitites - while I
> have seen one or maybe two cases where windows-related attacks were

Likely it is a difference.  Most attacks we see are windows related, 
exploiting the inherent weakness of that platform, and is relative ease 
of compromise in order to compromise harder to take down systems.  Why 
break through the heavily fortified door when the window (pun 
un-intended) is so easy to crack?  This is the nature (outside of 
incessant ssh probes) of all of the exploits we have seen be successful 
at our customers sites.

> involved, I have seen dozens and dozens of cases where ssh key theft was
> involved. I have a blacklist of literally hundreds of stolen ssh keys
> from a very large number of sites, and I dearly miss a key revocation
> mechanism in ssh.
> 
> We try to educate our users to use either a good strong password or to
> use ssh keys together with the ssh agent and agent forwarding, so that
> the private key never needs to leave the user's personal workstation.

We have started hearing about malware infected USB dongles.  If you have 
a password equivalent stored on your workstation ... it is at risk.

> 
>> Fake security, aka security theatre (c.f.
>> http://en.wikipedia.org/wiki/Security_theater ) are things you get
>> when people want to seem like they are doing something, even if the
>> thing doesn't help, or worse, gives you a false sense of security. See
>> every anti-virus/anti-phishing package out there for windows. If you
>> think you are safe because you are running them, you are sadly
>> mistaken.
> 
> And on our side of the fence, we get things like Trusted IRIX, with a
> really elaborate, checkbox-compliant permissions system. Of course,
> since it was built on IRIX, any serious attacker would cut through it
> like a hot knife through molten butter, but there obviously wasn't a
> checkbox for that.

Trusted computing, trusted Irix, etc. are examples of what I am talking 
about.  You have a sense of security.  Whether its warranted or not is a 
completely separate question.

Most of our users are companies, research universities, etc.  We hear 
horror stories from admins on compromises.  We do get an occasional call 
from a customer, wondering how a system behind a firewall could be 
compromised (remember that theatre and false sense of security?). 
Forensic examination showed us the path in, happily riding along the 
same connection that the user had, grabbing their keystrokes, and 
replaying them.  Installing bits, and attempting rootkits.

I have a nice little collection of rootkit detritus and dejecta, as well 
as logs of what the cracker attempted, all while getting in via the same 
compromised machine the legitimate user logged in to.

It didn't really get bad ... until the user typed the root password in.

No, wasn't bad until then, most of the defenses held.

Their cluster, they have root.  We tried warning them that there was no 
conceivable scenario in which they ever needed to be root.

We were ignored.

Their IT staff was none too pleased.

I wrote up a whole series of posts on it, detailing everything (apart 
from the victims name/id/location/university) so that some others could 
learn and protect themselves.  My descriptions managed to get me ... 
moderated ... by someone who claimed I was being alarmist ... for 
posting the gory details and making suggestions to the same community on 
how to avoid it.

I am simply saying that what we see may be different, and that I hear 
far too much "one-size-fits-all" security prescriptions, that often fail 
to deter attacks, and provide what I think is a false sense of security 
if you follow that and ignore the other issues.  I see to much of "if we 
install a firewall, we will be secure" mindset running about.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615