[Beowulf] Re: "hobbyists"

[Perry E. Metzger]

"Robert G. Brown" <rgb at phy.duke.edu> writes:
>> Yes, Run a box with sshd on it connected to the internet and watch your
>> logs for a few days. You will find numerous attempts to try thousands
>> of possible account names and passwords -- brute force cracking.
>
> Well, yeah, sure, I know about that as I DO watch my logs -- I just
> haven't heard of one of these attacks SUCCEEDING in pretty much forever,
> for obvious reasons.

The worms that do that attack self-spread, so the fact that you're
seeing it in your logs means that they've succeeded.

> These attacks run through a list of "likely user names" (including well
> known system names like "root") and then run through a short list of
> "likely passwords" such as user=test password=test.  Short because ssh
> enforces a roughly 1 second interval between tries, which limits the
> total number of attempts that can be made to < 100,000/day,

That limits the number of attempts that may be made against your
particular machine. At the same time that they're attacking your
machine, that one instance is attacking a vast number of other
randomly selected boxes. There are also a vast number of the things
running out there, so in the long run, they succeed quite a bit of the
time.

There are many other similar sorts of things out there that you are
less aware of -- things knob-twisting on ebay, yahoo and gmail
accounts (I would explain why people want to steal something that's
free but its a long story), online banking, etc.

The deal is, this is a sort of biological ecosystem. Every possible
niche is filled. If you can imagine an attack, someone out there is
doing it at this point.

> In fact, to me it looks like most of the attacks are generated by
> more or less the same code and probably reuse most of the same names
> and password guesses

Don't be overly confident because of how primitive such things
seem. They succeed constantly. As I noted, if they didn't, you
wouldn't see them in your logs all the time.

> Sure, it goes on and on.  I don't really LIKE seeing this, especially on
> a server with sensitive information, but that is precisely why one
> configures such servers with tight controls and runs a password checker.

I don't run a password checker. I simply have no mechanisms in use
that *use* passwords, so it is irrelevant. Then again, I'm far more
paranoid than most people are. Professional hazard.

>> These attacks are done by automated malware that spreads itself around
>> from machine to machine for nefarious purposes -- good luck trying to
>> track down the puppet masters. I've tracked the bad guys down a few
>> times but they're always somewhere like Bucharest anyway, and the
>> locals don't care to arrest them.
>
> Agreed.  All part of the Russian Diabolical Supercomputer we discussed
> six months ago or so -- a cloud of unknown proportions and evil
> intent:-)

A few times, friends of mine have suggested I should rent botnet time
to run computational chemistry problems. It would certainly be vastly
cheaper than my own clusters, but unfortunately it is also against my
morals.

>> It is true that this is only one of many modern attack vectors and
>> that buffer overflows, drive by malware downloads into browsers, etc.,
>> are all far more common ways in, but you will indeed get hacked by
>> automated systems if you leave an sshd on and have accounts with weak
>> passwords.
>
> Agreed as well.  Which is why professionals generally check for weak
> passwords (as do most of the password tools nowadays).

Nah. Professionals don't even use passwords any more, as I said. (I'm
forced to use them on most e-Commerce sites because just about no one
does client cert based authentication, but my machines don't use
passwords. Most of my buddies machines don't use them either.)

> As I said, this sort of attack only checks for weak combinations in
> a tiny, tiny fraction of the phase space, and usually they only run
> a their dictionary once (or for at most a day or two) and then move
> on.  So many IP numbers to try to crack, so little time.

Any given instance only tries a small random subset of the phase space
at a time on a given machine. However, the worms are pretty good about
picking random subsets. Over time, you will find they slowly will find
most of the more weakly defended boxes, just like water wearing down a
stone.

> BTW, one can prevent nearly all of the attacks like the ones above just
> by running sshd on a nonstandard port.

People do all sorts of things -- port knocking, filters that
autoblacklist IPs that have too many errors, etc. It is simpler and
safer just not to use passwords.

> They are almost never driven by a real portscanner-driven attack,

Only because they're not attacking you in particular. If someone had
reason to attack you in particular, then you would have more to worry
about. Again, I find it is better just to make sure that there is
nothing to attack.

> Moving the port also reduces the network load on your servers by a tiny
> amount; it costs resources just to say nope, no, not here, wrong
> username, sorry, try again later 10,000 or so times a day, and it clogs
> up log monitoring programs and /var/log as well.

I have professional reasons to want to know that this sort of thing is
happening. If it irritates you try sshd_sentry, sshd-blacklist, or any
one of fifty other little scripts that will stop the kneebiters.

>>> There are also still -- relatively rarely -- buffer overwrite attacks
>>> discovered.
>>
>> Rarely? You haven't been reading full-disclosure lately I see. There
>> are a half dozen new such vulns found a day.
>
> Not like in the old days, when lpd nearly always had one, syslogd had
> one for years, etc.

No, really, it is still like the old days. lpd and syslogd probably
still have plenty (just read the code), but nowadays no one smart runs
them exposed on the net. Syslogd has long since had patches added so
that it doesn't need to listen to the net by default. Does that mean
the syslogd code base is great now? No.

Remember when X servers ran listening to all comers? One of my
employees, at my request, was the one who contributed the "-nolisten
tcp" patch to the X consortium that is now more or less the default on
all installations. Does that mean that X has no buffer overflows?
Certainly not. It just means that they're not exploitable by arbitrary
people these days so it isn't a big target.

It is, to some extent, a question of how many people are interested in
a particular attack vector. Internet Explorer is a major attack vector
for people who make money at this, so they work hard finding the bugs
in it, of which there are an apparent endless number. I believe that
more than 250 days last year, Internet Explorer had a known but as yet
unpatched vulnerability. That's why the overwhelming majority of
Windows boxes are zombies, including almost certainly most of yours
unless you are a really unusual sysadmin.

I encourage trying to read full-disclosure, bugtraq, and the
like. You'll get depressed very fast.

Here is just the May 1 traffic on Bugtraq, cut and pasted from the
mailing list archive (http://seclists.org/bugtraq/2008/May/):

# XSS in AstroCam Steffen Wendzel (May 01 2008)
# iDefense Security Advisory 04.30.08: Akamai Download Manager Arbitrary Program Execution Vulnerability iDefense Labs (May 01 2008)
# [SECURITY] [DSA 1564-1] New wordpress packages fix several vulnerabilities Thijs Kinkhorst (May 01 2008)
# Team SHATTER Security Advisory: Oracle Database Buffer Overflow in SYS.DBMS_AQJMS_INTERNAL (DB15) Team SHATTER (May 01 2008)
# mjguest 6.7 (ALL VERSION) Xss & Redirection Vuln irancrash_at_gmail.com (May 01 2008)
# vlBook 1.21 (ALL VERSION) irancrash_at_gmail.com (May 01 2008)
# Team SHATTER Security Advisory: Oracle Database SQL Injection in SYS.DBMS_CDC_UTILITY.LOCK_CHANGE_SET (DB02) Team SHATTER (May 01 2008)
# Team SHATTER Security Advisory: Oracle Database Buffer Overflow in SYS.KUPF$FILE_INT.GET_FULL_FILENAME (DB11) Team SHATTER (May 01 2008)
# [SECURITY] [DSA 1565-1] New Linux 2.6.18 packages fix several vulnerabilities dann frazier (May 01 2008)
# php-addressbook v2.0 Multiple Remote Vulnerabilities (LFI/XSS) irancrash_at_gmail.com (May 01 2008)
# Re: netOffice Dwins 1.3 Remote code execution. luiswang_at_user.sourceforge.net (May 01 2008)
# BlackBook v1.0 Multiple XSS Vulnerabilities irancrash_at_gmail.com (May 01 2008) 

A large number of new vulnerabilities a day, half a dozen or more of
which are always new buffer overflow exploits.

> The core daemons run by a standard linux box get fairly strongly
> scrutinized.

If you're smart, you're listening on:

* DNS, with bind configured to run chrooted and unprivileged
* sshd running with priv sep
* ntpd running chrooted and unprived (though not all OSes will allow
  you to do that.)
* maybe SMTP via postfix, which runs chrooted and unprived
* and NOTHING ELSE.

And if you're really smart, those daemons are further tied down with
various bondage and discipline equipment like apparmor or SE Linux or
what have you.

Everything you listen on is another attack vector that will be
exploited one day.

> That doesn't mean that there aren't any, only that ones in tools
> that are at all likely to be managing a public port are discovered
> relatively infrequently and patched very quickly,

I think that the real trick is that most machines aren't paying
attention to very many ports.

> I'm not trying to minimize the threat posed by buffer overwrite attacks,
> BTW.  They are very serious, obviously.  However, again, the risk is
> very, very significantly mitigated by tools such as yum or apt.

If you're lucky. The problem is that we find out about most of these
things after someone is already mass exploiting it. If you're smart,
you'll make sure that it is unlikely anyone can exploit your boxes in
the first place by being exceptionally mean about how you run the very
few things you run. There is doubtless some way to exploit bind
if it is running chrooted, unprived and with some sort of ACLs
preventing use of unneeded syscalls, but it is much less likely that
you'll be the one that gets hit if you do all of that when many people
don't.

> In the linux community it isn't that uncommon for an exploit to be
> discovered in some important app or daemon Wednesday morning, for it
> to be closed up by Wednesday afternoon, for the patched code to be
> dropped into the major toplevel repos by Wednesday evening, to be
> mirrored overnight and automatically updated on user desktops
> between Thursday and Saturday morning depending on your distance
> down the distribution/repo tree.  Not a lot of room for traction
> there, either.

Not true. At this point, there are automated attack engines that can
have any new zero-day exploit dropped in and will be on hundreds of
thousands of boxes the same hour they're found. Really. 24 hours is
too long.

If you don't believe me on all of this, I suggest reading the
literature. It is very grim. There is also time to register for
Usenix Security if you would like to hear a very long parade of
exceptionally depressing papers being presented about a month from
now.

>>> But weak passwords that are brute force guessed[...]?
>>> Only on a poorly managed network,
>>
>> That would be 95% of networks. I've done a lot of network audits in my
>> day, too.
>
> Well, Duke's are pretty good, or the people being cracked are deeply
> ashamed and I'm not hearing of them.

Most people who get cracked never even know, so why would they say
anything?

University networks are among the worst -- they're filled with boxes
with no proper management. Even Jeff Schiller at MIT will cop to his
net being filled with exploited boxes and that's a place where most of
the infrastructure is hardened by world class experts.

If you really believe your local net is very good, run a sniffer on it
for a while -- or talk to someone who's job is to run one.

> So I'm not sure I'd agree with your 95% of all networks are poorly
> managed -- that is a rather large and possibly hyperbolic assignment
> of incompetence, after all, more rhetoric than substance.

If you want to believe that, sure.

Most networks are in small offices where there isn't even a
professional sysadmin. The average network has no one associated with
it who knows anything much about computers at all, and was set up by a
consultant who barely knews anything either. Think about what your
dentist's office or the network at a small town city hall is like. 95%
is being generous. It is probably higher.

> However, my experience is far from universal,

I've done computer security consulting for most of the last couple of
decades. (So why am I on this list, you ask? Well, a very long story,
but for the last few years I've been much more interested in
scientific research than in my putative day job, which I've scaled
back to only about a third of my time. Having a computer science
background has enormous pluses when you find yourself doing scientific
computing in mid life...)

> We actually have had a really competent "security czar" on campus for a
> decade or so as well as good sysadmin groups and a few really bright
> lights to lead them, so the mentorship here is perhaps better than
> average.  That's what really makes the difference in the long run.

Your "security czar" probably spends 2/3rds of her time horribly
depressed at the futility of the whole thing. Most of the people in
that job that I know are very very depressed about how well it can be
done in a world where the average box is a zombie.


-- 
Perry E. Metzger		perry at piermont.com