[Beowulf] High Performance SSH/SCP

[Robert G. Brown]

On Thu, 14 Feb 2008, Mark Hahn wrote:

>> Hopefully the PSC folks (or other parties) can convince the OpenSSH
>> maintainers to allow some options (dynamic buffers, disabled encryption
>> of the non-handshake payload, etc.) that result in higher-bandwidth
>> transfers.
>
> the switch-to-none feature is a good one.  I'm a bit skeptical about
> the buffering changes, since you can accomplish the same thing with sysctls. 
> (has anyone ever experienced a real case where too-large sysctl net memory 
> settings caused problems?  obviously, attempting to do long-fat-pipe 
> transfers to a heavily used web server might be a problem, since the latter 
> wants tight controls on sock mem use.)

I think it is hard to improve over the TCP defaults for messages unless
you know something very specific about the message types -- its pretty
easy to rob peter (latency) and pay paul (bandwidth) or vv.  So I'm not
really that excited about the buffering changes either, although if
somebody were to demonstrate that they are a uniform win for (say) 90%
of all traffic patterns and not too big a loss for the losers I could be
convinced.

The main reason I think no encryption is a sane option is the usual one
-- encryption is expensive, on both ends, and it can consume BOTH CPU
AND network in lots of cases -- encryption can easily expand a message
size and in either event requires an actual computation on both ends to
recover the message.  I think being able to turn it off, iff enabled in
/etc/ssh/sshd.config by the site sysadmin, is a very sane thing to want
to be able to do in problems where the cost-benefit of encryption is
poor (that is, unimportant messages that you don't care if the world
reads).  I'm assuming a sane sysadmin would only turn on the feature for
e.g. cluster nodes in a fairly well controlled network anyway -- I can't
see people turning it on on WAN-accessible clients too often.

Still, for many parallel applications it doesn't really matter.  The
"real" IPCs are managed by e.g. PVM or MPI or a socket you completely
control on both ends.  ssh is mostly used just to securely fork off
tasks or daemons across cluster nodes and then goes away and no longer
is used for communications.  With an exception for little demo shorty
parallel programs like my venerable "taskmaster" perl script intended to
just show one way of doing parallel work for EP tasks.

What the openssh people don't seem to "get" is that by FORCING people to
use encryption, they are actually keeping rsh alive and a potential
security risk for all sorts of people in the cluster business for whom
performance is more important than security given their networking
environment and goals.  Otherwise, who would ever install it?

They are not, in other words, increasing the net security of the linux
universe, because ssh with a secure handshake only is WAY more secure
and environmentally friendly (in the sense that one can pass an
environment) than rsh on its best day.  It's not so easy any more to
hijack an established TCP connection.  With rsh for "security", what can
one do?  No password, anybody can spoof in as you on the wire.  With
password, anyone can snoop the wire.  The only security possible is
keeping everybody off the wire, which sucks and isn't always possible.
So they'd make lots of people very happy if they would actually give
cluster people a WAY to make rsh finally go away.

    rgb

-- 
Robert G. Brown                            Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977