RSH scaling problems...

[Robert G. Brown]

On Wed, 18 Dec 2002, Jesse Becker wrote:

> I did some testing on a local system, and come up with some times for 
> transfering a file via SSH using various ciphers.  The file in question 
> was generated via this command (which generated 525 megs of data):  
>    dd if=/dev/urandom of=./stuff bs=1M count=500
> 
> I used all of the ciphers below, repeated each test 10 times, and recorded 
> the results.  The systems were otherwise unused.  Both boxes are dual Xeon
> 2.4GHz boxes, 2GB of RAM, and a single UATA 40GB IBM Deskstar 120GXP 
> drive.  Both systems run Redhat 7.3, with 2.4.18-5 smp kernels.  There is 
> an Intel 1000/PRO 82544GC Gigabit NIC in each box (using the eepro100.c: 
> $Revision: 1.36 $ modules), and the switch is (IIRC) a Netgear GS516T 16 
> port gigabit switch.
> 
> Both boxes use the latest offical release of OpenSSH 3.1p1 from Redhat.
> 
>    Cipher             Transfer time (sec)         Avg.(sec)   Rate
> -------------   -----------------------------     ----        ----
> aes128-cbc:     20 18 21 20 21 20 18 20 20 19 ==> 19.7        26.6
> aes192-cbc:     22 21 22 22 22 21 22 21 19 20 ==> 21.2        24.7
> aes256-cbc:     23 23 22 24 23 23 23 22 23 23 ==> 22.9        22.9
> blowfish-cbc:   19 18 19 17 16 19 16 17 18 19 ==> 17.8        29.4
> cast128-cbc:    18 20 20 18 19 20 20 20 20 20 ==> 19.5        26.9
> arcfour:        19 19 19 17 19 18 16 19 18 18 ==> 18.2        28.8
> 3des-cbc:       44 46 45 45 46 44 45 43 45 45 ==> 44.8        11.7

Very useful.  In a way, it is a shame that the openssh folks completely
eliminated the -c none option.  If they made it something that had to be
explicitly enabled in ssh[,d]_config and that was officially
discouraged, it would have permitted one to authenticate passwords with
a proper secure handshake but still use ssh for rsh-comparable file
transfers in a cluster sandbox (presumed firewalled and internally
secure).  Still...

> I don't have rsh enabled anywhere, but copying the same files over an NFS 
> (v3) mount took about 11.7 seconds for 525MB, and 54 seconds for 2.1GB.  
> These are averages of 5 transfers, and it works out to about 44.8 and 39.8 
> Mb/sec respectively.

...so you've got wirespeed limitations of perhaps 100-100 MB/sec, and disk
speed limitations that are likely less than that.  So the speed penalty
is 2-3 in total speed for using ssh to transfer huge files.  This isn't
particularly terrible for most file transfer applications, unless all
you are doing all of the time is file transfer and it is significantly
rate-limiting.  

The ssh latency (time to make and break connections) is more
significantly greater -- as much as 10x the time required to make an rsh
connection, if I recall correctly from my earlier tests -- but you're
talking 10x a very small absolute number, leading to bigger but still
small number, negligible in MOST cases (and where it isn't negligible,
you probably shouldn't be using rsh OR ssh to manage the connection).

> > The only significant bitch that I have with ssh these days is that the
> > openssh designers viewed a feature of rsh -- the ability to remotely
> > initiate a process and then disconnect, leaving the backgrounded process
> 
> Have you tried using screen(1)?  It works very well with SSH, although 
> it's not as useful for the "fire-n-forget" method of running things.

<smile>I first used screen back in the late 80' or early 90's, IIRC, to
give me primitive VT100 "windowing" through a kermit/modem connection.
I haven't used it much since I got linux first running at home, since a
small stack of xterms and rsh (now ssh) was so much more efficient and
easy to rotate and manipulate.  

I also got addicted to the rsh feature that a named symlink to the
binary acted as a de facto command, e.g.

ln -s /usr/bin/rsh ganesh
ganesh

would execute "/usr/bin/rsh ganesh", making it REALLY easy to execute
commands on hosts -- create a directory with a list of hostname links on
your path, and every hostname becomes a remote command.  So I could just
type

ganesh worklikehell &

and make ganesh work like hell in the background;-)

Alas, the openssh folks in their wisdom doubly trashed this -- even when
I wrote a notty wrapper that explicitly disconnects from the control
terminal, forks, and execs to background a process, ssh won't allow you
to disconnect until all children of the initiating instance of the shell
are dead OR you bop it on the head with interactively typed escaped
disconnects, AND they eliminated the perfectly lovely and harmless
symlink feature.  The first is just plain fascist of the designers and
counterproductive to a huge fraction of the work ssh might be put to
doing -- in order to work around it I'd have to write an expect script
or really hack the source, the latter most risky in a key security
program.  I haven't gone the expect route, yet, but I may yet.

I have written a near perfect script-based wrapper that reproduces the
symlink trick -- even now I have a directory on my path with symlinks
(to the wrapper script) so that in order to ssh login to ganesh I just
type "ganesh".  Silly, but those keystrokes really add up when one does
a lot of bopping around... -- I suppose I should probably further hack
the script to add the expect feature, but alas it is NOT easy to do --
one has to escape once for EACH LEVEL of ssh one works through.  If I'm
at home, ssh'd into ganesh (my department desktop) and from there ssh'd
into a node (not infrequently true, as the nodes aren't
router-accessible from outside the physics domain) I have to escape my
disconnect TWICE or I'll disconnect from ganesh and not just the node.
Working from ganesh directly I have to escape ONCE.  Tres suck.  I don't
know of any variable that maintains the number of ssh layers from a
primary login, so I'd likely have to make one, if possible.

So although I love ssh relative to rsh and strongly advocate its use, it
isn't perfect and these three things, in particular: a "none" cipher
(use at your own deliberate risk), the ability to cleanly background
jobs and still automatically disconnect/notty at logout and the hell
with std[in,out,err] (again at your own risk, of course), and put back
the harmless and lovely symlink feature.  Sure, nobody but the
terminally elderly probably even know that it ever existed or ran
MAKEHOSTS in their lives, but we creaky oldsters actually found it
useful and who knows, it might one day be useful again...

Alas, this is the difference between BSD and linux... linux tends to be
a lot freer with features and less slavishly attuned to an abstract
vision of an operational stack at the expense of functionality, but
openssh is a (primarily) bsd project...:-( I suppose one could possibly
hack these features into the linux "portability goop" (their term for
the OS-specific layer on top of the universally shared operational
core), but I suspect not, or at least not easily.  The cipher almost
certainly not, and quite possibly not the disconnect either.

   rgb

> 
> --Jesse
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu