Login delay w/ ssh

[Brian LaMere]

Ok now mind you, we're just talking a few seconds here...

But I want to get rid of it!

The ssh daemon on the other side responds /immediately/ with its pub key to
let me know who it is (if I ssh to it after deleting its known_hosts entry,
it responds immediately with the key to add for it).  Problem is that once
the key is added, and for subsequent logins, there is a delay of a few
seconds before it asks for password (or allows me on, if I'm authenticating
with a key).  For the purposes of what I need to do with something, the
delay is bad.

Anyone know how to get rid of it?  "nodelay" as an option on the auth lines
of /etc/pam.d/sshd didn't change anything.  So, just curious if anyone knows
off-hand how to deal with this.