Just today my server has started being slow at ssh login when a terminal is not requested. The delay seems to be random between 0 and 40 seconds and is reproducible every time. The surprising part is that if I request a tty (-t) the speed is as normal.Here is the output of ssh -tt -vvv user@server echo with timestamp in ms:0.000 OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 20140.000 debug1: Reading configuration data /home/tange/.ssh/config0.000 debug1: /home/tange/.ssh/config line 1: Applying options for *0.000 debug1: /home/tange/.ssh/config line 12: Applying options for *.pi0.000 debug1: /home/tange/.ssh/config line 160: Applying options for *0.000 debug1: Reading configuration data /etc/ssh/ssh_config0.000 debug1: /etc/ssh/ssh_config line 19: Applying options for *0.000 debug2: ssh_connect: needpriv 00.000 debug1: Connecting to xxx.pi [7.2.5.7] port 22.0.027 debug1: Connection established.0.027 debug3: Incorrect RSA1 identifier0.027 debug3: Could not load /home/tange/.ssh/id_rsa as a RSA1 public key0.027 debug1: identity file /home/tange/.ssh/id_rsa type 10.027 debug1: identity file /home/tange/.ssh/id_rsa-cert type -10.027 debug1: identity file /home/tange/.ssh/id_dsa type -10.027 debug1: identity file /home/tange/.ssh/id_dsa-cert type -10.027 debug1: identity file /home/tange/.ssh/id_ecdsa type -10.027 debug1: identity file /home/tange/.ssh/id_ecdsa-cert type -10.027 debug1: identity file /home/tange/.ssh/id_ed25519 type -10.027 debug1: identity file /home/tange/.ssh/id_ed25519-cert type -10.027 debug1: Enabling compatibility mode for protocol 2.00.027 debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu20.066 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4+deb7u20.066 debug1: match: OpenSSH_6.0p1 Debian-4+deb7u2 pat OpenSSH* compat 0x040000000.066 debug2: fd 3 setting O_NONBLOCK0.066 debug3: load_hostkeys: loading entries for host xxx.pi from file /home/tange/.ssh/known_hosts0.067 debug3: load_hostkeys: found key type ECDSA in file /home/tange/.ssh/known_hosts:1500.067 debug3: load_hostkeys: loaded 1 keys0.068 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp5210.068 debug1: SSH2_MSG_KEXINIT sent0.094 debug1: SSH2_MSG_KEXINIT received0.094 debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha10.094 debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss0.094 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se0.095 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se0.095 debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-960.095 debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-960.095 debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none0.095 debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none0.095 debug2: kex_parse_kexinit: 0.095 debug2: kex_parse_kexinit: 0.095 debug2: kex_parse_kexinit: first_kex_follows 0 0.095 debug2: kex_parse_kexinit: reserved 0 0.095 debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha10.095 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp2560.095 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se0.095 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se0.095 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-960.095 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-960.095 debug2: kex_parse_kexinit: none,zlib@openssh.com0.095 debug2: kex_parse_kexinit: none,zlib@openssh.com0.095 debug2: kex_parse_kexinit: 0.095 debug2: kex_parse_kexinit: 0.095 debug2: kex_parse_kexinit: first_kex_follows 0 0.095 debug2: kex_parse_kexinit: reserved 0 0.095 debug2: mac_setup: setup hmac-md50.095 debug1: kex: server->client aes128-ctr hmac-md5 zlib@openssh.com0.095 debug2: mac_setup: setup hmac-md50.095 debug1: kex: client->server aes128-ctr hmac-md5 zlib@openssh.com0.095 debug1: sending SSH2_MSG_KEX_ECDH_INIT0.095 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY0.153 debug1: Server host key: ECDSA 0c:17:b3:93:f6:47:73:de:33:24:6e:db:bf:67:44:1b0.153 debug3: load_hostkeys: loading entries for host xxx.pi from file /home/tange/.ssh/known_hosts0.153 debug3: load_hostkeys: found key type ECDSA in file /home/tange/.ssh/known_hosts:1500.153 debug3: load_hostkeys: loaded 1 keys0.153 debug3: load_hostkeys: loading entries for host 7.2.5.7 from file /home/tange/.ssh/known_hosts0.154 debug3: load_hostkeys: found key type ECDSA in file /home/tange/.ssh/known_hosts:1510.155 debug3: load_hostkeys: loaded 1 keys0.155 debug1: Host 'xxx.pi' is known and matches the ECDSA host key.0.155 debug1: Found key in /home/tange/.ssh/known_hosts:1500.156 debug1: ssh_ecdsa_verify: signature correct0.156 debug2: kex_derive_keys0.156 debug2: set_newkeys: mode 10.156 debug1: SSH2_MSG_NEWKEYS sent0.156 debug1: expecting SSH2_MSG_NEWKEYS0.156 debug2: set_newkeys: mode 00.156 debug1: SSH2_MSG_NEWKEYS received0.156 debug1: Roaming not allowed by server0.156 debug1: SSH2_MSG_SERVICE_REQUEST sent0.306 debug2: service_accept: ssh-userauth0.306 debug1: SSH2_MSG_SERVICE_ACCEPT received0.765 debug2: key: /home/tange/.ssh/id_rsa (0x7f9b662d4bb0),0.765 debug2: key: /home/tange/.ssh/id_rsa_openindiana (0x7f9b662d6a70),0.765 debug2: key: /home/tange/.ssh/id_tym (0x7f9b662d6540),0.765 debug2: key: /home/tange/.ssh/id_dsa ((nil)),0.765 debug2: key: /home/tange/.ssh/id_ecdsa ((nil)),0.765 debug2: key: /home/tange/.ssh/id_ed25519 ((nil)),0.924 debug1: Authentications that can continue: publickey,password0.924 debug3: start over, passed a different list publickey,password0.924 debug3: preferred publickey,keyboard-interactive,password0.924 debug3: authmethod_lookup publickey0.924 debug3: remaining preferred: keyboard-interactive,password0.924 debug3: authmethod_is_enabled publickey0.924 debug1: Next authentication method: publickey0.924 debug1: Offering RSA public key: /home/tange/.ssh/id_rsa0.924 debug3: send_pubkey_test0.924 debug2: we sent a publickey packet, wait for reply0.983 debug1: Server accepts key: pkalg ssh-rsa blen 10470.983 debug2: input_userauth_pk_ok: fp e1:95:e3:ff:99:a6:3a:b5:53:5a:54:59:d0:72:94:7f0.983 debug3: sign_and_send_pubkey: RSA e1:95:e3:ff:99:a6:3a:b5:53:5a:54:59:d0:72:94:7f1.196 debug1: Enabling compression at level 6.1.196 debug1: Authentication succeeded (publickey).1.196 Authenticated to xxx.pi ([7.2.5.7]:22).1.196 debug2: fd 5 setting O_NONBLOCK1.196 debug3: fd 6 is O_NONBLOCK1.197 debug1: channel 0: new [client-session]1.197 debug3: ssh_session2_open: channel_new: 01.197 debug2: channel 0: send open1.197 debug1: Requesting no-more-sessions@openssh.com1.197 debug1: Entering interactive session.1.230 debug2: callback start1.230 debug2: fd 3 setting TCP_NODELAY1.230 debug3: packet_set_tos: set IP_TOS 0x081.230 debug2: client_session2_setup: id 01.230 debug1: Sending environment.1.230 debug3: Ignored env XDG_VTNR1.230 debug3: Ignored env XDG_SESSION_ID1.230 debug3: Ignored env SSH_AGENT_PID1.230 debug3: Ignored env CLUTTER_IM_MODULE1.230 debug3: Ignored env SELINUX_INIT1.230 debug3: Ignored env DM_CONTROL1.230 debug3: Ignored env SESSION1.230 debug3: Ignored env GPG_AGENT_INFO1.230 debug3: Ignored env GLADE_PIXMAP_PATH1.230 debug3: Ignored env XDG_MENU_PREFIX1.230 debug3: Ignored env SHELL1.230 debug3: Ignored env TERM1.231 debug3: Ignored env XDM_MANAGED1.231 debug3: Ignored env SSH_AGENT_LAUNCHER1.231 debug3: Ignored env KONSOLE_DBUS_SERVICE1.231 debug3: Ignored env KONSOLE_PROFILE_NAME1.231 debug3: Ignored env MOZ_NO_REMOTE1.231 debug3: Ignored env WINDOWID1.231 debug3: Ignored env UPSTART_SESSION1.231 debug3: Ignored env PERL_MB_OPT1.231 debug3: Ignored env SHELL_SESSION_ID1.231 debug3: Ignored env XTERM_SHELL1.231 debug3: Ignored env USER1.231 debug3: Ignored env LS_COLORS1.231 debug3: Ignored env ORACLE_SID1.231 debug3: Ignored env DEBEMAIL1.231 debug3: Ignored env GLADE_MODULE_PATH1.231 debug3: Ignored env SSH_AUTH_SOCK1.231 debug3: Ignored env DEFAULTS_PATH1.231 debug3: Ignored env SESSION_MANAGER1.231 debug3: Ignored env XDG_CONFIG_DIRS1.231 debug3: Ignored env PATH1.231 debug3: Ignored env DESKTOP_SESSION1.231 debug3: Ignored env _1.231 debug3: Ignored env QT_IM_MODULE1.231 debug3: Ignored env PWD1.231 debug3: Ignored env JOB1.231 debug3: Ignored env JAVA_HOME1.231 debug3: Ignored env XMODIFIERS1.231 debug3: Ignored env KONSOLE_DBUS_WINDOW1.231 debug1: Sending env LANG = en_US.UTF-81.231 debug2: channel 0: request env confirm 01.231 debug3: Ignored env MANDATORY_PATH1.231 debug3: Ignored env IM_CONFIG_PHASE1.232 debug3: Ignored env KONSOLE_DBUS_SESSION1.232 debug3: Ignored env XTERM_VERSION1.232 debug3: Ignored env XTERM_LOCALE1.232 debug3: Ignored env HISTCONTROL1.232 debug3: Ignored env SESSIONTYPE1.232 debug3: Ignored env HOME1.232 debug3: Ignored env XDG_SEAT1.232 debug3: Ignored env SHLVL1.232 debug3: Ignored env COLORFGBG1.232 debug3: Ignored env JDK_HOME1.232 debug3: Ignored env LANGUAGE1.232 debug3: Ignored env UPSTART_INSTANCE1.232 debug3: Ignored env LOGNAME1.232 debug3: Ignored env LESS1.232 debug3: Ignored env UPSTART_EVENTS1.232 debug3: Ignored env VISUAL1.232 debug3: Ignored env XDG_DATA_DIRS1.232 debug3: Ignored env DBUS_SESSION_BUS_ADDRESS1.232 debug3: Ignored env QT4_IM_MODULE1.232 debug3: Ignored env DEBFULLNAME1.232 debug3: Ignored env LESSOPEN1.232 debug3: Ignored env TEXTDOMAIN1.232 debug3: Ignored env WINDOWPATH1.232 debug3: Ignored env UPSTART_JOB1.232 debug3: Ignored env INSTANCE1.232 debug3: Ignored env DISPLAY1.232 debug3: Ignored env XDG_RUNTIME_DIR1.232 debug3: Ignored env PROFILEHOME1.232 debug3: Ignored env GLADE_CATALOG_PATH1.232 debug3: Ignored env ORACLE_HOME1.232 debug3: Ignored env GTK_IM_MODULE1.232 debug3: Ignored env XDG_CURRENT_DESKTOP1.233 debug3: Ignored env LESSCLOSE1.233 debug3: Ignored env PERL_MM_OPT1.233 debug3: Ignored env TEXTDOMAINDIR1.233 debug3: Ignored env BASH_FUNC_myfunc%%1.233 debug1: Sending command: echo1.233 debug2: channel 0: request exec confirm 11.233 debug2: callback done1.233 debug2: channel 0: open confirm rwindow 0 rmax 3276811.076 debug2: channel 0: rcvd adjust 209715211.076 debug2: channel_input_status_confirm: type 99 id 011.076 debug2: exec request accepted on channel 011.076 debug2: channel 0: rcvd eof11.076 debug2: channel 0: output open -> drain11.076 debug1: client_input_channel_req: channel 0 rtype exit-status reply 011.076 debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 011.076 debug2: channel 0: rcvd eow11.076 debug2: channel 0: close_read11.076 debug2: channel 0: input open -> closed11.076 debug2: channel 0: rcvd close11.076 debug3: channel 0: will not send data after close11.077 11.077 debug3: channel 0: will not send data after close11.077 debug2: channel 0: obuf empty11.077 debug2: channel 0: close_write11.077 debug2: channel 0: output drain -> closed11.077 debug2: channel 0: almost dead11.077 debug2: channel 0: gc: notify user11.077 debug2: channel 0: gc: user detached11.077 debug2: channel 0: send close11.077 debug2: channel 0: is dead11.077 debug2: channel 0: garbage collecting11.077 debug1: channel 0: free: client-session, nchannels 111.077 debug3: channel 0: status: The following connections are open:11.077   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)11.077 11.077 debug1: fd 1 clearing O_NONBLOCK11.077 debug3: fd 2 is not O_NONBLOCK11.077 Transferred: sent 5648, received 2656 bytes, in 9.9 seconds11.077 Bytes per second: sent 571.7, received 268.811.077 debug1: Exit status 011.077 debug1: compress outgoing: raw data 154, compressed 134, factor 0.8711.077 debug1: compress incoming: raw data 101, compressed 81, factor 0.80Any idea why this suddenly happens? And why -t is not affected?
Debugging ssh: slow, but fast with -t
ssh