# OpenSSH/Logging and Troubleshooting

Both the OpenSSH client and server offer a lot of choice as to where the
logs are written and how much information is collected.

A prerequisite for logging is having an accurate system clock using the
Network Time Protocol, NTP, or equivalent, service which provides
ongoing time synchronization with rest of the world. The more accurate
the time stamp in the log is, the faster it is to coordinate forensics
between machines or sites or service providers. If you have to contact
outside parties like a service provider, progress can usually only be
made with very exact times.

## Server Logs

By default sshd(8) sends logging
information to the system logs using the log level INFO and the system
log facility AUTH. So the place to look for log data from
sshd(8) is in **/var/log/auth.log**.
These defaults can be overridden using the **SyslogFacility** and
**LogLevel** directives. Below is a typical server startup entry in the
authorization log.

``` text
Mar 19 14:45:40 eee sshd[21157]: Server listening on 0.0.0.0 port 22.
Mar 19 14:45:40 eee sshd[21157]: Server listening on :: port 22.
```

In most cases the default level of logging is sufficient, but during
initial testing of new services or activities it is sometimes necessary
to have more information. Debugging info usually goes to **stderr**.
Starting with OpenSSH 7.6, **Match** blocks can set alternate log levels
for specific conditions.

The log excerpt below show the same basic server start up with increased
detail. Contrast the log level DEBUG1 below with the default above:

``` text
debug1: sshd version OpenSSH_6.8, LibreSSL 2.1
debug1: private host key #0: ssh-rsa SHA256:X9e6YzNXMmr1O09LVoQLlCau2ej6TBUxi+Y590KVsds
debug1: private host key #1: ssh-dss SHA256:XcPAY4soIxU2IMtYmnErrVOjKEEvCc3l5hOctkbqeJ0
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:QIWi4La8svQSf5ZYow8wBHN4tF0jtRlkIaLCUQRlxRI
debug1: private host key #3: ssh-ed25519 SHA256:fRWrx5HwM7E5MRcMFTdH95KwaExLzAZqWlwULyIqkVM
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
```

And here is the same startup using the most verbose level, DEBUG3:

``` text
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 217
debug2: parse_server_config: config /etc/ssh/sshd_config len 217
debug3: /etc/ssh/sshd_config:52 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: /etc/ssh/sshd_config:86 setting UsePrivilegeSeparation sandbox          
debug3: /etc/ssh/sshd_config:104 setting Subsystem sftp internal-sftp 
debug1: sshd version OpenSSH_6.8, LibreSSL 2.1
debug1: private host key #0: ssh-rsa SHA256:X9e6YzNXMmr1O09LVoQLlCau2ej6TBUxi+Y590KVsds
debug1: private host key #1: ssh-dss SHA256:XcPAY4soIxU2IMtYmnErrVOjKEEvCc3l5hOctkbqeJ0
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:QIWi4La8svQSf5ZYow8wBHN4tF0jtRlkIaLCUQRlxRI
debug1: private host key #3: ssh-ed25519 SHA256:fRWrx5HwM7E5MRcMFTdH95KwaExLzAZqWlwULyIqkVM
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on ::.
```

Every failed login attempt is recorded, once the value in directive
**MaxAuthTries** is exceeded the connection is broken. Below is a log
excerpt showing how the default log looks after some failed attempts:

``` text
...
Mar 19 11:11:06 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:06 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:07 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:08 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:09 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:10 server sshd[54798]: Failed password for root from 122.121.51.193 port 59928 ssh2
Mar 19 11:11:10 server sshd[54798]: error: maximum authentication attempts exceeded for root from 122.121.51.193 port 59928 ssh2 [preauth]
Mar 19 11:11:10 server sshd[54798]: Disconnecting authenticating user root 122.121.51.193 port 59928: Too many authentication failures [preauth]
...
```

It is not usually a good idea to allow root login, at least not root
login with authentication via password. Blocking password authentication
for root simplifies log analysis greatly, and in particular it
eliminates the time consuming question of who is trying to get in and
why. People that need full root level access can gain it through
su(1) for general activities. Or for
specific tasks which need root level access, those can be given those
privileges through custom-made entries for **sudo(8)** or
doas(1). Note that in those cases,
only specific services and programs should be allowed, not blanket
access which is an all too common misconfiguration seen with
**sudo(8)**. Alternatively, a single-purpose key made using
**forced-commands-only** could be used since some argue that providing
extra means of privilege escalation, such as
su(1), **sudo(8)**, or
doas(1), is more dangerous than
carefully providing remote root access through a key or certificate tied
to a specific function.

### Successful logins

By default, the server does not store much information about user
transactions. That is a good thing. It is also a good thing to recognize
when the system is operating as it should. So here is an example of a
successful SSH login:

``` text
Mar 14 19:50:59 server sshd[18884]: Accepted password for fred from 192.0.2.60 port 6647 ssh2
```

And here is an example using a key for authentication. It shows the key
fingerprint as a SHA256 hash in base64.

``` text
Mar 14 19:52:04 server sshd[5197]: Accepted publickey for fred from 192.0.2.60 port 59915 ssh2: RSA SHA256:5xyQ+PG1Z3CIiShclJ2iNya5TOdKDgE/HrOXr21IdOo
```

And here is an example of successful authentication with a user
certificate. The certificate\'s identification string is \"foobar\" and
the serial number is \"9624\". In this example the certificate is using
ECDSA and the key itself is using Ed25519. The certificate, being a
different key of its own, has a different SHA256 fingerprint from the
authentication key itself.

``` text
May 15 16:28:17 server sshd[50140]: Accepted publickey for fred from 192.0.2.60 port 44456 ssh2: ECDSA-CERT SHA256:qGl9KiyXrG6mIOo1CT01oHUvod7Ngs5VMHM14DTbxzI ID foobar (serial 9624) CA ED25519 SHA256:fZ6L7TlBLqf1pGWzkcQMQMFZ+aGgrtYgRM90XO0gzZ8
```

Prior to 6.8, the key\'s fingerprint was a hexadecimal MD5 hash.

``` text
Jan 28 11:51:43 server sshd[5104]: Accepted publickey for fred from 192.0.2.60 port 60594 ssh2: RSA e8:31:68:c7:01:2d:25:20:36:8f:50:5d:f9:ee:70:4c
```

In older versions of OpenSSH prior to 6.3 the key fingerprint is
completely missing from authentication logging.

``` text
Jan 28 11:52:05 server sshd[1003]: Accepted publickey for fred from 192.0.2.60 port 20042 ssh2
```

Here is an example of password authentication for an SFTP session, using
the server\'s internal-sftp subsystem. The logging for that subsystem
set to INFO.

``` text
Mar 14 20:14:18 server sshd[19850]: Accepted password for fred from 192.0.2.60 port 59946 ssh2
Mar 14 20:14:18 server internal-sftp[11581]: session opened for local user fred from [192.0.2.60]
```

Here is an example of a successful SFTP login using an RSA key for
authentication.

``` text
Mar 14 20:20:53 server sshd[10091]: Accepted publickey for fred from 192.0.2.60 port 59941 ssh2: RSA SHA256:LI/TSnwoLryuYisAnNEIedVBXwl/XsrXjli9Qw9SmwI
Mar 14 20:20:53 server internal-sftp[31070]: session opened for local user fred from [192.0.2.60]
```

Additional data, such as connection duration, can be logged with the
help of **xinetd**.

### Logging Problems from SSH Certificate Authentication

Usually, not much information is given about which certificate failed,
just why it failed authentication. Finding the account or actual
certificate in question can require some sleuthing. Generally no client
side information is disclosed and all investigation must occur server
side.

If the authentication is attempted again by other means, such as a
password, then when the connection is closed there will be a log entry
noting which account was involved. That is because so early in the
connection sequence the process ID for the disconnection is the same and
the account name and the originating address is included, giving a bit
of a clue in pursuit of a solution. Sometimes even the account name will
be included.

``` text
May  5 16:31:38 server sshd[252]: Connection closed by authenticating user fred 192.0.2.60 port 44470 [preauth]
```

However, if the connection is allowed to timeout without first making
any other authentication attempts by some other means, then there will
be nothing to go on except maybe the time of day.

``` text
May  5 16:33:00 server sshd[90593]: fatal: Timeout before authentication for 192.0.2.60 port 44718
```

Below are some common examples of log entries for failed
certificate-based log in attempts. There can be more than one problem
with a certificate but only one error will get logged at a time.

#### Expired or Not-yet-valid Certificate

Certificates which have not yet become valid or which have already
expired get a log entry as to the reason but neither the account or
certificate involved.

``` text
May  5 16:35:20 server sshd[252]: error: Certificate invalid: expired
```

Above is an expired certificate, below is a certificate which has not
yet become valid.

``` text
May  5 16:58:00 server sshd[90593]: error: Certificate invalid: not yet valid
```

Neither type of event gives more information.

#### Valid Certificate but Invalid Principal

Like with expired certificates, very little information is given about
the actual account or certificate. Here the certificate was tried with
the wrong account, one not listed among the certificate\'s principals.

``` text
May  5 17:29:52 server sshd[98884]: error: Certificate invalid: name is not a listed principal
May  5 17:29:56 server sshd[98884]: Connection closed by authenticating user fred 192.0.2.60 port 45114 [preauth]
```

If the client closes the connection on purpose, there may be some
information in the connection closed entry.

#### Valid Certificate but Invalid Source Address

If the certificate is limited to connecting from specific addresses or
host names, the log will complain if the connection comes from a
different address or host and identify the incorrect source address.

``` text
May  5 17:48:54 server sshd[2420]: cert: Authentication tried for fred with valid certificate but not from a permitted source address (192.0.2.61).
May  5 17:48:54 server sshd[2420]: error: Refused by certificate options
```

However, it will not be possible to identify the specific certificate
directly.

### Logging SFTP File Transfers

SFTP file transfers can be logged using **LogLevel** INFO or VERBOSE.
The log level for the SFTP server can be set in
sshd_config(5) separately from
the general SSH server settings.

``` {.apache .numberLines}
Subsystem internal-sftp -l INFO
```

By default the SFTP messages will also end up in *auth.log* but it is
possible to filter these messages to their own file by reconfiguring the
system logger, usually **rsyslogd(8)** or **syslogd(8)**. Sometimes this
is done by changing the log facility code from the default of AUTH.
Available options are LOCAL0 through LOCAL7, plus, less usefully, DAEMON
and USER.

``` {.apache .numberLines}
Subsystem internal-sftp -l INFO -f LOCAL6
```

If new system log files are assigned, it is important to remember them
in log rotation, too. Again, the **Match** directive can be use to
change the log level for certain connections.

The following log excerpts are generated from using the log level INFO.
A session starts with an open and end with a close. The number in the
brackets is the process id for the SFTP session and is the only way to
follow a session through the logs.

``` text
Oct 22 11:59:45 server internal-sftp[4929]: session opened for local user fred from [192.0.2.33]
...
Oct 22 12:09:10 server internal-sftp[4929]: session closed for local user fred from [192.0.2.33]
```

Here is an SFTP upload of a small file of 928 bytes named **foo** to the
home directory for user \'fred\'.

``` text
Oct 22 11:59:50 server internal-sftp[4929]: open "/home/fred/foo" flags WRITE,CREATE,TRUNCATE mode 0664
Oct 22 11:59:50 server internal-sftp[4929]: close "/home/fred/foo" bytes read 0 written 928
```

And a directory listing in the same session in the directory
**/var/www**.

``` text
Oct 22 12:07:59 server internal-sftp[4929]: opendir "/var/www"
Oct 22 12:07:59 server internal-sftp[4929]: closedir "/var/www"
```

And lastly here is a download of the same small 928-byte file called
**foo** from the home directory for the user \'fred\'.

``` text
Oct 22 12:08:03 server internal-sftp[4929]: open "/home/fred/foo" flags READ mode 0666
Oct 22 12:08:03 server internal-sftp[4929]: close "/home/fred/foo" bytes read 928 written 0
```

Successful transfers will be noted by a *close* message. Attempts to
download (open) files that do not exist will be followed by a *sent
status No such file* message on a line of its own instead of a *close*.
Files that exist but that the user is not allowed to read will create a
*sent status Permission denied* message.

### Logging Chrooted SFTP

Logging with the built-in **sftp-subsystem** inside a chroot jail,
defined by **ChrootDirectory**, needs a **./dev/log** node to exist
inside the jail. This can be done by having the system logger such as
syslogd(8) add additional log
sockets inside the chrooted directory when starting up. On some systems
that is as simple as adding more flags, like \"**-u -a
/chroot/dev/log**\", in **/etc/rc.conf.local** or whatever the
equivalent startup script may be.

Here is an example of an SFTP login with password to a chroot jail using
log level DEBUG3 for the SFTP-subsystem. The log shows a file upload:

``` text
Jan 28 12:42:41 server sshd[26299]: Connection from 192.0.2.60 port 47366
Jan 28 12:42:42 server sshd[26299]: Failed none for fred from 192.0.2.60 port 47366 ssh2
Jan 28 12:42:44 server sshd[26299]: Accepted password for fred from 192.0.2.60 port 47366 ssh2
Jan 28 12:42:44 server sshd[26299]: User child is on pid 21613
Jan 28 12:42:44 server sshd[21613]: Changed root directory to "/home/fred"
Jan 28 12:42:44 server sshd[21613]: subsystem request for sftp
Jan 28 12:42:44 server internal-sftp[2084]: session opened for local user fred from [192.0.2.60]
Jan 28 12:42:58 server internal-sftp[2084]: open "/docs/somefile.txt" flags WRITE,CREATE,TRUNCATE mode 0644
Jan 28 12:42:58 server internal-sftp[2084]: close “/docs/somefile.txt” bytes read 0 written 400
```

Remember that SFTP is a separate subsystem and that like the file
creation mode, the log level and log facility are set separately from
the SSH server in
sshd_config(5):

``` {.apache .numberLines}
Subsystem internal-sftp -l ERROR
```

### Logging Stability of Client Connectivity

When **ClientAliveInterval** is set in the server\'s configuration, the
server makes periodic probes of the clients which have established
connections. At normal log levels, these are not noted in the log until
something goes wrong.

If the **ClientAliveInterval** is exceeded more times in a row than
allowed by **ClientAliveCountMax** the client is officially declared
disconnected and the connection dropped. At the default log level of
INFO a brief message is logged, identifying the client which has been
dropped.

``` text
Sep  6 14:42:08 eee sshd[83709]: packet_write_poll: Connection from 192.0.2.97 port 57608: Host is down
```

At log level DEBUG, the client\'s responses to the polls will be logged
by the server showing that the session is still connected.

``` text
Sep  6 14:27:52 eee sshd[9075]: debug1: Got 100/147 for keepalive
```

Log level DEBUG2 and DEBUG3 will give even more information about the
connection. However, even at log level DEBUG3, the specific client being
polled will not be identified directly in the log messages and will have
to be inferred from the process id of the daemon if such information is
needed.

``` text
Sep  6 14:30:59 eee sshd[73960]: debug2: channel 0: request keepalive@openssh.com confirm 1
Sep  6 14:30:59 eee sshd[73960]: debug3: send packet: type 98
Sep  6 14:30:59 eee sshd[73960]: debug3: receive packet: type 100
Sep  6 14:30:59 eee sshd[73960]: debug1: Got 100/22 for keepalive
```

Again, when the **ClientAliveCountMax** is exceeded, the connection is
broken after the final failure of the client to respond. Here is how
that looks with the log level set to DEBUG2.

``` text
Sep  6 14:17:55 eee sshd[15780]: debug2: channel 0: request keepalive@openssh.com confirm 1
Sep  6 14:17:55 eee sshd[15780]: debug1: Got 100/22 for keepalive
Sep  6 14:18:37 eee sshd[15780]: debug2: channel 0: request keepalive@openssh.com confirm 1
Sep  6 14:18:37 eee sshd[15780]: packet_write_poll: Connection from 192.0.2.97 port 57552: Host is down
Sep  6 14:18:37 eee sshd[15780]: debug1: do_cleanup
Sep  6 14:18:37 eee sshd[48675]: debug1: do_cleanup
Sep  6 14:18:37 eee sshd[48675]: debug1: session_pty_cleanup: session 0 release /dev/ttyp0
```

The directives **ClientAliveInterval** and **ClientAliveCountMax**
normally apply to all clients connecting to the server. However, they
can be used inside a **Match** block and thus applied only to specific
connections.

### Logging Revoked Keys

If the **RevokedKeys** directive is used to point to a list of public
keys that have been revoked, sshd(8)
will make a log entry when access is attempted using a revoked key. The
entry will be the same whether a plaintext list of public keys is used
or if a binary Key Revocation List (KRL) has been generated.

If password authentication is allowed, and the user tries it, then after
the key authentication fails there will be a record of password
authentication.

``` text
Mar 14 20:36:40 server sshd[29235]: error: Authentication key RSA SHA256:jXEPmu4thnubqPUDcKDs31MOVLQJH6FfF1XSGT748jQ revoked by file /etc/ssh/ssh_revoked_keys
...
Mar 14 20:36:45 server sshd[29235]: Accepted password for fred from 192.0.2.10 port 59967 ssh2
```

If password authentication is not allowed,
sshd(8) will close the connection as
soon as the key fails.

``` text
Mar 14 20:38:27 server sshd[29163]: error: Authentication key RSA SHA256:jXEPmu4thnubqPUDcKDs31MOVLQJH6FfF1XSGT748jQ revoked by file /etc/ssh/ssh_revoked_keys
...
```

The account trying the revoked key remains a mystery though, so it will
be necessary to try to look up the key by its fingerprint from your
archive of old keys using `ssh-keygen -lf` and read the key\'s comments.
Although if a valid account cancels the connection without trying a
password after the key attempt fails, the usual message will still be
posted to the log.

``` text
Mar 14 20:44:04 server sshd[14352]: Connection closed by authenticating user fred 192.0.2.237 port 55051 [preauth]
...
```

That may provide some clue and allow filtering with a short AWK script,
if the messages are all in the same log file.

``` shell-session
$ awk '/revoked by file/ {
        pid[$5]++; key[$5]=$9; hash[$5]=$10; next;
    }
    pid[$5] && /closed by authenticating user/ {
        print key[$5], hash[$5], $10, $11;
        delete key[$5]; delete hash[$5]; delete pid[$5];
    }' /var/log/authlog
```

Similarly, if the client makes no attempt at logging in and just times
out, the message will say just that.

``` text
Mar 18 21:40:25 server sshd[9942]: fatal: Timeout before authentication for 198.51.100.236 port 53728
...
```

On the client side, no warning or error will be given if a revoked key
is tried. It will just fail and the next key or method will be tried.

### Brute force and Hail Mary attacks

It's fairly common to see failed login attempts almost as soon as the
server is connected to the net. Brute force attacks, where one machine
hammers on a few accounts trying to find a valid password, are becoming
rare. In part this is because packet filters, like NFables for Linux and
PF for the BSDs, can limit the number and rate of connection attempts
from a single host. The server configuration directive **MaxStartups**
can limit the number of simultaneous, unauthenticated connections.

``` text
...
Mar 18 18:54:44 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2
Mar 18 18:54:48 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2
Mar 18 18:54:49 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2
Mar 18 18:54:49 server sshd[54939]: error: maximum authentication attempts exceeded for root from 201.179.249.231 port 52404 ssh2 [preauth]
Mar 18 18:54:49 server sshd[54939]: Disconnecting authenticating user root 201.179.249.231 port 52404: Too many authentication failures [preauth]
...
```

Note the \"authenticating user\" is present in the logs from OpenSSH 7.5
and onward when a valid user name is attempted. When an invalid user
name is attempted, that is written too.

``` text
...
Mar 18 18:55:05 server sshd[38594]: Invalid user ubnt from 201.179.249.231 port 52471
Mar 18 18:55:05 server sshd[38594]: Failed password for invalid user ubnt from 201.179.249.231 port 52471 ssh2
Mar 18 18:55:09 server sshd[38594]: error: maximum authentication attempts exceeded for invalid user ubnt from 201.179.249.231 port 52471 ssh2 [preauth]
Mar 18 18:55:09 server sshd[38594]: Disconnecting invalid user ubnt 201.179.249.231 port 52471: Too many authentication failures [preauth]
...
```

The way to deal with brute force attacks coming from a single machine or
network is to customize the server host's packet filter to limit the
attacks or even temporarily block machines that overload the maximum
number or rate of connections. Optionally, one should also contact the
attacker's net block owner with the IP address and exact date and time
of the attacks.

A kind of attack common at the time of this writing is one which is
distributed over a large number of compromised machines, each playing
only a small role in attacking the server.

To deal with Hail Mary attacks, contact the attacker's net block owner.
A form letter with a cut-and-paste excerpt from the log is enough if it
gives the exact times and addresses. Alternately, teams of network or
system administrators can work to pool data to identify and blacklist
the compromised hosts participating in the attack.

#### Failed None For Invalid User

The SSH protocol specifies a number of possible authentication
methods[^1]. The methods `password`, `keyboard-interactive`, and
`publickey` are fairly common. A lesser known authentication method is
`none`, which will only succeed if the server requires no further
authentication such as if **PermitEmptyPassword** is set and the account
does not actually have a password[^2]. Some SSH clients including
OpenSSH\'s start by asking for `none` authentication and then use the
list of remaining possible authentication methods to decide what to do
next if that doesn\'t work.

``` text
...
Aug 10 19:09:05 server sshd[93126]: Failed none for invalid user admin from 125.64.94.136 port 27586 ssh2
...
```

So in other words, that is a brute force attack trying the `none`
authentication method. It is an attack which will only get into accounts
which have been explicitly set with an empty password and furthermore
have also been set up specifically to allow access by having both the
authentication method of `none` and the **PermitEmptyPasswords**
configuration directive enabled on the server. Most brute force attacks
try only `password` authentication, and some of those even check for the
`password` method and then give up if it is not available. Other
attackers may just hammer away pointlessly even if the method is not
available.

### Connections Seemingly From 127.0.0.1, ::1, or Other localhost Addresses

When the SSH server is accessed via a reverse tunnel to another machine,
the incoming connections will appear to be from the *localhost* address,
which is usually *127.0.0.1* or *::1*.

``` text
Mar 23 14:16:16 server sshd[9265]: Accepted password for fred from 127.0.0.1 port 40426 ssh2
```

If the port at the other end of the reverse tunnel is publicly
accessible, it will be probed and possibly attacked. Because of the
reverse tunnel, the attacks will then also appear to be coming from the
server\'s own loopback address:

``` text
Mar 23 14:20:17 server sshd[5613]: Invalid user cloud from ::1 port 57404
Mar 23 14:20:21 server sshd[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2
Mar 23 14:20:26 server sshd[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2
Mar 23 14:20:32 server sshd[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2
Mar 23 14:20:35 server sshd[5613]: Connection closed by invalid user cloud ::1 port 57404 [preauth]
```

Therefore the usual countermeasures like SSHGuard or Fail2Ban or other
similar intrusion detection systems cannot be used because the
*localhost* address is used by the tunnel for all login attempts,
regardless of their real origins.

A partial solution would be to bind the incoming connections to a
different IP address. The loopback interface would need an additional
permanent address, an alias, for that. That alias could then be assigned
when establishing the reverse tunnel:

``` shell-session
$ ssh -R 2022:127.2.2.1:22 fred@vps.example.org
```

Thus it would designate the source address for all logins coming in over
that tunnel. So, in that way, the alias would then show up in the logs
instead of the default loopback address when that reverse tunnel is
used:

``` text
Mar 23 18:00:13 server sshd[8525]: Invalid user cloud from 127.2.2.1 port 17271
Mar 23 18:00:15 server sshd[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2
Mar 23 18:00:19 server sshd[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2
Mar 23 18:01:23 server sshd[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2
Mar 23 18:01:26 server sshd[8525]: Connection closed by invalid user cloud 127.2.2.1 port 17271 [preauth]
```

If the ports do not need to be available to the open Internet, a full
solution would be just to ensure that they are not accessible from the
outside. This would be done by not using the **-g** option on the client
when making the reverse tunnel or else by setting the **GatewayPorts**
directive in sshd_config(5) back
to the default of **no**, or both. The system\'s built-in packet filter
can also be used. Then, even with the forwarded ports closed off from
the outside, the **ProxyJump** option can still be used to skip through
the jump host and use the setup for SSH access. However, since it is
sometimes necessary that these ports be accessible to the outside world,
this approach is not always an option.

## Client Logging

The OpenSSH client normally sends log information to **stderr**. The
**-y** option can be used to send output to the system logs, managed by
syslogd(8) or something similar.
Client log verbosity can be increased or decreased by changing the
**LogLevel** directive and the log facility changed with the
**SyslogFacility** directive in
ssh_config(5). Both require use
of the **-y** run time option and do nothing without it.

Alternatively, instead of using the **-y** option, using the **-E**
option sends log output to a designated file instead of **stderr**.
Working with the system logs or separate log files are something which
can be useful when running ssh(1) in
automated scripts. Below is an example of a connection to an interactive
shell with the normal level of client logging:

``` shell-session
$ ssh -l fred server.example.org    
fred@server.example.org‘s password: 
Last login: Thu Jan 27 13:21:57 2011 from 192.168.11.1
```

The same connection at the first level of verbosity gives lots of
debugging information, 42 lines more.

``` shell-session
$ ssh -v -l fred server.example.org
OpenSSH_6.8, LibreSSL 2.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to server.example.org [198.51.100.20] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fred/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7
debug1: match: OpenSSH_6.7 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:CEXGTmrVgeY1qEiwFe2Yy3XqrWdjm98jKmX0LK5mlQg
debug1: Host '198.51.100.20' is known and matches the ECDSA host key.
debug1: Found key in /home/fred/.ssh/known_hosts:2
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/fred/.ssh/id_rsa
debug1: Trying private key: /home/fred/.ssh/id_dsa
debug1: Trying private key: /home/fred/.ssh/id_ecdsa
debug1: Trying private key: /home/fred/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: password
debug1: Authentication succeeded (password).
Authenticated to server.example.org ([198.51.100.20]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 2 clearing O_NONBLOCK
Last login: Sat Mar 14 21:31:33 2015 from 192.0.2.111

...
```

The same login with the maximum of verbosity, **-vvv**, gives around 150
lines of debugging information. Remember that debugging information is
sent to **stderr** rather than **stdout**. This will only capture the
session in a file, debugging info goes only to the screen, not to the
output log:

``` shell-session
$ ssh -vvv -l fred  somehost.example.org  | tee ~/ssh-output.log
```

The tool tee(1) is like a T-pipe and
sends output two directions, one to **stdout** and one to a file.

The following will capture both the debugging info and the session text:

``` shell-session
$ ssh -vvv -l fred  somehost.example.org  2>&1  | tee ~/ssh-output.log
```

### Capturing Client Debugging Information Separately

Regular pipes and redirects work only with **stdout** so that if **-E**
is not used to capture debugging output the output on **stderr** must
instead be sent to stdout if one is going to capture it at the same time
as the actual session. That is done with an extra redirect, **2\>&1** to
capture **stderr**. Mind the spaces, or lack of them.

### Changing Client Debugging Levels At Runtime or On The Fly

At run time, when establishing a new connection, just use the **-v**
option.

``` shell-session
$ sftp -v -o "IdentityFile=~/.ssh/weblog.key_rsa" fred@server.example.org
```

The debugging verbosity on the client can be increased just like on the
server.

``` shell-session
$ sftp -vvv -o "IdentityFile=~/.ssh/weblog.key_rsa" fred@server.example.org
```

The extra information can be useful to see exactly what is being sent to
or requested of the server.

After the fact, once a connection is established, the escape sequences
**\~v** and **\~V** can be used to increase or decrease the verbosity on
the fly. Through them it is possible to change the verbosity of the
client in an established connection. When increasing, the client raises
its log level to VERBOSE, DEBUG, DEBUG2, and DEBUG3, in that order, if
starting from the default of INFO. Conversely, when lowering the log
level, the client will descend through ERROR, FATAL, to QUIET if
starting from the default of INFO.

## Debugging and Troubleshooting

The server logs are your best friend when troubleshooting. It may be
necessary to turn up the log level there temporarily to get more
information. It is then also necessary to turn them back to normal after
things are fixed to avoid privacy problems or excessively use of disk
space.

For example, the SFTP-subsystem logging defaults to ERROR, reporting
only errors. To track transactions made by the client, change the log
level to INFO or VERBOSE:

``` {.apache .numberLines}
Subsystem internal-sftp  -l INFO 
```

Caution. Again, operating with elevated logging levels would violate the
privacy of users, in addition to filling a lot of disk space, and should
generally not be used in production once the changes are figured out.
Elevated log messages should really be sent to a separate log file
during the time they are collected.

By default, some systems send only the normal messages to the regular
system log files and ignore the elevated messages. Some save all the
messages by default. If the elevated system log messages are not showing
up in any of the system logs, the former may be the reason. Either way,
check the system log configuration and make sure that the extra messages
are only sent to a separate log file and not mixed with the regular
system logs. Change the configuration if necessary. This helps keep the
logs tidy as well as protect privacy. The system log settings are found
in the system log daemon\'s configuration file, the exact name of which
will vary depending on what is installed, but common ones are
syslog.conf(5) and
rsyslog.conf(5). Notice that this machine configuration below has the
more detailed DEBUG messages for the AUTH facility going to a separate
log file from the regular AUTH messages:

``` shell-session
$ grep '^auth\.' /etc/syslog.conf 
auth.info                                               /var/log/authlog
auth.debug                                              /var/log/authdebug
```

See syslog(3) for the log facilities
and log levels. It is best to limit the time the debugging information
is collected and to actively watch while it is collected. However, if it
is running for any length of time, and especially if it is left
unattended even for a short while, be sure to remeber to add the special
log file to the log rotation schedule so that it cannot fill up the
partition.

**Match** blocks can help further by setting log levels for specific
situations and avoid a situation where everything is logged intensely.

Also, the manual pages for OpenSSH are very well written and many times
problems can be solved by finding the right section within the right
manual page. At the very minimum, it is important to skim through the
four main manual pages for both the programs and their configuration and
become familiar with at least the section headings.

-   ssh(1)
-   ssh_config(5)
-   sshd(8)
-   sshd_config(5)

Then once the right section is found in the manual page, go over it in
detail and become familiar with its contents. The same goes for the
other OpenSSH manual pages, depending on the activity. Be sure to use
the version of OpenSSH available for your system and the corresponding
manual pages, preferably those that are installed on your system to
avoid a mismatch. In some cases, the client and the server will be of
different versions, so the manual pages for each must be looked up
separately. It is also a good idea to review OpenSSH\'s release notes
when a new version is published.

With a few exceptions below, specific examples of troubleshooting are
usually given in the cookbook section relevant to a particular activity.
So, for example, sorting problems with authentication keys is done in
the section on Public Key
Authentication itself.

### Debugging a script, configuration or key that uses sudo(8)

Usually log levels only need to be changed when writing and testing a
script, a new configuration, some new keys, or all three at once. When
working with **sudo(8)**, it is especially important to see exactly what
the client is sending so as to enter the right pattern into
**/etc/sudoers** for safety. Using the lowest level of verbosity, the
exact string being sent by the client to the remote server is shown in
the debugging output:

``` shell-session
$ rsync -e "ssh -v -i /home/webmaint/.ssh/bkup_key -l webmaint" \
        -a server.example.org:/var/www/ var/backup/www/
...
debug1: Authentication succeeded (publickey).
Authenticated to server.example.org ([192.0.2.20]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending command: rsync --server --sender -vlogDtpre.if . /var/www/
receiving incremental file list
... 
```

What **sudoers** then needs is something like the following, assuming
account \'webmaint\' is in the group \'webmasters\':

``` {.apache .numberLines}
%webmasters ALL=(ALL) NOPASSWD: /usr/local/bin/rsync --server \
--sender -vlogDtpre.if . /var/www/
```

The same method can be used to debug new server configurations or key
logins. Once things are set to run as needed, the log level settings can
be lowered back to INFO for sshd(8) and
to ERROR for **internal-sftp**. Additionally, once the script is left to
run in fully automated mode, the client logging information can be set
use the syslog(3) system module
instead of **stderr** by setting the **-y** option when it is launched.

### Debugging a server configuration

Running the server in debug mode provides a lot of information about the
connection and a smaller amount about the server configuration. The
server\'s debugging level (**-d**) can be raised once, twice (**-dd**)
or thrice (**-ddd**).

``` shell-session
$ /usr/sbin/sshd -d
```

Note that the server in this case does not detach and become a daemon,
so it will terminate when the SSH connection terminates. The server must
be started again in order to make a subsequent connection from the
client. Though in some ways this is a hassle, it does make sure that
session data is a unique set and not mixes of multiple sessions and thus
possibly different configurations. Alternately, another option (**-e**)
when debugging sends the debugging data to **stderr** to keep the system
logs clean.

In recent versions of OpenSSH, it is also possible to log the debug data
from the system logs directly to a separate file and keep noise out of
the system logs. Since OpenSSH 6.3, the option **-E** will append the
debug data to a particular log file instead of sending it to the system
log. This facilitates debugging live systems without cluttering the
system logs.

``` shell-session
$ /usr/sbin/sshd -E /home/fred/sshd.debug.log
```

On older versions of OpenSSH, if you need to save output to a file while
still viewing it live on the screen, you can use **tee(1)**.

``` shell-session
$ /usr/sbin/sshd -ddd 2>&1 | tee /tmp/foo
```

That will save output to the file *foo* by capturing what
sshd(8) sent to **stderr**. This works
with older versions of OpenSSH, but the **-E** option above is
preferable.

If the server is remote and it is important to reduce the risk of
getting locked out, the experiments on the configuration file can be
done with a second instance of sshd(8)
using a separate configuration file and listening to a high port until
the settings have been tested.

``` shell-session
$ /usr/sbin/sshd -dd -p 22222 -f /home/fred/sshd_config.test
```

It is possible to make an extended test (**-T**) of the configuration
file. If there is a syntax error, it will be reported, but remember that
even sound configurations can still lock you out. The extended test mode
can be used by itself, but it is also possible to specify particular
connection parameters to use with **-C**.
sshd(8) will then process the
configuration file in light of the parameters passed to it and output
the results. Of particular use, the results of **Match** directives will
be shown. So the **-T** option can be supplemented with the **-C**
option to show precisely which configuration will be used for various
connections.

When passing specific connection parameters to
sshd(8) for evaluation, *user*, *host*,
and *addr* are the minimum required for extended testing. The following
will print out the configurations that will be applied if the user
*fred* tries to log in to the host *server.example.org* from the address
*192.0.2.15*:

``` shell-session
$ /usr/sbin/sshd -T -C user=fred,host=server.example.org,addr=192.0.2.15
```

Two more parameters, *laddr* and *lport*, may also be passed. They refer
to the server\'s IP number and port connected to.

``` shell-session
$ /usr/sbin/sshd -T -C user=fred,host=server.example.org,addr=192.0.2.15,laddr=192.0.2.2,lport=2222
```

Those five variables should be able to describe any possible incoming
connection.

### Debugging a client configuration

Sometimes when debugging a server configuration it is necessary to track
the client, too. Since OpenSSH 6.8, the -G option makes
ssh(1) print its configuration after
evaluating **Host** and **Match** blocks and then exit. That allows
viewing of the exact configuration options that will actually be used by
the client for a particular connection.

``` shell-session
$ ssh -G -l fred server.example.org
```

Client configuration is determined in three ways. The first is by
run-time options, then by the account\'s own configuration file, or
lastly the system-wide client configuration file. The priority is in
that order and whichever value is found first gets used. With
sftp(1) the options are also passed to
ssh(1).

#### Invalid or Outdated Ciphers or MACs

A proper client will show the details of the failure. For a bad Message
Authentication Code (MAC), a proper client might show something like the
following when trying to foist a bad MAC like hmac-md5-96 onto the
server:

``` text
no matching mac found: client hmac-md5-96 server umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
```

And for a bad cipher, a proper client might show something like this
when trying to foist an arcfour cipher on the server:

``` text
no matching cipher found: client arcfour server chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
```

Sometimes when troubleshooting a problem with the client it is necessary
to turn to the server logs. In OpenSSH 6.7 unsafe MACs were
removed and in OpenSSH 7.2
unsafe ciphers were removed,
but some third-party clients may still try to use them to establish a
connection. In that case, the client might not provide much information
beyond a vague message that the server unexpectedly closed the network
connection. The server logs will, however, show what happened:

``` text
fatal: no matching mac found: client hmac-sha1,hmac-sha1-96,hmac-md5 server hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160 [preauth]
```

More recent versions would show a simpler error for a bad MAC.

``` text
fatal: Unable to negotiate with 192.0.2.37 port 55044: no matching MAC found. Their offer: hmac-md5-96 [preauth]
```

A bad cipher would be reported like this:

``` text
fatal: Unable to negotiate with 192.0.2.37 port 55046: no matching cipher found. Their offer: arcfour [preauth]
```

The error message in the server log might not say which MACs or ciphers
are actually available. For those, the extended test mode can be used to
show the server settings and, in particular, the MACs or ciphers
allowed. In its most basic usage the extended test mode would just be
-T, as in `/usr/sbin/sshd -T | grep -E 'cipher|macs'` with no other
options. For more details and options, see the previous section on
\"Debugging a server configuration\" above.

One solution there is to upgrade the client to one that can handle the
right ciphers and MACs. Another option is to switch to a different
client, one that can handle the modern ciphers or MACs.

### Debugging Key-Based Authentication

The most common causes of failure for public key authentication seem to
be for either of two reasons:

-   Mangling the public key on the way to getting it into
    **authorized_keys** on the server
-   Incorrect permissions for the files and directories involved, ether
    on the client or the server. These are the directory for the keys,
    usually **\~/.ssh/**, or its parent directories, or the
    **authorized_keys** file, or the private key itself.

As of the time of this writing, it looks like pretty much every failure
of key-based authentication described on mailing lists and forums is
solved by addressing either or both of those two situations. So, when
encountering the error message \"Permission denied
(publickey,keyboard-interactive)\", or similar, see the section on
Public Key
Authentication.
Then see the manual page for sshd(8)
and its section on authorized keys.

A very rare third case is if the public and private key files don\'t
match and are not from the same key pair. As mentioned in the section on
Public Key
Authentication,
the public and private keys need to match and be part of the same key
pair. That is because even before the SSH client uses private key
cryptographically, it looks at the file name of the proposed private key
and then sends the public key matching that same name, if it exists. If
the public key that is on the client does not match the public key on
the server in **authorized_keys** then the connection will be denied
with the error \"Permission denied (publickey,keyboard-interactive)\" or
similar. That alone is a very good reason to give unique descriptive
file names. Note that as mentioned above there are usually other causes
to that same error message besides having mismanaged the public key file
on the client machine.

The file names for both parts of each key pair have to be kept organized
so that the contents match. As for a solution, the way out in the long
term is to more carefully manage the keys and their file names. It is
solved on the short term by deleting the offending public key file or
using the private key to regenerate a new one, overwriting the offending
one. Again, this is an unusual edge case and not a common cause of that
error.

#### SSH Too Many Authentication Failures

When there are multiple keys in the authentication agent, the client
will try them against the server in an unpredictable order. If the
client happens to cycle through enough of the wrong keys first and hits
the server\'s **MaxAuthTries** limit before finding the right key, the
server will naturally break off the connection with an error message
about too many authentication failures:

``` text
"Received disconnect from 203.0.113.110 port 22:2: Too many authentication failures 
Authentication failed."
```

With increased verbosity, the keys tested and rejected will also be
shown:

``` shell-session
$ ssh -v 203.0.113.110
...
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/fred/.ssh/key.06.rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering RSA public key: /home/fred/.ssh/key.02.rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering RSA public key: /home/fred/.ssh/key.03.rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering RSA public key: /home/fred/.ssh/key.04.rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering RSA public key: /home/fred/.ssh/key.01.rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering RSA public key: /home/fred/.ssh/key.05.rsa
Received disconnect from 203.0.113.110 port 22:2: Too many authentication failures
Authentication failed.
```

Each key in the agent gets an annotation which says whether or not the
key file was supplied by the user, either in the configuration file or
as a run-time argument. The client prefers keys that were specified in
the configuration and are also currently in the agent. Then it will try
try them in the order in which they were supplied. [^3]

There are two solutions if you see the \"Too many authentication
failures\" error:

One way around this error is to remove keys from the agent one at a time
using ssh-add(1) with the **-d**
option until there is just the right key left. Refer to each key by its
file system path, for example: `ssh-add -d ~/.ssh/some.key.rsa` Because
the private key to be removed is looked up in the agent based on the
corresponding public key both files must exist. Without matching a
public key file, the private key cannot be removed individually from the
authentication agent. Instead the whole lot may be removed all at once
using the **-D** option. However, is not always practical to do either
when many remote systems are used frequently and the agent needs to be
kept well-stocked. This is probably not the most practical way.

Another way around this error, and probably the most practical method,
is to limit the client to trying only a specific key using the
**IdentitiesOnly** configuration directive in conjunction with the
**IdentityFile** configuration directive. The latter points explicitly
to the right key. Both can be added either as run-time options or in the
client\'s configuration file. As a run-time option, they can be used
like this:

``` shell-session
$ ssh -o IdentitiesOnly=yes -i ~/.ssh/server14.example.org.rsa -l fred server14.example.org
```

Or these two options could be added to the client configuration file in
something like the following way instead.

``` {.apache .numberLines}
Host server14 server14.example.org
        HostName server14.example.org
        IdentitiesOnly yes
        IdentityFile /home/fred/.ssh/server14.example.org.rsa
        User fred
```

In that way, the server could be reached with either the short name or
the fully qualified domain name, whatever names are listed under the
**Host** directive.

``` shell-session
$ ssh server14
```

Remember that options are selected from the client configuration file on
a first-match basis. Because the first-match wins, specific rules must
come before more general rules.

### Debugging Chrooted SFTP Accounts

The most common problem seems to be bad directory permissions. The
chroot directory, and all directories above it, must be owned by root
and not writable by any other user or group. Even though these
directories\' group memberships do not have to be root, if any of them
is not root then it must not be group writable either. Failure to use
the correct ownership will result in not being able to log in with the
affected accounts. The errors when login is attempted will look like
this from the client side:

``` shell-session
$ sftp fred@192.0.2.206
fred@192.02.206's password: 
packet_write_wait: Connection to 192.0.2.206: Broken pipe
Couldn't read packet: Connection reset by peer
```

The error message is much clearer on the server side: syntaxhighlighting
lang=\"text\"\> Aug 4 23:52:38 server sshd\[7075\]: fatal: bad ownership
or modes for chroot directory component \"/home/fred/\"
`</syntaxhighlighting>`{=html}

Check the directory permissions for the chroot target and all
directories above it. If even one is off it must be fixed so that it is
owned by root and not writable by any others. There are many, many
routes to get there. Here are two was to set up chroot permissions:

-   One quick way to fix the permissions is to change both ownership and
    group membership of the directory to root. Same for all directories
    above the chroot target.

``` shell-session
$ ls -lhd /home/ /home/fred/
drwxr-xr-x 3 root  root  4.0K Aug  4 20:47 /home/
drwxr-xr-x 8 root  root  4.0K Aug  4 20:47 /home/fred/
```

That will work with the **ChrootDirectory** directive set to **%h** but
has some drawbacks that will quickly become obvious when adding files or
directories.

-   Another easy way to fix the permissions is to change both the
    account\'s home directory and the **ChrootDirectory** directive.
    Arrange the account\'s home directory so that it is under a unique
    directory owned by root, such as the user name itself:

``` shell-session
$ ls -lhd /home/ /home/fred/ /home/fred/fred/
drwxr-xr-x 3 root  root  4.0K Aug  4 20:47 /home/
drwxr-x--- 3 root  fred  4.0K Aug  4 20:47 /home/fred/
drwxr-x--- 8 fred  fred  4.0K Aug  4 20:47 /home/fred/fred/
```

Then chroot the account to the parent directory and combine that with an
alternate starting directory working from the user name token with the
**-d** option for the SFTP server.

``` {.apache .numberLines}
ChrootDirectory /home/%u
ForceCommand internal-sftp -d %u
```

Then when the account connects it will see only its own directory and no
other parts of the system.

### Debugging RC Scripts Interfering with SFTP Sessions

The SFTP connection will drop if there are any extraneous data either
direction on **stdin**, from the client or the server. A common mistake
in that area is if **/etc/ssh/sshrc** or **\~/.ssh/rc** send anything at
all to **stdout** instead of being quiet. There the output, which would
be **stdout** on the server, is received by the client on **stdin**, but
matches no correct protocol and thus causes the client to disconnect.
So, even in the case of using the RC scripts, the response from the
server must remain 8-bit clean or an error will occur:

``` shell-session
$ sftp server.example.org
Received message too long 1400204832
```

That one message will be the main clue. Increasing the verbosity of the
SFTP client with **-v** won\'t provide more relevant information.

Also, the standard logs on the server will only show that the client
disconnected and not provide any information why. At higher levels of
logging, some extraneous reads and corresponding discards might be
noticed but that is all. Below is a log sample recorded at the verbosity
DEBUG3 showing such an example.

``` text
...
debug2: subsystem request for sftp by user fred
debug1: subsystem: exec() /usr/libexec/sftp-server
Starting session: subsystem 'sftp' for fred from 198.51.100.38 port 37446 id 0
...
debug2: channel 0: read 13 from efd 12
debug3: channel 0: discard efd
debug2: channel 0: read 12 from efd 12
debug3: channel 0: discard efd
debug2: channel 0: read 15 from efd 12
debug3: channel 0: discard efd
debug2: channel 0: read 18 from efd 12
debug3: channel 0: discard efd
...
```

Again, neither RC script is allowed produce any output on **stdout**
during use of SFTP or it will ruin the connection. If an RC script does
produce output, it must be redirected to a system log, to a file, or
sent to **stderr** instead of **stdout**. Regular interactive SSH
connections are not disturbed by use of **stdout** and the client will
just display whatever is sent. See the manual page for
sshd(8) in the section \"SSHRC\" for
more.

The same restriction goes for any other part of the SSH service which
runs over **stdin** and **stdout**, such as **ProxyJump** or some uses
of **ProxyCommand**. So another example of potential interference would
be when using **LocalCommand** with the client to specify a command to
execute on the local machine after successfully connecting to the
server. Any output from it also needs to be redirected to **stderr**. If
**LocalCommand** ends up interfering with **ProxyJump** then the
connection will appear to hang at the stage when **stdout** gets used.

### Debugging When An SSH Agent Has The Correct Private Key But Does Not Use It

In older versions of OpenSSH, the public key must also be available on
the client at the time the private key is loaded into the agent. If it
is not then without the matching public key the agent will not be able
to use a private key unless other arrangements are made. A symptom of
this is that while specifying the key via a run-time argument works, the
same key does not work via the agent.

Upgrading to a more recent version of OpenSSH is a better option.
Otherwise a work-around is to specify the private key either as a
run-time argument to the client or in the
ssh_config(5) file, from there
the client will find the correspondingly named public key file.
Importantly the client will still use the key in the agent yet use the
designated matching public key file, so the private key file does not
have to contain anything at all and could even be empty.

``` shell-session
$ ssh -i some_key_ed25519 fred@server.example.org
```

However, if it is undesirable to have the private key accessible on the
file system or if the private key is only in the agent and not itself
available via the file system, then the public key can be specified
directly instead.

``` shell-session
$ ssh -i some_key_ed25519.pub fred@server.example.org
```

Either way, another way is to name the key in
ssh_config(5) instead using the
**IdentityFile** configuration directive. If the file with the public
key is missing, it can be regenerated from the private key using
ssh-keygen(1) with the **-y**
option.

## SSH Client Error Messages And Common Causes

As rehash of the above, below are some client-side error messages with
some of the more common reasons[^4] for those messages. Neither list of
errors nor the reasons for the errors are comprehensive. The suggestions
for causes just touch on one or two of the commonly seen reasons. There
is no substitute for checking the actual logs, especially on the server.
The server log files are usually **/var/log/auth.log** on most system or
a variant of that name. Occasionally you will find the information in
the file **/var/log/secure** instead.

**No address associated with name:**

``` shell-session
$ ssh nonesuch.example.org 
ssh: Could not resolve hostname nonesuch.example.org: no address associated with name
```

The destination\'s host name does not exist in DNS. Was it spelled
correctly?

**Operation timed out:**

``` shell-session
$ ssh 198.51.100.89 
ssh: connect to host 198.51.100.89 port 22: Operation timed out
```

There\'s not a system associated with that IP address, or else a packet
filter is causing trouble.

**Connection timed out:**

``` shell-session
$ ssh 198.51.100.89 
ssh: connect to host 198.51.100.89 port 22: Connection timed out
```

You can\'t get there from here. It is probable that the destination
machine is disconnected from the network or that the network which it is
on is not reachable.

**Connection refused:**

``` shell-session
$ ssh www.example.org
ssh: connect to host www.example.org port 22: Connection refused
```

The destination system exists but there is no SSH service available at
the port being tried. Is the destination right? It might be the wrong
system, SSH might not be running at all, SSH might be listening on
another port, or a packet filter might be blocking connections.

**Permission denied:**

``` shell-session
$ ssh www.example.org
fred@www.example.org: Permission denied (publickey,keyboard-interactive).
```

That is usually a sign of a problem cause by the wrong user name, wrong
authentication method, wrong SSH key or SSH certificate, or wrong file
permissions in the authorized keys file on the destination system. If it
is a matter of SSH key based authentication, see the chapter on  Public
Key
Authentication
for more thorough coverage. If it is a question of SSH certificates, see
the corresponding  Certificate-based
Authentication
chapter. It can also be a matter of server-side settings with
**AllowGroups**, **DenyGroups**, or similar configuration directives at
the destination. Any of those possibilities can really only be
identified and solved by checking the log output from
sshd(8).

**No matching host key type found:**

``` shell-session
$ ssh www.example.org
Unable to negotiate with 198.51.100.89 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss
```

The SSH daemon on that server is really outdated. Contact the system
administrator about an upgrade and get them moving. More details can be
seen with the **-v** option on the client.

``` shell-session
$ ssh -v fred@example.org
...
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: (no match)
Unable to negotiate with 198.51.100.89 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss
```

In the above shell session, it is the server which is badly in need of
an update as shown by the deprecated key exchange algorithms which it
tries to use. Again, for emphasis, the solution is to update the
outdated software. If a specific, old version of an particular operating
system absolutely must be used for a while longer, see about getting a
back port of the SSH daemon. Many GNU/Linux distros even have specific
back port repositories.

If you would like to see which key exchange algorithms the client
supports try using the **-Q** option.

``` shell-session
$ ssh -Q kex
diffie-hellman-group1-sha1
diffie-hellman-group14-sha1
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group18-sha512
diffie-hellman-group-exchange-sha1
diffie-hellman-group-exchange-sha256
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
curve25519-sha256
curve25519-sha256@libssh.org
sntrup761x25519-sha512@openssh.com
```

 

[^1]:
    
[^2]:
    
[^3]:
    
[^4]:
    