Title: Demystifying SSH
Date: 2019-06-17
Tags: ssh
Slug: demystifying-ssh
Description: Setting up and understanding remote terminal connections to your server using SSH for fun and not-for-profit
Author: decentral1se
Status: published
Category: fundamentals

## Introduction

Some of the essential things that separate a server from other computers is that first they are usually not where you are and second that often come without screen and keyboard. With homebrewservers this is particularly the case when using Single Board Computers (SBC).

In order to use a server you need to connect to it over the network using the command-line interface or shell. This is usually done with a program called SSH which stands for Secure Shell.

One of the more important and foundational skills needed for experimenting and
maintaining servers is understanding, using and troubleshooting SSH.

Setting up and using SSH can be a challenge at first. There are many moving parts to
consider. Working with SSH means knowing something about user accounts, permissions, public key cryptography, protocols, clients, servers and agents. And yet despite so much to consider, SSH is praised as something easy to use and quite often presupposed knowledge
between peers.

With this in mind, there is a need to demystify SSH. In this guide we aim to
show how to setup an SSH client and server as well as discuss common issues and
approaches for day to day use and troubleshooting.

## Prerequisites

A basic understanding of [the command-line] is required.

[the command-line]: http://write.flossmanuals.net/command-line/introduction/

The SSH ecosystem is very well established. It is available on all modern GNU/Linux
distributions, MacOS and Windows. You can use your home server or if you don't have one yet you can use your own personal laptop to experiment (in this case, your laptop will
play the role of both the server and client as explained later).

The commands shown in this guide were run on a [Debian Stretch] distribution
but the actual tool names should be the same on other distributions.

[Debian Stretch]: https://en.wikipedia.org/wiki/Debian_version_history#Debian_9_.28Stretch.29

## SSH and OpenSSH

The term "SSH" can refer to a number of different but related things.

SSH stands for the [Secure Shell] which is a protocol for describing how to
provide a secure channel of communication from a client to a server. However,
it more typically refers to the [OpenSSH suite of tools] which are the programs
you will install and use when dealing with SSH.

[Secure Shell]: https://en.wikipedia.org/wiki/Secure_Shell
[OpenSSH suite of tools]: https://en.wikipedia.org/wiki/OpenSSH

The OpenSSH tools are an implementation of the SSH protocol. This means that
these tools follow the behaviour described in the documents of the protocol.

## Installing the SSH server and client

It is important to understand the client/server architecture of SSH. If you are
remotely connecting to your home server from your laptop, then your laptop is
the client and the home server is the server.

There are two packages which contain all the tools that the OpenSSH tool suite
provides. The [openssh-server] and [openssh-client] packages.

[openssh-server]: https://packages.debian.org/stretch/openssh-server
[openssh-client]: https://packages.debian.org/stretch/openssh-client

To install the SSH server on your home server, run:

```bash
$ sudo apt install -y openssh-server
```

And on your client, run:

```bash
$ sudo apt install -y openssh-client
```

## Creating a user account on the server

In order for your client to connect to the server, a user account must be
created on the server. This can be done with the following command (where
`homebrewer` is the new user account):

```bash
$ sudo useradd --create-home --shell /bin/bash homebrewer
```

You should then set a password for this new user:

```bash
$ sudo passwd homebrewer
```

This password will be used in the following step.

## Logging in for the first time

You can now SSH into the server from your client. Assuming that you have a DNS
entry which points to your home server (`myhomebrewserver.com`) then you can
log in with:

```bash
$ ssh homebrewer@myhomebrewserver.com
```

On first connection to a new server, you will be shown a fingerprint and asked
if you would like to continue connecting. For now, choose to continue without
validating the fingerprint.

You should then be asked for the password that you entered when creating the
account. If you have any issues at this point, please see the [Troubleshooting
SSH] section.

[Troubleshooting SSH]: #troubleshooting-ssh

## Passwords, keys and security

As we have seen so far, connecting to an SSH server using password
authorisation is relatively simple. However, password authorisation is
typically recommended against due to [security considerations].

Security is relative and you may not be concerned with defending against a
[brute-force attack]. However, since other methods of authorisation are so
commonly used and often the source of of problems when dealing with SSH
connectivity, we will also cover this topic also.

[security considerations]: http://docs.hardentheworld.org/Applications/OpenSSH/#disable-password-authentication
[brute-force attack]: https://en.wikipedia.org/wiki/Brute-force_attack

The alternative to password authorisation is [public key cryptography]. The
idea is to generate two keys, one public and the other private. The public key
part can be shared publicly and will be registered with the SSH server. The
private key part cannot be shared with anyone else and will be used on the
client side for authentication.

[public key cryptography]: https://en.wikipedia.org/wiki/Public-key_cryptography

For those not familiar with public key encryption, a brief and pragmatic
introduction is provided by [this ssd.eff.org guide].

[this ssd.eff.org guide]: https://ssd.eff.org/en/module/key-concepts-encryption#1

## Validating server host keys

When logging into your server you were asked to validate a key fingerprint:

```
The authenticity of host 'myhomebrewserver (666.666.666.666)' ed.
ECDSA key fingerprint is SHA256:lUhBmQXjkOL0zyoDNarUIbhd3RXo7X
Are you sure you want to continue connecting (yes/no)?
```

When an SSH server is installed, key pairs are generated on the server. The
design of SSH is such that when you make a secure connection you should be sure
that where you are connecting to is the server you expect.

On the server, you can validate that this is the fingerprint you expect:

```bash
$ sudo ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key
256 SHA256:lUhBmQXjkOL0zyoDNarUIbhd3RXo7X root@myhomebrewserver (ECDSA)
```
These values should match the ones you were first shown. Where the `ECDSA` in
the first message corresponds to the key file name in `/etc/ssh/`. The SSH
server will generate key pairs for each algorithm it supports.

As you have accepted your server host key fingerprint, the public key of the
server will be placed in your `$HOME/.ssh/known_hosts` file on your client
where it will be remembered for future connections.

If the host key ever changes, you will see something like:

```
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...
```

This simply means that that the host key you accepted for this host has changed
and you should first figure out why this is the case before making a secure
connection.

## Dealing with SSH keys

### Generating a public and private key pair

Now that we understand the idea of host keys, we should generate client keys.
On your client, run the following:

```
$ ssh-keygen -t ed25519
```

This uses the [currently preferred algorithm] for generating keys. The key
generation will ask you to enter a passphrase. It is typically recommended use
a strong passphrase (see [diceware]) for the reason that if someone else gets
your private key part, they still will still not be able to use it because they
also need to know the passphrase.

[currently preferred algorithm]: https://wiki.archlinux.org/index.php/SSH_keys#Generating_an_SSH_key_pair
[diceware]: https://www.rempe.us/diceware/#eff

You should now have two keys in your `$HOME/.ssh` folder. A `id_ed25519`
(private key part) and a `id_ed25519.pub` file (public key part).

Your public key part might look something like:

```
$ cat $HOME/.ssh/id_ed25519
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOTWIy+Em89QR/Xs6RUmr+0U0F3GVXY/UA2MXs9MoqdY myuser@myhost
```

You can then start an `ssh-agent` instance and register the private key part:

```bash
$ eval "$(ssh-agent -s)"
$ ssh-add $HOME/.ssh/id_ed25519
```

The `ssh-agent` should ask for your passphrase. The agent is useful because it
will store the passphrase of your key and re-use it the next time you log into
the server. This can be particularly useful later on when you need to use
multiple SSH keys.

You can confirm that the `ssh-agent` has loaded this key by running:

```bash
$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOTWIy+Em89QR/Xs6RUmr+0U0F3GVXY/UA2MXs9MoqdY myuser@myhost
```

And finally, you can then configure your client to use this key. Create a
`$HOME/.ssh/config` file and put something like the following configuration
into it (remember to replace with your server details):

```bash
Host myhomebrewserver
  Hostname myhomebrewserver.com
  User homebrewer
  Port 22
  IdentityFile ~/.ssh/id_ed25519
```

This is useful because it allows you to only type:

```bash
$ ssh myhomebrewserver
```

And the correct hostname, user, port and key will be chosen. However, before
connecting, you must register your public key part on the server.

### Register the public key with the server

On the server, you should register the public key. This is achieved by putting
the public key part of the key into the `/home/homebrewer/.ssh/authorized_keys`
file.

You can do this like so (replace the public key part with the one you generated):

```bash
$ sudo -su homebrewser
$ cd $HOME && mkdir .ssh
$ echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOTWIy+Em89QR/Xs6RUmr+0U0F3GVXY/UA2MXs9MoqdY myuser@myhost" > .ssh/authorized_keys
```

You must then ensure that your SSH server is accepting keys as a method of
authentication. You should edit the `/etc/ssh/sshd_config` file which is the
main configuration file for the SSH server.

You need to ensure the following is present:

```bash
PubkeyAuthentication yes
```

After editing the file, save your changes and then validate the configuration:

```bash
$ sudo sshd -t
```

No output should be shown if everything is validating correctly. This is
important to do before restarting the SSH server in case you lock yourself out
of the server.

You can restart the SSH server with:

```bash
$ sudo systemctl restart sshd
```

Now it should be possible to connect from your client with:

```bash
$ ssh myhomebrewserver
```

If you have any issues, please see the [Troubleshooting SSH] section.

[Troubleshooting SSH]: #troubleshooting-ssh

## Troubleshooting SSH

Despite our best intentions, we are often confronted with a failed login attempt:

```
$ ssh myhomebrewserver
Permission denied (publickey).
```

This is not the most helpful message. In order to go about solving this issue,
we need to focus our detective work on the client/server architecture and try
to discover which side is responsible for the problem. Hopefully the following
tips can help you in this process.

### On the server

Here are some questions to ask yourself:

  * Is your public key registered on the server in the `$HOME/.ssh/authorized_keys` folder?

  * Are the `$HOME/.ssh` permissions correct? (see this [SO thread](https://superuser.com/questions/215504/permissions-on-private-key-in-ssh-folder))

  * Is the SSH server running? (`sudo systemctl status sshd`)

  * Is the `/etc/ssh/ssd_config` correct?
    * Use the [sshd_config](https://man.openbsd.org/sshd_config) as a
      reference. Don't forget to run `sudo sshd -t` to validate your
      changes. After validating, always restart the SSH server after
      making configuration changes

  * What does `sudo tail -f /var/log/auth.log` say?
    * Every time an SSH client attempts to connect, a log
      entry will be put into this file. Some hint as to why
      it is failing should be here.

### On the client

Here are some questions to ask yourself:

  * What does `ssh -vvvvv myhomebrewserver` tell you?

  * Are the `$HOME/.ssh` folder permissions correct? (see this [SO thread](https://superuser.com/questions/215504/permissions-on-private-key-in-ssh-folder))

  * Is the SSH server available at the port you expect? (`sudo nmap -p 22 myhomebrewserver`)

  * Is your `$HOME/.ssh/config` correct?

  * What is registered with the local `ssh-agent`?
    * Check which keys are registered with `ssh-add -L`. If you
      see unrecognised keys, you can kill and restart the agent:
      * `sudo pkill ssh-agent`
      * ` eval "$(ssh-agent -s)"`
      * `ssh-add ~/.ssh/id_ed25519`

## Conclusions

As we can see, SSH is no simple topic! There are many moving parts and a number
of topics which require familiarity in order to be able to get remote access to
your server.

## Further Reading

  * [/etc/ssh/sshd_config reference](https://man.openbsd.org/sshd_config)
  * [SSH server hardening](http://docs.hardentheworld.org/Applications/OpenSSH/)