Hardening SSH access

After the initial setup we harden the SSH access – the most important door to your server.

What are we building?

By the end you log in with an SSH key instead of a password, and neither root login nor password login is possible from outside. This makes the automated password-guessing attacks that hit every publicly reachable server around the clock run completely into the void. Tested with OpenSSH 10.0p2 on Debian 13.

Prerequisites

A set-up server with a sudo user
Access to the VNC console in the netcup SCP as a safety line in case you lock yourself out

Step by step

Step 1: Generate an SSH key pair

If you don't have a key yet, generate an Ed25519 key pair on your own machine (not on the server!):

ssh-keygen -t ed25519

You can accept the suggested storage location with Enter. Set a passphrase – it protects the key if your machine falls into the wrong hands:

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/YOUR_USER/.ssh/id_ed25519):
Created directory '/home/YOUR_USER/.ssh'.
Enter passphrase for "/home/YOUR_USER/.ssh/id_ed25519" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/YOUR_USER/.ssh/id_ed25519
Your public key has been saved in /home/YOUR_USER/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:... koch@your-laptop

The Created directory line only appears if you didn't have a ~/.ssh yet. After that comes the "randomart image", a small ASCII graphic of the fingerprint – you can ignore it. What matters are the two files: ~/.ssh/id_ed25519 (private, stays with you) and ~/.ssh/id_ed25519.pub (public, goes on the server).

Step 2: Transfer the public key

ssh-copy-id appends your public key to the ~/.ssh/authorized_keys file of your server user – with the correct file permissions:

ssh-copy-id koch@YOUR_SERVER_IP

The server asks for the user's password one last time here – after this it won't again:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/YOUR_USER/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
koch@YOUR_SERVER_IP's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'koch@YOUR_SERVER_IP'"
and check to make sure that only the key(s) you wanted were added.

Number of key(s) added: 1 is the success message. On a second run you get WARNING: All keys were skipped because they already exist on the remote system. instead – then the key is already in place. Test right afterwards that key login works:

ssh koch@YOUR_SERVER_IP

You should now be logged in without a server password (only the passphrase of your key may be requested). Only once that works do we continue – in the next step we disable password login.

Step 3: Disable password login and root login

Open the SSH server configuration:

sudo nano /etc/ssh/sshd_config

Set (or uncomment) these two lines:

PermitRootLogin no
PasswordAuthentication no

PermitRootLogin no blocks direct root login completely, PasswordAuthentication no allows only key logins.

⚠️ Warning

On cloud images there are often extra files under /etc/ssh/sshd_config.d/ that override these settings. Check with grep -r PasswordAuthentication /etc/ssh/sshd_config.d/ and adjust any matches as well.

Step 4: Test the configuration and restart SSH

Check the configuration for syntax errors before you restart – a typo would otherwise take the SSH service down:

sudo sshd -t

No output means: all good. Then apply it:

sudo systemctl restart ssh

🛑 Caution

Test key login now in a second terminal session before you close the first – otherwise you might lock yourself out.

To verify: sudo systemctl status ssh should show active (running), and a login attempt as root must now be rejected with Permission denied (publickey).

When things go wrong

SSH keeps asking for the password despite ssh-copy-id. Usually the file permissions on the server are wrong: SSH ignores authorized_keys if group or others may write to ~/.ssh. The log on the server (sudo journalctl -u ssh --since -5min) then shows Authentication refused: bad ownership or modes for directory /home/koch/.ssh. Fix it with chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys, and also check that you're connecting with the right user (koch@…, not root@…).

Permission denied (publickey) – and you can't get in at all anymore. Password login was disabled before the key worked. No drama: open the VNC console in the netcup SCP, log in locally there, set PasswordAuthentication yes, restart SSH and start again at step 2.

After the restart the SSH service no longer starts. Syntax error in the sshd_config – a mistyped PasswordAuthentification is enough (that's why you always run sshd -t before restarting). Log in via the VNC console; sudo sshd -t then names file, line and option: /etc/ssh/sshd_config: line 125: Bad configuration option: PasswordAuthentification, followed by /etc/ssh/sshd_config: terminating, 1 bad configuration options.

The settings seem to take effect, but password login still works. A file under /etc/ssh/sshd_config.d/ (often 50-cloud-init.conf) overrides your values. sudo sshd -T | grep -i passwordauthentication shows the actually effective setting – adjust matches in the extra files and restart SSH.

Maintenance & backups

Back up the private key: Without ~/.ssh/id_ed25519 you can only reach the server via the VNC console. Back up the key encrypted (e.g. in a password manager) or add a second key from another device to authorized_keys.
Keep an eye on logins: sudo journalctl -u ssh --since today shows you all login attempts. OpenSSH 10 logs the individual connections under the name sshd-session, for example as Connection closed by authenticating user root 203.0.113.10 port 50718 [preauth]. Failed attempts on port 22 are normal and, thanks to the key requirement, harmless – you can automatically ban such bots later with Fail2ban.
Updates: OpenSSH gets its security updates through the normal apt upgrade – so keep the server up to date, ideally automated.