Disabling SSH Password Login: Solving cloud-init Reverts
When operating cloud instances, SSH security is always a top concern. Recently, to enhance server security, I tried to disable password login and use only SSH key authentication. I changed PasswordAuthentication to 'no' in the sshd_config file and restarted the sshd service, but surprisingly, I could still connect with a password. Why weren't my settings applied? After much troubleshooting, I finally found the reason and solved the problem. I want to share the issue I faced and the resolution process for anyone in a similar situation. I hope this experience helps you in your server operations.
What this post covers
The role of the /etc/ssh/sshd_config.d/ directory in sshd configurationHow cloud-init manages SSH settingsCommon reasons why SSH password login disabling might not applyHow to verify actual SSH settings using the sshd -T commandSafe procedures for changing SSH settings to prevent locking yourself out of the server*Target Audience:* Developers and system administrators who are puzzled why password login isn't disabled after modifying sshd_config*Difficulty:* Intermediate
Problem: Password login persists despite sshd_config modification
To enhance the security of my cloud instance, I decided to disable SSH password login and use only key authentication. Following standard procedures, I opened the /etc/ssh/sshd_config file and changed the PasswordAuthentication setting from 'yes' to 'no'. Then, I restarted the SSH service using the systemctl restart sshd command. However, when I tried to log in with a password from a new terminal session, the password prompt still appeared, and I could connect successfully. It seemed as if the changes weren't applied at all, despite modifying the settings and restarting the service. Rebooting didn't help, and I checked for typos in the configuration file multiple times but found no issues. It was a frustrating and perplexing situation.
Root Cause Analysis: cloud-init Drop-in File Priority
To find the root cause, I revisited how sshd reads its configuration. I recalled that while sshd typically reads /etc/ssh/sshd_config, modern Linux systems like Ubuntu also read additional configuration files (drop-in files) within the /etc/ssh/sshd_config.d/ directory. These drop-in files are loaded later than the main configuration file or can override main settings based on specific priorities. In cloud instances, the cloud-init service often creates files like 50-cloud-init.conf in the /etc/ssh/sshd_config.d/ path to manage SSH settings during initialization. This file indeed existed on my instance, and it explicitly contained 'PasswordAuthentication yes'. The setting in this drop-in file was overriding the 'no' setting I applied in /etc/ssh/sshd_config. I found the problematic file using the following command: sudo grep -ri passwordauthentication /etc/ssh/sshd_config /etc/ssh/sshd_config.d/ This command clearly showed that 'PasswordAuthentication yes' was included in the 50-cloud-init.conf file, confirming the issue.
Problem Resolution: Directly Modifying cloud-init Drop-in File and Safety Procedures
Once the cause was identified, the solution was clear. I needed to change 'PasswordAuthentication yes' to 'no' in the /etc/ssh/sshd_config.d/50-cloud-init.conf file, or comment out/delete that line. However, a crucial safety procedure is involved here. Before disabling SSH password login, you must thoroughly verify that SSH key authentication is working correctly. If key authentication is not properly set up and you disable password login, you could face a severe situation where you lock yourself out of the server. I kept my current SSH session active and opened a new terminal to attempt connecting with an SSH key. Only after confirming that key access worked correctly did I edit the /etc/ssh/sshd_config.d/50-cloud-init.conf file. Contents of /etc/ssh/sshd_config.d/50-cloud-init.conf file: PasswordAuthentication no After making this change, I restarted the sshd service again: sudo systemctl restart ssh This process ensured the configuration change was applied. Since I had already verified key access, I could proceed with the work with peace of mind.
Final Verification: SSH Connection Test and sshd -T for Applied Values
After completing the configuration change and service restart, the next step was to verify that the changes were properly applied. First, I opened a new terminal and attempted to connect using an SSH key. Key authentication still worked without any issues. Next, I tried to log in with a password. As expected, I received a 'Permission denied (publickey).' message, confirming that password access was rejected. This meant password login was successfully disabled. Finally, I used the sshd -T command to check the final configuration values actually applied to the sshd service. This command outputs the final configuration that sshd uses, combining all configuration files and drop-in files, making it the most reliable way to confirm actual application. sudo sshd -T | grep -i passwordauthentication When I ran this command, the output showed 'passwordauthentication no'. This conclusively verified that the SSH password login issue caused by the cloud-init drop-in file was completely resolved. This method will likely be very useful for future SSH configuration problems.
Conclusion
Through this experience, I learned the important lesson that the SSH daemon's configuration is not solely determined by the sshd_config file but must comprehensively consider drop-in files within the sshd_config.d/ directory. Especially in cloud environments, it's crucial to remember that provisioning tools like cloud-init can create these drop-in files and override default settings. After making configuration changes, it's good practice to always use the sshd -T command to verify the actual final values applied. And most importantly, before disabling SSH passwords, always verify key access in a separate session to prevent accidentally locking yourself out of the server. I hope this record helps other developers facing similar issues.
