Why SSH Password Login Persisted: The cloud-init Drop-in Configuration Trap
There was a time when I tried to disable SSH password login and use only key authentication to enhance server security. Despite changing the PasswordAuthentication setting to 'no' in the /etc/ssh/sshd_config file and restarting the sshd daemon, I still encountered a situation where password access was possible. I was frustrated that the configuration file I modified wasn't being applied. It made me pause, thinking that some unexpected setting was being enforced. Even though I followed the usual SSH configuration change procedure, I spent some time figuring out why I wasn't getting the desired result. This problem often occurs in cloud environments. I finally found some time on the weekend to organize my findings.
What this post covers
Understanding how OpenSSH configurations are merged from multiple filesIdentifying the mechanism by which cloud-init overwrites SSH settingsHow to check the actual applied SSH configuration using the sshd -T commandFollowing the safe procedure to disable SSH password login*Target Audience:* Developers or system administrators who are confused because their SSH password login disablement settings are not being applied.Difficulty: Intermediate
Problem Situation: The perplexity of password login persisting after sshd_config changes
I planned to enhance server security by disabling SSH password login and allowing only key authentication. Following the standard procedure, I opened the /etc/ssh/sshd_config file, changed the PasswordAuthentication entry to 'no', and restarted the sshd service with sudo systemctl restart ssh. However, I was surprised to find that I could still log in with a password. It was difficult to understand why the changes weren't applied, even though I had clearly modified the settings and restarted the service. I checked the file several times for typos and carefully looked for any commented-out lines. But the problem wasn't in the sshd_config file itself. I had a hunch that the settings were being overwritten elsewhere. This is a common situation you might encounter when initially setting up cloud instances.
Root Cause Analysis: The Trap of cloud-init Generated Drop-in Configurations
The root cause of the problem was that sshd doesn't just read the /etc/ssh/sshd_config file; it also reads additional configuration files within the /etc/ssh/sshd_config.d/ directory. In a cloud instance environment, a tool called cloud-init handles initial setup during boot. In this process, it creates a drop-in file named something like 50-cloud-init.conf in the /etc/ssh/sshd_config.d/ path. This file explicitly contained the PasswordAuthentication yes setting. When sshd merges multiple configuration files, settings in files read later often take precedence. In my case, the 'yes' setting in the drop-in file created by cloud-init took precedence over the 'no' setting in the sshd_config file I modified, keeping password login active. This hidden file made my main configuration changes ineffective. I used the grep command to find the problematic file: sudo grep -ri passwordauthentication /etc/ssh/sshd_config /etc/ssh/sshd_config.d/. This command searches for the string 'passwordauthentication' (case-insensitive) in /etc/ssh/sshd_config and all files within the /etc/ssh/sshd_config.d/ directory. This allowed me to confirm the existence of the 50-cloud-init.conf file.
Resolution Process: Directly Modifying cloud-init Drop-in File and Safe Application
Once the cause was identified, the solution was simple: directly modify the /etc/ssh/sshd_config.d/50-cloud-init.conf file generated by cloud-init. I changed the PasswordAuthentication yes line within that file to PasswordAuthentication no, or I commented out or deleted the line entirely. However, it's crucial to follow important safety procedures during this process. Before disabling password login, you must verify that key authentication is working correctly. I kept an existing SSH session open and then opened a new terminal to attempt connecting to the server using my key file. Only after confirming that key access worked without issues did I proceed with disabling password login. This is because if key access doesn't work and you disable passwords, you might permanently lose access to your server. After modifying the configuration file, you need to restart the sshd service for the changes to take effect. # Change cloud-init drop-in's 'yes' to 'no' # Modify the corresponding line in /etc/ssh/sshd_config.d/50-cloud-init.conf. PasswordAuthentication no sudo systemctl restart ssh I restarted the sshd service with the command above to apply the changed settings.
Verification and Results: Confirming Final Settings with sshd -T and Connection Test
After changing the settings and restarting the service, it's important to verify the final configuration actually applied to sshd. The sudo sshd -T command is very useful here. This command shows in detail how sshd is currently operating and outputs the final merged result of all configuration files. Through this result, I could clearly confirm that the PasswordAuthentication setting was indeed applied as 'no'. sudo sshd -T | grep -i passwordauthentication I was relieved to see 'passwordauthentication no' output when I ran this command. This confirmed that the final settings were correctly reflected. Afterwards, I performed a connection test again from a new terminal. SSH connection was successful with the key file, and when I intentionally tried to connect using a password, I received a 'Permission denied' message, confirming that access was rejected. Thus, the goal of enhancing server security was successfully achieved. In such complex configuration environments, it's crucial to always verify the final applied values directly.
Conclusion
When dealing with SSH configurations in a cloud environment, I learned that you shouldn't just look at the /etc/ssh/sshd_config file, but also check the drop-in files within the /etc/ssh/sshd_config.d/ directory. Configuration files generated by initialization tools like cloud-init can act as unexpected variables. The most important thing is to always verify the actual application of changed settings using the sudo sshd -T command and to follow the safety procedure of first verifying key authentication access in a separate session before disabling password login. I need to keep this well-documented to avoid repeating such mistakes, even when busy.
