Imagine the following situation: You care for live servers and work in a team of let’s say five, six or even more people. Access to the servers is granted through ssh. The people login either as root (yes, you should not do that, but that is not the point here) or as user with sudo rights or they just share an unprivileged account. Authentication is done via ssh keys.
Now somebody leaves your team. Either as he has a new job or he just got fired. Of course you start deleting his key from all those ~./ssh/authorized_keys files. You have been smart before as you forced your buddies to use their real name or mail address as comment in the key. Easy identification.
But then you start thinking: How do I know I am deleting the right keys? Let’s say the target user is a smart bad guy. He just might have done the following: He looks for somebody who seldomly logs in. Maybe a manager has a key just for security purposes or something like that. Now he exchanges the order of the keys and its comments if they are in a shared authorized_keys or he even exchanges the authorized_keys files when they belong to different users, so you just think you are deleting the right keys but disable another person – in the worst case even yourself.
Of course you can start working around this with trip wire, shell scripts and so on, but be honest: Being able to change the comment in an ssh key without disturbing a checksum or even a signature that rings bells and whistles is a pain for every security minded administrator.
Feel free to hint me an easy solution for this that you might already have implemented.
Why bother checking the comment at all? Just match the public key itself using some simple grep syntax.
Be honest: Are you doing this? Actually I prefer hard crypto stuff that prevents you from making grave errors.
Each person should login with their own account with sudo privileges, and root login is completely disabled (in /etc/ssh/sshd_config ).
Then there is a separate authorized_keys file for each user, so just delete the relevant file. You should also remove the user in question from the sudoers file (or remove them from the group that has sudo privileges).
Of course it is possible that a user having root privileges could set up any number of back doors. One that springs to the mind for the above set up is that the user could
sudo su another_user
and add their ssh_key to that users authorized_keys file. But with decent logging you have some chance of catching them, depending how competent they are.
No logins as root, user authorized keys in ldap
What you say is very true – what if an admin just add his keys to other people accounts, with false comments: would this be noticed? I guess not…
The outside world should not be able to access said machine(s) directly over the internet anyway, besides default ports like 80, 443, 53, 25 and maybe 993/995?! Only allow certain machines to access the server(s) on other ports, easy to set up using IP tables.
If they had root and you don’t trust them, you’re basically screwed. There are a lot of schemes folks could come up with to ensure ssh authorized_keys are fair (require everyone to send their public key to you, store them on a secure machine, periodically run a script to check for non-matching keys in each user’s ~/.ssh, etc.), but that isn’t sufficient. For example, the attacker could have installed a kernel module that deceives you or circumvents any block you place against them.
When your admins leave, you do your best to close the doors behind them but you must simply accept that they might be smarter than you and still have a way in. Unless you’re willing to nuke all affected servers down to the ground and re-build them, you’re just kidding yourself.
Why don’t you manage SSH keys using some database backend? Every user can upload his key using some PHP frontend. The key is therefore stored in DB and authorized_keys files are refreshed daily using CRON job. You, as an administrator, have privileges to manage every key assigned to any person. Users can only upload/change their own keys.
We manage key files via a package. We create a new install file (rpm or dpkg) with the updated key files. We place this in a repository and the patching process applies the changes to the key files. This also allows you to audit compliance and keep track of version history. You can use cvs/svn to track the changes to the key files…
Thanks for your comments, guys. Let me add that I know, of course, there is no possibility to enforce a trusted situation when other people have root and even when they don’t – if they are smart enough. The point is how difficult it is to cause trouble, how techie you have to be to get an idea of how to change things for your own advantage. Theoretically, I agree, everybody can do everything, but as we know, every additional barrier is a good choice.
So what I’d really like to see is something in the ssh daemon that matches some of your suggestions. Registering ssh keys, signing them, and stuff. Of course there are alternatives like ldap but when you are an agency caring for dozens of live servers distributed about a couple of providers you cannot centralize and have to deal with a lot of isolated machines.
I think what comes quite close is the suggestion of Jim K – a packet management driven delivery of ssh keys. I also thought about a check script that greps all home dirs from a /etc/passwd, adds the authorized_keys path to it and checks if these files exist and what they contain – kind of current access summary. So in the end it is about writing your own scripts. As I don’t seem to be isolated in my task I hope there will come up some official stuff, soon that prevents us from inventing the wheel again and again.
Thank you, guys.
After you delete the key, Keep a copy the key under your own personal home folder. If you delete the wrong key, you have a backup.
I love how people who administrate little environments glibly say everybody should use sudo 100% of the time.
Now try doing adhoc system administration tasks on 2000 servers under time constraints and tell me how „log on as yourself and use sudo“ scales for you.
This problem doesn’t just exist for „root“, either. You may have other less-privileged accounts for which such remote administration is necessary, not to mention file transfer capabilities which may by necessity be exposed outside the firewall.
I personally find that a combination of making authorized_keys read-only for all but root, and using properly configured sudo policies to be a good process here. It won’t catch issues 100% of the time, but it does give a reasonable level of security without wasting too much time on various methods of tracking – you could take this a step further and add either a COW or Versioning system to manage the updates to authorized_keys, that goes beyond my requirements though
As to syberghost’s comment, I agree that root is often the simplest way to manage a large deployment, however I don’t agree with flippant use of sudo or giving root out to anything but trusted senior admins.
Quite apart from potentially intentional harm, a simple slip on the commandline by an inexperienced user when running with open sudo privs/as root can do a LOT of harm
actually just found this post, point three takes my process one step further by moving the file completely out of the users domain
http://it.toolbox.com/blogs/unix-sysadmin/playing-with-openssh-public-keys-28377
You should utilize the fingerprint of the SSH key. The fingerprint IS the „checksum“ you’re talking about. Changing the comment does NOT change the fingerprint.
Check the fingerprint of the keys and only remove keys with the matching fingerprint. Checking the fingerprint of ssh keys should be common practice anyway – since it is the best way to verify the owner of a key over the phone.
This is very basic public key verification practice.
You can just add a comment to the end of the key like this so you dont forget who the key belongs to:
ssh-rsa AAA…Gi4w== someone@blah WRITECOMMENTHERE
@manpage authorized_keys:
Each line in these files contains the following fields: hostnames, bits, exponent, modulus, comment. The fields are separated by spaces.
Cheers
this of course only works ok when you fix security like DESERTED mentions. This should be ok then i guess, or do I overlook some security-issue here?
I know its been quite some time since this post, but another solution is to use certificate authentication. This will allow you to have one trusted CA server with strict access that will generate user certificates. By doing this you have the option to give different users specified certificate lifespans via cert expiration, and any users that need to be removed before the cert expiration can have their cert placed on revokation list. For more info check out the CERTIFICATES section of ssh-keygen(1).
This is where configuration management (Chef, Puppet, Ansible, etc) comes in handy.
In Chef, you store the keys in encrypted databags and if you need to remove a key, you remove it from the databag and run the recipe that updates authorized_keys. Problem solved.