Large Scale SSH
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Lucas, Michael W.
SSH is arguably the most widely deployed systems administration tool. It's also arguably the least well configured. Many sysadmins already know how to disable passwords and chroot users, but when it comes to managing host and user SSH keys, we rely on manually copying files around. That's fine with three servers and two sysadmins, but not so good with three thousand servers and two hundred sysadmins. We'll fill in those gaps. SSH is arguably the most widely deployed systems administration tool. It's also arguably the least well configured. Many sysadmins already know how to disable passwords and chroot users, but when it comes to managing host and user SSH keys, we rely on manually copying files around. That's fine with three servers and two sysadmins, but not so good with three thousand servers and two hundred sysadmins. This talk takes you through: Key file formats Distributing the host key cache to OpenSSH and PuTTY clients Distributing client configurations OpenSSH configuration for automatic deployment and cloud configuration Improving the security of known_hosts SSH keys for automation Building and deploying SSH Certificate Authorities Attendees will leave understanding how to scale their SSH systems-and, more importantly, when to scale those systems to the next level. Based on the book "SSH Mastery, 2nd Edition"
 Okay. So just to double check, I'm going to give it just another moment. Are there any open SSH developers in the room? No, no, because Colin, we pretty much ignore you. What Okay. Maybe you don't have LDAP. Everybody's network is a little different. The important thing here is that authorized keys command is a script. Do you have a database on the network, any kind of directory, any anything? You know, you are a sysadmin. Scripts are your thing. If your company has a standard where you are using, you know, you provide all your data via WordPress, XML, RPC over HTTP via an SSH tunnel, that's fine. An Excel spreadsheet on a sysvol, whatever. Just write a script to access it. Host keys. Host keys are theakan works. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. NONE hosts to all of the client machines or you can have the client look up NONE hosts. Once you're at the point where you're distributing this, you can also set global client SSH settings in SSH underscore config. So if your organization does something like run SSHD on and off port or you have special requirements or you need this little flag twiddled on all of the clients everywhere, you set it globally and it works for everyone. So when it comes to clients, the most common one other than SSH, sorry, other than open SSH is SSH is Putty. Putty of course stores its host keys in the registry because it's Windows and you put everything in the registry. Why wouldn't you? Putty ships with a Python script to convert NONE hosts into registry keys. Run the script, you have a reg file, give it to your AD admin and eventually they'll put it in user login scripts and distribute it. Now, if your AD admin is slow, you can always do things like email the registry file to all your users and teach them to blindly double click on email registry files before using SSH. I'm going to recommend subject lines like this one weird trick makes your Putty work. Putty admins hate when you click on this or if you do this once you may find your AD administrator becomes much more responsive. So if you don't want to distribute, you can have the SSH client check on the network through SSHFP DNS records. Now, people tell you this requires DNS sec. Well, I hear folks arguing against that. If you want to shuffle security critical data around in unencrypted UDP across the network that's just begging to be spoofed. Okay, it's not my network. So, you generate an SSHFP record with SSH key gen minus R in the host name and it spits out a bunch of SSHFP records. Put those in your zone file. If you are using key files from a different host, add the minus F flag. So, that's fairly straightforward for traditional SSH use. Or you look them up from the network, client and server alike. SSH certificates are something very, very different. And specifically, they're different from the TLSX509 certs that you all know and loathe from web servers and other applications. So, what's a certificate authority? It's really just a method of delegating trust. It says, I trust this thing. I trust Verif sign. I trust Let's Encrypt. I trust Wems House of Cheap Certs. Whatever. So, and it takes two feature, two technological features. Encryption and signing. And these features are both in regular SSH keys. So, an SSH certificate authority is just an SSH key that you decide you're going to use as a certificate authority. It really is exactly like a self-signed TLS certificate. Because you don't want anyone outside your organization to trust this. There is no reason why hosts and clients on my network should trust certificates issued by Curt's network. Curt has his own network and I really, by design, I should not trust it. And it's not Curt's fault. It's just we don't need external trust. So, certificate authorities, if your network is large enough to require them, then you really do need automation. You have to be able to copy files and restart SSHD reliably. Look at Ansible if you haven't. So, one thing you see in a certificate is an expiration date. It's real tempting when you do a self-signed certificate to give it a 25-year expiration date. But they will be insecure. It still contained the first ever SSH version 1 user authentication public key I had ever created. And I really only noticed it when the FreeBSD cluster disabled SSH v1. And suddenly I couldn't get into free fall and oops. What? Yes. So, remember you're going to need automation. Have these expire roughly every year. Add bonus time in case someone gets appendicitis. And ideally you will renew these in half the expiration time. Because... Yes? The reason to not have 25-year certificate expiration is that that would put it past 2038 and bad things might happen. Oh. Wow, we're close to doomsday. Yes, for those at home, year 2038 is within 25 years. So, time counters might roll over and yeah, don't do that. So, expiration is key. Our expiration is why we need automation. Worst case, I want to be sure that all the certificates in my company expire two days after I retire. So, organizing a certificate authority. Have one for users and one for hosts. You could use the same. I mean, it's theoretically possible, but organizations change. And one day the group responsible for users won't have access to setting up servers and vice versa and no, separate ones. Also, don't put your certificate authority in Etsy SSH. That just confuses the hosts' native SSH service from this certificate authority. Here I've created user local SSHCA users and hosts as directories for each CA. Give each host in there their own host or user their own subdirectory in that. If nothing else, you're going to be seeing a lot of files labeled authorized keys from a lot of users. You're going to have a lot of files labeled, you know, SSH host ECDSA25519 underscore key.pub. And you could rename these as part of your process to add the host name or the user name. And we all know that at some point this will go horribly wrong. And it's much better to just separate them by directory and don't go trying to change every single file name that you process. Sorry, I mean, say change, you mean automatically, you don't, it's not a main process. I would hope it's automatic. Yeah, so why not, I mean, because you're putting it in their name, why not just make that name the file? Why make it a directory? I would put each, I would... You got user slash... User slash authorized keys....and then whatever file name was in. Why not just put my file into Darcy itself? Yeah, I mean, that's what I do. I put my file, Darcy, instead of into a directory called Darcy with my file name. You know, Darcy, you're probably quite capable of doing that cleanly. I've seen too many things that seem to be obvious go horribly wrong to recommend it. It may be my life in enterprise IT where somebody hits the wrong button. They separate things as blatantly as possible. And then again, every sysadmin has their biases. We're talking against BKAC errors. Yeah. So, let's talk about generating keys. There is the SSH keygen command. This should look awfully familiar. And you define the type of key. Why am I pointing the laser at my laptop and not the screen? Ah, yep, Darcy has me messed up. Okay. Yeah, oh yeah. Use the minus F flag to give it a file name and minus C lets you put a comment in. And be very clear in your comment on that this is a CA key and what type and put a date in. So, if you have a certificate authority, you have to teach. Sorry, I should go back. This gives you a private and a public key. Protect your private keys. Yeah. If you lose your certificate authority private key, then whoever has it can generate certificates and this violates your whole org. And wow, I said I wouldn't be offended or hurt if Peter Wem, sorry, if anyone left the room. But if you drive Peter Wem out, well, I'm good. The violator is okay. Uh-huh. Everything is fine. Yep. So, SSHD authenticates users. It must trust the user certificate authority. So, create a file that contains all of the trusted user certificate authority public keys, one key per line, and in SSHD config, use this trusted user CA key. And that's really it. For the SSH client, it needs to trust the host certificate authority. And the SSH known hosts file is for exactly trusted keys. So, rather than have a separate option for certificate authorities, you modify the public key with this at cert authority marker and the domains that it is good for, and then have your key. So, certificates also have a concept of an identity and this identity will show up at various points when you use it. What the certificate is for, you set the identity when you create the cert and it cannot change. You have to regenerate a new cert to give it a different identity. And the identity gets logged whenever that certificate is used. I begin identities for hosts with host underscore and users with user underscore because being evitistical, I have accounts named MWL and machines named MWL. So, I'm also going to recommend that you keep your certificates. If you generate a cert, keep it for at least a cycle or so after it should be out of production. It's easier to revoke when you have a copy. And again, you cannot trust users. So, how do you actually create a certificate? Well, OpenSSH has this beautiful SSH key sign command that is completely irrelevant to signing keys. You want to use SSH key gen. And what you've got minus SS is your certificate authority minus I is the identity. I'm creating a key for the host sloth minus H means this is a host cert. Minus N is the host that this is good for. Minus V is the expiration date in relative date format. And then there's your, the name of your key file. Here I'm signing all of the dot pub. Here I'm creating certificates from all of the public keys for this host. And in this example, I did not trim down the supported algorithms, but you create all of these dash cert files that correspond with the dot pub files. The host certificate option to give the certificate that goes with the key. I'm going to recommend that you put each cert next to the key that it's paired with. Again, because humans are easily confused. You can also look at the certificate contents on the command line with the minus capital L option. And it'll tell you things, you know, what cert signed it, what is this key for, when does it expire. What host is it good for, etc. Usual things you would expect to find in a TLS cert without all of the extra stuff that gets stuck in a TLS cert. So, if you want to test a host cert, move the clients known hosts out of the way and try to log in. You should connect and be logged in without being prompted to verify the host key. If that doesn't work, add minus Vs until you get an explanation. And go search for what that error code means. My usual error turns out to be that I don't install the certificate authority on the client. For whatever reason, you, of course, will invent your own most common error. So, user certificates. Again, SSH key gen, not SSH key sign. And the real difference here, there's no minus H flag. That's it. For the user-to-user certificate, copy the certificate file into home SSH right next to the corresponding public key file. And at that point, go into server. And you know perfectly well, if the authorized key file exists and has that magic key, they'll be logged in. And you can't trust users to not add extra stuff, so you want to disable authorized keys. In SSHD, if you set authorized keys filed to none, the only way a user can log in is with that certificate. And again, if for some reason your first attempt doesn't work, add minus Vs until an error pops up. So, does that make sense so far? I'm going to take silence as consent. So, last bit here. Does anyone here work for, say, Google, Facebook, Amazon, giant companies with millions of servers? Oh, you do. Do you know how they deploy and run SSH internally? You do. Excellent. You know better than I do. Would you like to finish the talk? Well, feel free to jump in then and illuminate things or considerations that I have screwed up. So, there are organizations that need more than 65,000 UIDs. There are organizations that have so much information that they really don't want to put, they don't want to have everyone poking at LDAP. Because they're going to need tens of thousands of LDAP servers and this is just terrible. So, this is where principles come in. A principle is a named entity that is not tied to a host name or a UID. And it can be structured any way you want. And a principle can be authorized to log in via SSH. So, you can have, say, I have the principle of Kurt M. And you can tell the server that principle Kurt M may log in as root. So, that's where the authorized principles file comes in. Here I've got a sample one. A few principles that exist. The principle everywhere root. If this is tied to, if the server is presented with a certificate that has this everywhere root principle on the search, they can log in. If they have the Europe root or Europe database principles. So, when you create the user certificate, you would assign principles to that certificate. And so, this server is clearly in Europe. It's a database server. And folks on that team can log into it. You can also do this authorized principles file at the SSH principles percent U. And say someone presents a cert and they want to log in as root. SSHD will check at the SSH principles root. And check that file for who can log in as that. So, you might have a root file, a DB admin file, a web admin file for just having just a few accounts on the host. Can we have multiples? Yes. Just like in this slide. Yes, people can be in multiple groups. People can, certificates can have multiple principles. So, here I'm creating a cert. And since this is a huge organization that I'm condemned to work for, it's got the identity starts with user. It has my employee ID number. It has my name. Minus n gives the principle, which I'm in peasants and vermin. And this one is good for 52 weeks. So, if someone needs me to be an additional principles, I need a new certificate. So, principles file. Copy your authorized principles file across your millions of servers with automation. No. No. There's a script to look up principles for a user when someone tries to log in as that user. So, and again, use an individual, unprivileged user to run that command. So, this is how you would safely log in as root on your machines because it has to be an assert. It has to have the principle it says you can log in as root. And that assert is uniquely tied to a person. So, and that person had to have used the passphrase to unlock their private key to use the cert. So, if you have your millions of hosts and hundreds of thousands of sysadmins and you need to divide up roles, well, you know, your database admin logs in as dbadmin, but the system records everything you need to blame them, which is really what's important. So, hey, 45 minutes. Perfect. So, I'm going to take questions, but before I do that, I need to make the IRS happy. I write books. You can buy them. So, this trip is now tax deductible. Questions. Oh, God, it's Colin. Yes. Well, I think that putting all the principles in a blockchain would be a really interesting patch to submit to the OpenSSH developers. I think they'd love to see that. No comment. Yes, sir. The company I work for is actually doing exactly that. Okay. Everything in a giant blockchain. Wait. The company you're working for is putting all the principles in a giant blockchain? Kind of. Okay. You need to submit a paper next year to BSDCAN. One, why they did that, and two, what happened to the company? Come on. Are you all stunned? Yes, sir. Will your next book be about key management and have the word violated in the title? No, I'm not writing a book on key management or violation. Consent is a thing. Yes, Will. This seems really complicated. It is. And a lot of hand-built stuff. And I'm curious at what point does this complexity bring you close to Kerberos? At what point does the complexity of certificates bring you close to Kerberos? Well, I'm going to fall back here on tools, not policy. The nice thing about certificates is the network can be down. And in many cases, they'll still work. If you've distributed authorized principles, your Kerberos controller could be a smoking crater and you could still log in. Tools, not policy. Choose your disaster mode. Choose your headache. Organizations like Facebook found this was the least appalling way to run their systems. The fact that someone is going to build a web-based system for maintaining and submitting authorized keys and CAs and whatnot, especially after this talk hits YouTube, but really, if you need certificates, you have a bunch of machines. You have teams for this. Let them do their thing. Hi, Darcy. He can do it without any errors. Wait, wait. A comment and a question? You misunderstand the nature of the question session. There will be a question and then an answer will be provided. So, here it is. No, no, that is the rule. So here is my question to you. What is your comment? Peter, please start breathing again. Just a comment. My employer is moving from Kerberos to Brasov's system, and yes, it does work at very life scales much better than the Kerberos method. Okay. The comment is that Peter's employer is moving from Kerberos to certificates because it really does work better, which I'm reading as I should not write Kerberos mastery, but a vast relief. Is there anyone else who has a question or if they dare a comment? Yes, sir? With the PKI, is there a good way to distribute a remote certificates list? Revoked. Does the revoked cert marker just mark the cert as revoked and push it out? The end goal of all of this is that when your user sees a warning, it means something, something other than just type yes. Any other questions? Come on. Colin again. Okay. So the question is, which Colin carefully phrased as a question for some weird reason, he heard a rumor that some large organization that came up earlier in this talk deployed the same host he across all hosts. Does anyone know any truth? Can anyone confirm or deny? I can say it's a terrible idea, but nobody can confirm or deny. Okay. My faith in humanity is not complete. Oh, we have a hand in the corner. Oh, God, no. Please don't tell me. No, no. There is a use case. There is a use case for this. If you're running a cluster of firewalls and you're running carp, you're always hitting S-H via that carp. And you're running what? Carp. Carp. For something like a cluster. And you're always SSHing and you want both hosts to have the same key. I can understand why people would be tempted in a cluster. What I generally do is have a separate management IP for each of the firewalls so that I can uniquely identify the firewall. That's why everybody's saying cluster. Yeah. I would not use the same host key on each and I want to SSH to a particular machine in the cluster, not just some random host in it. You're the management network in that, right? Yeah. Yeah. You're the same management. Yeah, I would certainly not do that. I can understand that there are times when you may not have a choice because of some administrative concern. But that, you know, problems with your management are beyond the scope of this talk. Yes, Phillip. There are environments where you can control the SSH config of the client. So you might want to fill the authorized keys and server to add something like the server authority list. Why did you add that to the talk? Why did I not add things when you cannot control the client? Because the variety of ways in which you can be brutalized by terrible environments would fill a book far larger than my entire collection of things I've written. So you can control why I don't authorize the keys. You can add a certain server authority list or you're never allowed to do what for what and all that and you can add that on the host. So you can control it a bit. Oh, yes. You can control features and such through the cert authority list. And I didn't add them to the talk because I wanted to get the talk in at between 45 and 50 minutes and leave some time for questions. But that's in the book. I believe that my marketing demands that I say I have the book around and you can get it. If you want to know all this cool stuff that there wasn't time to talk about. Yes, sir. Sorry, how can I challenge you a bit on that last group? Okay. So it's not a question. That's a warning that that's like a preface to a question. There is a scenario where you're using AuthPF on the cluster and all you're using, all your users are as a state chain to that cluster. Okay. Being redirected to the different web is that they're called for? That for AuthPF to the cluster, I would certainly say that, yes, you could use the same key across multiple SSHs. However, I would also say that for the management of the host, it should go to a unique per host IP and have the cluster SSH service be only for AuthPF. And Patrick is waving his hand around. Oh, yeah. And people are starting to walk in the door, which I believe is how nature tells presenters to get out. Thank you all for coming.
 Okay. So just to double check, I'm going to give it just another moment. Are there any  open SSH developers in the room? No, no, because Colin, we pretty much ignore you.  What  Okay. Maybe you don't have LDAP. Everybody's network is a little different. The important  thing here is that authorized keys command is a script. Do you have a database on the  network, any kind of directory, any anything? You know, you are a sysadmin. Scripts are your  thing. If your company has a standard where you are using, you know, you provide all your  data via WordPress, XML, RPC over HTTP via an SSH tunnel, that's fine. An Excel spreadsheet  on a sysvol, whatever. Just write a script to access it. Host keys. Host keys are the akan works.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  NONE hosts to all of the client machines or you can have the client look up NONE hosts.  Once you're at the point where you're distributing this, you can also set global client SSH settings in SSH underscore config.  So if your organization does something like run SSHD on and off port or you have special requirements or you need this little flag twiddled on all of the clients everywhere,  you set it globally and it works for everyone.  So when it comes to clients, the most common one other than SSH, sorry, other than open SSH is SSH is Putty.  Putty of course stores its host keys in the registry because it's Windows and you put everything in the registry.  Why wouldn't you? Putty ships with a Python script to convert NONE hosts into registry keys.  Run the script, you have a reg file, give it to your AD admin and eventually they'll put it in user login scripts and distribute it.  Now, if your AD admin is slow, you can always do things like email the registry file to all your users and teach them to blindly double click on email registry files before using SSH.  I'm going to recommend subject lines like this one weird trick makes your Putty work.  Putty admins hate when you click on this or if you do this once you may find your AD administrator becomes much more responsive.  So if you don't want to distribute, you can have the SSH client check on the network through SSHFP DNS records.  Now, people tell you this requires DNS sec.  Well, I hear folks arguing against that.  If you want to shuffle security critical data around in unencrypted UDP across the network that's just begging to be spoofed.  Okay, it's not my network. So, you generate an SSHFP record with SSH key gen minus R in the host name and it spits out a bunch of SSHFP records.  Put those in your zone file.  If you are using key files from a different host, add the minus F flag.  So, that's fairly straightforward for traditional SSH use.  Or you look them up from the network, client and server alike.  SSH certificates are something very, very different.  And specifically, they're different from the TLSX509 certs that you all know and loathe from web servers and other applications.  So, what's a certificate authority?  It's really just a method of delegating trust.  It says, I trust this thing.  I trust Verif sign.  I trust Let's Encrypt.  I trust Wems House of Cheap Certs.  Whatever.  So, and it takes two feature, two technological features.  Encryption and signing.  And these features are both in regular SSH keys.  So, an SSH certificate authority is just an SSH key that you decide you're going to use as a certificate authority.  It really is exactly like a self-signed TLS certificate.  Because you don't want anyone outside your organization to trust this.  There is no reason why hosts and clients on my network should trust certificates issued by Curt's network.  Curt has his own network and I really, by design, I should not trust it.  And it's not Curt's fault.  It's just we don't need external trust.  So, certificate authorities, if your network is large enough to require them, then you really do need automation.  You have to be able to copy files and restart SSHD reliably.  Look at Ansible if you haven't.  So, one thing you see in a certificate is an expiration date.  It's real tempting when you do a self-signed certificate to give it a 25-year expiration date.  But they will be insecure.  It still contained the first ever SSH version 1 user authentication public key I had ever created.  And I really only noticed it when the FreeBSD cluster disabled SSH v1.  And suddenly I couldn't get into free fall and oops.  What?  Yes.  So, remember you're going to need automation.  Have these expire roughly every year.  Add bonus time in case someone gets appendicitis.  And ideally you will renew these in half the expiration time.  Because...  Yes?  The reason to not have 25-year certificate expiration is that that would put it past 2038 and bad things might happen.  Oh.  Wow, we're close to doomsday.  Yes, for those at home, year 2038 is within 25 years.  So, time counters might roll over and yeah, don't do that.  So, expiration is key.  Our expiration is why we need automation.  Worst case, I want to be sure that all the certificates in my company expire two days after I retire.  So, organizing a certificate authority.  Have one for users and one for hosts.  You could use the same.  I mean, it's theoretically possible, but organizations change.  And one day the group responsible for users won't have access to setting up servers and vice versa and no, separate ones.  Also, don't put your certificate authority in Etsy SSH.  That just confuses the hosts' native SSH service from this certificate authority.  Here I've created user local SSHCA users and hosts as directories for each CA.  Give each host in there their own host or user their own subdirectory in that.  If nothing else, you're going to be seeing a lot of files labeled authorized keys from a lot of users.  You're going to have a lot of files labeled, you know, SSH host ECDSA25519 underscore key.pub.  And you could rename these as part of your process to add the host name or the user name.  And we all know that at some point this will go horribly wrong.  And it's much better to just separate them by directory and don't go trying to change every single file name that you process.  Sorry, I mean, say change, you mean automatically, you don't, it's not a main process.  I would hope it's automatic.  Yeah, so why not, I mean, because you're putting it in their name, why not just make that name the file?  Why make it a directory?  I would put each, I would...  You got user slash...  User slash authorized keys. ...and then whatever file name was in. Why not just put my file into Darcy itself?  Yeah, I mean, that's what I do.  I put my file, Darcy, instead of into a directory called Darcy with my file name.  You know, Darcy, you're probably quite capable of doing that cleanly.  I've seen too many things that seem to be obvious go horribly wrong to recommend it.  It may be my life in enterprise IT where somebody hits the wrong button.  They separate things as blatantly as possible.  And then again, every sysadmin has their biases.  We're talking against BKAC errors.  Yeah.  So, let's talk about generating keys.  There is the SSH keygen command.  This should look awfully familiar.  And you define the type of key.  Why am I pointing the laser at my laptop and not the screen?  Ah, yep, Darcy has me messed up.  Okay.  Yeah, oh yeah.  Use the minus F flag to give it a file name and minus C lets you put a comment in.  And be very clear in your comment on that this is a CA key and what type and put a date in.  So, if you have a certificate authority, you have to teach.  Sorry, I should go back.  This gives you a private and a public key.  Protect your private keys.  Yeah.  If you lose your certificate authority private key, then whoever has it can generate certificates and this violates your whole org.  And wow, I said I wouldn't be offended or hurt if Peter Wem, sorry, if anyone left the room.  But if you drive Peter Wem out, well, I'm good.  The violator is okay.  Uh-huh.  Everything is fine.  Yep.  So, SSHD authenticates users.  It must trust the user certificate authority.  So, create a file that contains all of the trusted user certificate authority public keys, one key per line, and in SSHD config, use this trusted user CA key.  And that's really it.  For the SSH client, it needs to trust the host certificate authority.  And the SSH known hosts file is for exactly trusted keys.  So, rather than have a separate option for certificate authorities, you modify the public key with this at cert authority marker and the domains that it is good for, and then have your key.  So, certificates also have a concept of an identity and this identity will show up at various points when you use it.  What the certificate is for, you set the identity when you create the cert and it cannot change.  You have to regenerate a new cert to give it a different identity.  And the identity gets logged whenever that certificate is used.  I begin identities for hosts with host underscore and users with user underscore because being evitistical, I have accounts named MWL and machines named MWL.  So, I'm also going to recommend that you keep your certificates.  If you generate a cert, keep it for at least a cycle or so after it should be out of production.  It's easier to revoke when you have a copy.  And again, you cannot trust users.  So, how do you actually create a certificate?  Well, OpenSSH has this beautiful SSH key sign command that is completely irrelevant to signing keys.  You want to use SSH key gen.  And what you've got minus SS is your certificate authority minus I is the identity. I'm creating a key for the host sloth minus H means this is a host cert.  Minus N is the host that this is good for.  Minus V is the expiration date in relative date format.  And then there's your, the name of your key file.  Here I'm signing all of the dot pub.  Here I'm creating certificates from all of the public keys for this host.  And in this example, I did not trim down the supported algorithms, but you create all of these dash cert files that correspond with the dot pub files.  The host certificate option to give the certificate that goes with the key.  I'm going to recommend that you put each cert next to the key that it's paired with.  Again, because humans are easily confused.  You can also look at the certificate contents on the command line with the minus capital L option.  And it'll tell you things, you know, what cert signed it, what is this key for, when does it expire.  What host is it good for, etc.  Usual things you would expect to find in a TLS cert without all of the extra stuff that gets stuck in a TLS cert.  So, if you want to test a host cert, move the clients known hosts out of the way and try to log in.  You should connect and be logged in without being prompted to verify the host key.  If that doesn't work, add minus Vs until you get an explanation.  And go search for what that error code means.  My usual error turns out to be that I don't install the certificate authority on the client.  For whatever reason, you, of course, will invent your own most common error.  So, user certificates.  Again, SSH key gen, not SSH key sign.  And the real difference here, there's no minus H flag.  That's it.  For the user-to-user certificate, copy the certificate file into home SSH right next to the corresponding public key file.  And at that point, go into server.  And you know perfectly well, if the authorized key file exists and has that magic key, they'll be logged in.  And you can't trust users to not add extra stuff, so you want to disable authorized keys.  In SSHD, if you set authorized keys filed to none, the only way a user can log in is with that certificate.  And again, if for some reason your first attempt doesn't work, add minus Vs until an error pops up.  So, does that make sense so far?  I'm going to take silence as consent.  So, last bit here.  Does anyone here work for, say, Google, Facebook, Amazon, giant companies with millions of servers?  Oh, you do.  Do you know how they deploy and run SSH internally?  You do. Excellent. You know better than I do. Would you like to finish the talk?  Well, feel free to jump in then and illuminate things or considerations that I have screwed up.  So, there are organizations that need more than 65,000 UIDs.  There are organizations that have so much information that they really don't want to put, they don't want to have everyone poking at LDAP.  Because they're going to need tens of thousands of LDAP servers and this is just terrible.  So, this is where principles come in.  A principle is a named entity that is not tied to a host name or a UID.  And it can be structured any way you want.  And a principle can be authorized to log in via SSH.  So, you can have, say, I have the principle of Kurt M.  And you can tell the server that principle Kurt M may log in as root.  So, that's where the authorized principles file comes in.  Here I've got a sample one.  A few principles that exist. The principle everywhere root.  If this is tied to, if the server is presented with a certificate that has this everywhere root principle on the search, they can log in.  If they have the Europe root or Europe database principles.  So, when you create the user certificate, you would assign principles to that certificate.  And so, this server is clearly in Europe. It's a database server.  And folks on that team can log into it.  You can also do this authorized principles file at the SSH principles percent U.  And say someone presents a cert and they want to log in as root.  SSHD will check at the SSH principles root.  And check that file for who can log in as that.  So, you might have a root file, a DB admin file, a web admin file for just having just a few accounts on the host.  Can we have multiples?  Yes. Just like in this slide.  Yes, people can be in multiple groups.  People can, certificates can have multiple principles.  So, here I'm creating a cert. And since this is a huge organization that I'm condemned to work for,  it's got the identity starts with user.  It has my employee ID number.  It has my name.  Minus n gives the principle, which I'm in peasants and vermin.  And this one is good for 52 weeks.  So, if someone needs me to be an additional principles, I need a new certificate.  So, principles file. Copy your authorized principles file across your millions of servers with automation.  No.  No. There's a script to look up principles for a user when someone tries to log in as that user.  So, and again, use an individual, unprivileged user to run that command.  So, this is how you would safely log in as root on your machines because it has to be an assert.  It has to have the principle it says you can log in as root. And that assert is uniquely tied to a person.  So, and that person had to have used the passphrase to unlock their private key to use the cert.  So, if you have your millions of hosts and hundreds of thousands of sysadmins and you need to divide up roles,  well, you know, your database admin logs in as dbadmin, but the system records everything you need to blame them,  which is really what's important.  So, hey, 45 minutes. Perfect.  So, I'm going to take questions, but before I do that, I need to make the IRS happy.  I write books. You can buy them.  So, this trip is now tax deductible.  Questions.  Oh, God, it's Colin. Yes.  Well, I think that putting all the principles in a blockchain would be a really interesting patch to submit to the OpenSSH developers.  I think they'd love to see that.  No comment. Yes, sir.  The company I work for is actually doing exactly that.  Okay.  Everything in a giant blockchain.  Wait. The company you're working for is putting all the principles in a giant blockchain?  Kind of.  Okay. You need to submit a paper next year to BSDCAN.  One, why they did that, and two, what happened to the company?  Come on. Are you all stunned?  Yes, sir.  Will your next book be about key management and have the word violated in the title?  No, I'm not writing a book on key management or violation.  Consent is a thing.  Yes, Will.  This seems really complicated.  It is.  And a lot of hand-built stuff.  And I'm curious at what point does this complexity bring you close to Kerberos?  At what point does the complexity of certificates bring you close to Kerberos?  Well, I'm going to fall back here on tools, not policy.  The nice thing about certificates is the network can be down.  And in many cases, they'll still work.  If you've distributed authorized principles, your Kerberos controller could be a smoking crater  and you could still log in.  Tools, not policy.  Choose your disaster mode.  Choose your headache.  Organizations like Facebook found this was the least appalling way to run their systems.  The fact that someone is going to build a web-based system for maintaining and submitting authorized keys and CAs and whatnot,  especially after this talk hits YouTube,  but really, if you need certificates, you have a bunch of machines.  You have teams for this.  Let them do their thing.  Hi, Darcy.  He can do it without any errors.  Wait, wait.  A comment and a question?  You misunderstand the nature of the question session.  There will be a question and then an answer will be provided.  So, here it is.  No, no, that is the rule.  So here is my question to you.  What is your comment?  Peter, please start breathing again.  Just a comment.  My employer is moving from Kerberos to Brasov's system,  and yes, it does work at very life scales much better than the Kerberos method.  Okay.  The comment is that Peter's employer is moving from Kerberos to certificates because it really does work better,  which I'm reading as I should not write Kerberos mastery, but a vast relief.  Is there anyone else who has a question or if they dare a comment?  Yes, sir?  With the PKI, is there a good way to distribute a remote certificates list?  Revoked.  Does the revoked cert marker just mark the cert as revoked and push it out?  The end goal of all of this is that when your user sees a warning, it means something,  something other than just type yes.  Any other questions?  Come on.  Colin again.  Okay.  So the question is, which Colin carefully phrased as a question for some weird reason,  he heard a rumor that some large organization that came up earlier in this talk deployed the same host he across all hosts.  Does anyone know any truth?  Can anyone confirm or deny?  I can say it's a terrible idea, but nobody can confirm or deny.  Okay.  My faith in humanity is not complete.  Oh, we have a hand in the corner.  Oh, God, no.  Please don't tell me.  No, no.  There is a use case.  There is a use case for this.  If you're running a cluster of firewalls and you're running carp, you're always hitting S-H via that carp.  And you're running what?  Carp.  Carp.  For something like a cluster.  And you're always SSHing and you want both hosts to have the same key.  I can understand why people would be tempted in a cluster.  What I generally do is have a separate management IP for each of the firewalls so that I can uniquely identify the firewall.  That's why everybody's saying cluster.  Yeah.  I would not use the same host key on each and I want to SSH to a particular machine in the cluster, not just some random host in it.  You're the management network in that, right?  Yeah.  Yeah.  You're the same management.  Yeah, I would certainly not do that.  I can understand that there are times when you may not have a choice because of some administrative concern.  But that, you know, problems with your management are beyond the scope of this talk.  Yes, Phillip.  There are environments where you can control the SSH config of the client.  So you might want to fill the authorized keys and server to add something like the server authority list.  Why did you add that to the talk?  Why did I not add things when you cannot control the client?  Because the variety of ways in which you can be brutalized by terrible environments would fill a book far larger than my entire collection of things I've written.  So you can control why I don't authorize the keys. You can add a certain server authority list or you're never allowed to do what for what and all that and you can add that on the host.  So you can control it a bit.  Oh, yes.  You can control features and such through the cert authority list.  And I didn't add them to the talk because I wanted to get the talk in at between 45 and 50 minutes and leave some time for questions.  But that's in the book.  I believe that my marketing demands that I say I have the book around and you can get it.  If you want to know all this cool stuff that there wasn't time to talk about.  Yes, sir.  Sorry, how can I challenge you a bit on that last group?  Okay.  So it's not a question.  That's a warning that that's like a preface to a question.  There is a scenario where you're using AuthPF on the cluster and all you're using, all your users are as a state chain to that cluster.  Okay.  Being redirected to the different web is that they're called for?  That for AuthPF to the cluster, I would certainly say that, yes, you could use the same key across multiple SSHs.  However, I would also say that for the management of the host, it should go to a unique per host IP and have the cluster SSH service be only for AuthPF.  And Patrick is waving his hand around.  Oh, yeah.  And people are starting to walk in the door, which I believe is how nature tells presenters to get out.  Thank you all for coming.
Michael_W_Lucas__SSH_Key_Management_0_107.png Michael_W_Lucas__SSH_Key_Management_108_251.png Michael_W_Lucas__SSH_Key_Management_252_395.png Michael_W_Lucas__SSH_Key_Management_396_527.png Michael_W_Lucas__SSH_Key_Management_528_923.png Michael_W_Lucas__SSH_Key_Management_924_1055.png Michael_W_Lucas__SSH_Key_Management_1056_1211.png Michael_W_Lucas__SSH_Key_Management_1212_1331.png Michael_W_Lucas__SSH_Key_Management_1332_2171.png Michael_W_Lucas__SSH_Key_Management_2172_3875.png Michael_W_Lucas__SSH_Key_Management_3876_3935.png Michael_W_Lucas__SSH_Key_Management_3936_5231.png Michael_W_Lucas__SSH_Key_Management_5232_5963.png Michael_W_Lucas__SSH_Key_Management_5964_6803.png Michael_W_Lucas__SSH_Key_Management_6804_7055.png Michael_W_Lucas__SSH_Key_Management_7056_7187.png Michael_W_Lucas__SSH_Key_Management_7188_7463.png Michael_W_Lucas__SSH_Key_Management_7464_7595.png Michael_W_Lucas__SSH_Key_Management_7596_7775.png Michael_W_Lucas__SSH_Key_Management_7776_7823.png Michael_W_Lucas__SSH_Key_Management_7824_8123.png Michael_W_Lucas__SSH_Key_Management_8124_8255.png Michael_W_Lucas__SSH_Key_Management_8256_8327.png Michael_W_Lucas__SSH_Key_Management_8328_10451.png Michael_W_Lucas__SSH_Key_Management_10452_13799.png Michael_W_Lucas__SSH_Key_Management_13800_16535.png Michael_W_Lucas__SSH_Key_Management_16536_17759.png Michael_W_Lucas__SSH_Key_Management_17760_18011.png Michael_W_Lucas__SSH_Key_Management_18012_18419.png Michael_W_Lucas__SSH_Key_Management_18420_19403.png Michael_W_Lucas__SSH_Key_Management_19404_19883.png Michael_W_Lucas__SSH_Key_Management_19884_23063.png Michael_W_Lucas__SSH_Key_Management_23064_24647.png Michael_W_Lucas__SSH_Key_Management_24648_27011.png Michael_W_Lucas__SSH_Key_Management_27012_28955.png Michael_W_Lucas__SSH_Key_Management_28956_30911.png Michael_W_Lucas__SSH_Key_Management_30912_34811.png Michael_W_Lucas__SSH_Key_Management_34812_38147.png Michael_W_Lucas__SSH_Key_Management_38148_42599.png Michael_W_Lucas__SSH_Key_Management_42600_44063.png Michael_W_Lucas__SSH_Key_Management_44064_44291.png Michael_W_Lucas__SSH_Key_Management_44292_45287.png Michael_W_Lucas__SSH_Key_Management_45288_46151.png Michael_W_Lucas__SSH_Key_Management_46152_46523.png Michael_W_Lucas__SSH_Key_Management_46524_46655.png Michael_W_Lucas__SSH_Key_Management_46656_47135.png Michael_W_Lucas__SSH_Key_Management_47136_48359.png Michael_W_Lucas__SSH_Key_Management_48360_49043.png Michael_W_Lucas__SSH_Key_Management_49044_50495.png Michael_W_Lucas__SSH_Key_Management_50496_51407.png Michael_W_Lucas__SSH_Key_Management_51408_52259.png Michael_W_Lucas__SSH_Key_Management_52260_53459.png Michael_W_Lucas__SSH_Key_Management_53460_54035.png Michael_W_Lucas__SSH_Key_Management_54036_54539.png Michael_W_Lucas__SSH_Key_Management_54540_55859.png Michael_W_Lucas__SSH_Key_Management_55860_57851.png Michael_W_Lucas__SSH_Key_Management_57852_59003.png Michael_W_Lucas__SSH_Key_Management_59004_61343.png Michael_W_Lucas__SSH_Key_Management_61344_62999.png Michael_W_Lucas__SSH_Key_Management_63000_65135.png Michael_W_Lucas__SSH_Key_Management_65136_65507.png Michael_W_Lucas__SSH_Key_Management_65508_65603.png Michael_W_Lucas__SSH_Key_Management_65604_65723.png Michael_W_Lucas__SSH_Key_Management_65724_65843.png Michael_W_Lucas__SSH_Key_Management_65844_81791.png Michael_W_Lucas__SSH_Key_Management_81792_81813.png