Deployment Showdown
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Strack, Sven
N. N.
Sven will host a live competition where entrants show how many different cloud platforms they can deploy a working Plone instance to in 15 minutes.
 Cool, welcome to our deployment showdown, or at least that was the whole idea, because the whole idea was you get 50 minutes and in 50 minutes you try to deploy as many plant sites as you can, on as many different cloud providers as you can, with as many different deployment tools as you can, and we had some kind of rule set before that, for example you were not allowed to prepare accounts beforehand, everything had to be done during deployment, but unfortunately no one signed up for the competition. No chickens! And I'm kind of sad because I really don't know why, I mean, so now we are going to start, okay, let's make it easier, and then we took all the rules away, and still no one was showing up. So in place of showdown we will do now some demos of stuff like deployments, Docker, demo with Ansible, with Opsworks, so I'm really sorry. Hopefully we have a solstice one coming too, and I'm also really sorry because I was also supposed to deploy something, oh it's hard to read. Anyway, with Mr. Henry and I have some, at the moment something currently is something like 18 different deployments which I can run in 11 minutes and then I have the sites running, but unfortunately I was kind of busy with preparing training and other stuff, so I forgot to do a little bit push and some stuff is still on my home machine, but I promise when I'm back in two weeks in Europe I will do every week one blog post with one example, and I had really cool stuff like, I don't know, using Gobla, using Ubuntu Snapcore, using JoJo OpenShift and all the stuff, but yeah, you will have to read the blog posts, I'm sorry. Now I started demo time and Steve, do you would like to start? So in short, nobody signed up for the competition because the competition was rigged. I mean seriously, nobody was going to beat Spam. There was no possibility anybody would beat Spam. Let me find my mini DVI port. There it is. Okay, I am not detecting the display. Okay, there it is. Oh dear, bro down, that's horrifying. So let me show you the tool that I would have used to compete if it wasn't a rigged competition. So for some time the installer team has been working, and I guess I have been the lead on this, on an Ansible playbook that will install a full stack operation for Plone. And the basic idea behind this is that cloud servers are nearly disposable these days and you can practically treat them as a unit and very quickly deploy very repeatable instances to them. And we decided to work on Ansible mainly because it made sense to us and because Ansible has very few requirements to work. So we can target just about any Linuxy sort of environment, actually assuming it's Debian family or Red Hat family, but that covers a great deal. We've got probably about five or six people actively contributing to it now, including getting some help on Red Hat environment, which is great because I know very little about Reb Hat. So when I say full stack, we are really trying to cover everything from the firewall down to the actual ZO cluster, and including portions that are not part of the delivery stack, but you need for a real server monitoring, logging, a mail transfer agent, and automated platform updates. Let me zoom up here and show you the actual pieces that we're using for this. The firewall is a completely separate component because no two system administrators really agree on what the firewall should be anyway, but also it has to run under both Red Hat environments and Debian environments. And we've got one where if you're willing to put up with our choices, it works beautifully. For basically the HTTP engine and the rewrite engine, we're using EngineX, Varnish for the reverse proxy cache, HA proxy for load balancing, and then our favorite NoSQL solution to deliver a clone down at the bottom, ZO clients. And then on the server administration side, we're running PostFix for a mail transfer agent in a send-only configuration. We've got it set up so it's very easy to add a mail relay if you're, say, working in a cloud environment which doesn't allow you to directly connect to well-known SMTP ports like Google doesn't. It's a rack space, doesn't they don't think? We've got platform auto updates, MUNIN node for monitoring, log watch to just check log files, fail to ban to just do a block the most obvious attacks on port 22. All of these are very particular choices and they might not be your choices, but one of the things that we're guaranteeing by working on this is that we're going to try to maintain it for some period of time. So we're not necessarily after the coolest new toy on the block, but we're after one that's well-known in the Plone community. And I think if you, for example, ask what's a good proxy cache, people are not going to be astonished when you say varnish. And they're not going to be astonished if you say that you're using HA proxy for your clustering, for the load balancing. So how does it all look? Well, you have to create a manifest file that tells Ansible, the tool we're using, how to connect to the remote servers. And once that's in place, oh, actually, that's just a ping command. Let me get back here. You can just start running and it will, for as many hosts as you want to be setting up at a time, it will be running, it will run them in parallel to set them up. I'm setting up two hosts. I cheated a little bit, which I'm sure Sven would have done also, by doing the update of the operating system packages and shipping over a build out cache in advance. Actually, I mainly cheated because I didn't want you folks to have to wait through all of that. But it's running now. And while it's running, let me say a few things about other portions of the operation. We're using Supervisor for Process Control. Since we know about everything that is going into this and everything that's making it up, we know things like how many ZEO clients are using and where they're based. There are a whole bunch of tricks we can do to integrate things. For example, we write out a message of the day file so that when you log on to this server in six months, it tells you this server is maintained via Ansible. Don't change configuration directly. But it also tells you all of the demons running on all of the ports so that you can figure it out later on. You can figure out where the HAProxy administrative status port is, things like that. Since we're aiming for production servers, everything is pretty well locked down. That doesn't mean that you can't do better. But if you figure out some obvious ways to improve the lockdown, tell us. But we're assuming that right from the start, the only ports that are going to be visible to the world are going to be port 80 and port 443. And that's it. And everything else, if you want to administer the machine, you're going to have to use this as tunnels to do it. And that includes to get at the Zope Management interface or the top of the ZODB. Very cute thing, very big payoff for having all of this knowledge about how the parts fit together is that one of the things that this generates for you is a restart script. Now you may be running one ZEO client or you may be running 40, and this will set up anywhere in between. But when you update packages and when you make a change, if you're running a production server, you don't want it to go down for that purpose. So we generate a restart script that iterates through all of the Zope clients, takes them out of the HA proxy cluster, marks them down for maintenance, restarts the ZEO client, then warms up the cache by fetching either the homepage URL or whatever you specify it should be fetching. And only after it's gotten all of that does it return it to the cluster. So you're going to have a very busy site. You're going to have a very complex site where the homepage might ordinarily take 30 or 40 seconds to render. Those of you who saw the Washington Trails Association site that David Glick was working on. The homepage render for that site is somewhere between 30 seconds to a minute for the first time it hits a client. We cannot afford that, so we warm up the ZODB cache ahead of time. So let's see where we are in our run. Oh, it's running build out, and this might even take a while. And I don't want to make you sit through all of that. So instead, what I will show you is we have got full training materials. These are on the training site, a very lengthy narrative description of how all this works. Thank you. That is intended to really be an introduction to it. It even works you through things like the semantics of YAML, which is Ansible's configuration language. Also over on docs.plone.org, which is a total joy to work with thanks to Sven's work. We have got full documentation on every single option in the Ansible playbook, everything that you might want to change. And I think we have covered a lot. For example, you can make a lot of changes to your build out, like adding eggs and adding versions just by changing configuration options. Or you can give it a GitHub address and it will fetch your whole build out via GitHub and then build your operation. In any case, every bit of that is documented. And if you discover something you need it to do that it is not doing already and you can convince one or two of us that it is a same maintainable thing to do, we will put it in there and add another option because we really want this to be used because we want the maintenance shared through the community. So let's see. Probably build out still running. You know that story. Of course, it runs a lot faster the next time, but I am going to pass the microphone to our next victim.
 Cool, welcome to our deployment showdown, or at least that was the whole idea, because  the whole idea was you get 50 minutes and in 50 minutes you try to deploy as many plant  sites as you can, on as many different cloud providers as you can, with as many different  deployment tools as you can, and we had some kind of rule set before that, for example  you were not allowed to prepare accounts beforehand, everything had to be done during deployment,  but unfortunately no one signed up for the competition.  No chickens!  And I'm kind of sad because I really don't know why, I mean, so now we are going to  start, okay, let's make it easier, and then we took all the rules away, and still no one  was showing up.  So in place of showdown we will do now some demos of stuff like deployments, Docker, demo  with Ansible, with Opsworks, so I'm really sorry.  Hopefully we have a solstice one coming too, and I'm also really sorry because I was also  supposed to deploy something, oh it's hard to read.  Anyway, with Mr. Henry and I have some, at the moment something currently is something  like 18 different deployments which I can run in 11 minutes and then I have the sites running,  but unfortunately I was kind of busy with preparing training and other stuff, so I forgot  to do a little bit push and some stuff is still on my home machine, but I promise when  I'm back in two weeks in Europe I will do every week one blog post with one example,  and I had really cool stuff like, I don't know, using Gobla, using Ubuntu Snapcore,  using JoJo OpenShift and all the stuff, but yeah, you will have to read the blog posts,  I'm sorry.  Now I started demo time and Steve, do you would like to start?  So in short, nobody signed up for the competition because the competition was rigged.  I mean seriously, nobody was going to beat Spam.  There was no possibility anybody would beat Spam.  Let me find my mini DVI port.  There it is.  Okay, I am not detecting the display.  Okay, there it is.  Oh dear, bro down, that's horrifying.  So let me show you the tool that I would have used to compete if it wasn't a rigged competition.  So for some time the installer team has been working, and I guess I have been the lead  on this, on an Ansible playbook that will install a full stack operation for Plone.  And the basic idea behind this is that cloud servers are nearly disposable these days and  you can practically treat them as a unit and very quickly deploy very repeatable instances  to them.  And we decided to work on Ansible mainly because it made sense to us and because Ansible has  very few requirements to work.  So we can target just about any Linuxy sort of environment, actually assuming it's Debian  family or Red Hat family, but that covers a great deal.  We've got probably about five or six people actively contributing to it now, including  getting some help on Red Hat environment, which is great because I know very little  about Reb Hat.  So when I say full stack, we are really trying to cover everything from the firewall down  to the actual ZO cluster, and including portions that are not part of the delivery stack, but  you need for a real server monitoring, logging, a mail transfer agent, and automated platform  updates.  Let me zoom up here and show you the actual pieces that we're using for this.  The firewall is a completely separate component because no two system administrators really  agree on what the firewall should be anyway, but also it has to run under both Red Hat  environments and Debian environments.  And we've got one where if you're willing to put up with our choices, it works beautifully.  For basically the HTTP engine and the rewrite engine, we're using EngineX, Varnish for the  reverse proxy cache, HA proxy for load balancing, and then our favorite NoSQL solution to deliver  a clone down at the bottom, ZO clients.  And then on the server administration side, we're running PostFix for a mail transfer agent  in a send-only configuration.  We've got it set up so it's very easy to add a mail relay if you're, say, working in a  cloud environment which doesn't allow you to directly connect to well-known SMTP ports  like Google doesn't.  It's a rack space, doesn't they don't think?  We've got platform auto updates, MUNIN node for monitoring, log watch to just check log  files, fail to ban to just do a block the most obvious attacks on port 22.  All of these are very particular choices and they might not be your choices, but one of  the things that we're guaranteeing by working on this is that we're going to try to maintain  it for some period of time.  So we're not necessarily after the coolest new toy on the block, but we're after one  that's well-known in the Plone community.  And I think if you, for example, ask what's a good proxy cache, people are not going to  be astonished when you say varnish.  And they're not going to be astonished if you say that you're using HA proxy for your  clustering, for the load balancing.  So how does it all look?  Well, you have to create a manifest file that tells Ansible, the tool we're using, how to  connect to the remote servers.  And once that's in place, oh, actually, that's just a ping command.  Let me get back here.  You can just start running and it will, for as many hosts as you want to be setting up  at a time, it will be running, it will run them in parallel to set them up.  I'm setting up two hosts.  I cheated a little bit, which I'm sure Sven would have done also, by doing the update of  the operating system packages and shipping over a build out cache in advance.  Actually, I mainly cheated because I didn't want you folks to have to wait through all  of that.  But it's running now.  And while it's running, let me say a few things about other portions of the operation.  We're using Supervisor for Process Control.  Since we know about everything that is going into this and everything that's making it  up, we know things like how many ZEO clients are using and where they're based.  There are a whole bunch of tricks we can do to integrate things.  For example, we write out a message of the day file so that when you log on to this server  in six months, it tells you this server is maintained via Ansible.  Don't change configuration directly.  But it also tells you all of the demons running on all of the ports so that you can figure  it out later on.  You can figure out where the HAProxy administrative status port is, things like that.  Since we're aiming for production servers, everything is pretty well locked down.  That doesn't mean that you can't do better.  But if you figure out some obvious ways to improve the lockdown, tell us.  But we're assuming that right from the start, the only ports that are going to be visible  to the world are going to be port 80 and port 443.  And that's it.  And everything else, if you want to administer the machine, you're going to have to use this  as tunnels to do it.  And that includes to get at the Zope Management interface or the top of the ZODB.  Very cute thing, very big payoff for having all of this knowledge about how the parts  fit together is that one of the things that this generates for you is a restart script.  Now you may be running one ZEO client or you may be running 40, and this will set up anywhere  in between.  But when you update packages and when you make a change, if you're running a production  server, you don't want it to go down for that purpose.  So we generate a restart script that iterates through all of the Zope clients, takes them  out of the HA proxy cluster, marks them down for maintenance, restarts the ZEO client,  then warms up the cache by fetching either the homepage URL or whatever you specify it  should be fetching.  And only after it's gotten all of that does it return it to the cluster.  So you're going to have a very busy site.  You're going to have a very complex site where the homepage might ordinarily take 30 or 40  seconds to render.  Those of you who saw the Washington Trails Association site that David Glick was working  on.  The homepage render for that site is somewhere between 30 seconds to a minute for the first  time it hits a client.  We cannot afford that, so we warm up the ZODB cache ahead of time.  So let's see where we are in our run.  Oh, it's running build out, and this might even take a while.  And I don't want to make you sit through all of that.  So instead, what I will show you is we have got full training materials.  These are on the training site, a very lengthy narrative description of how all this works.  Thank you.  That is intended to really be an introduction to it.  It even works you through things like the semantics of YAML, which is Ansible's configuration  language.  Also over on docs.plone.org, which is a total joy to work with thanks to Sven's work.  We have got full documentation on every single option in the Ansible playbook, everything  that you might want to change.  And I think we have covered a lot.  For example, you can make a lot of changes to your build out, like adding eggs and adding  versions just by changing configuration options.  Or you can give it a GitHub address and it will fetch your whole build out via GitHub  and then build your operation.  In any case, every bit of that is documented.  And if you discover something you need it to do that it is not doing already and you  can convince one or two of us that it is a same maintainable thing to do, we will put  it in there and add another option because we really want this to be used because we  want the maintenance shared through the community.  So let's see.  Probably build out still running.  You know that story.  Of course, it runs a lot faster the next time, but I am going to pass the microphone to our  next victim.
Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_0_71.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_72_167.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_168_1211.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_1212_1595.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_1596_2207.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_2208_3695.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_3696_4871.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_4872_5315.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_5316_7775.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_7776_10631.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_10632_11015.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_11016_11099.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_11100_13115.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_13116_16151.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_16152_16895.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_16896_18287.png Day_2__Deployment_Showdown__by_Sven_Strack_EtCoQwMwY10_18288_18644.png