Django Deployments Done Right
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Baumgartner, Peter
There's no single standard toolkit for deploying Django sites. In our years of consulting, we've seen lots of deployment systems in the wild and where they break down or cause pain. Independent of the system you use (Salt, Ansible, Fabric, Chef, Docker, etc.), there are a few principles a good deployment should follow: Deployments don't take the site down or interrupt active users on the site. Deployments don't involve more than one step or are completely automated. Deployments are fast. A failed deployment never takes down the current running version of the code. Rolling back to a previous deployment is a single step. By following these principles, deployments go from being error-prone, nerve-wracking experiences to trivial non-events in your daily development cycle. This talk will walk you through the steps necessary to create a better deployment process.
 Cool, thanks everybody. Sorry for the issues getting the display going. I am going without my notes here, so hopefully I remember everything. Really quick about me. I'm the founder at Lincoln Loop. Here's a shot of a few of us at PyCon last year, this year. And at Lincoln Loop we do Django development and consulting and everything that goes along with building Django sites like DevOps, JavaScript, UX design. Today we'll talk about deployment, which is something, a topic I'm kind of interested in at the moment. I'm also the co-author of this book, High Performance Django, and it's all about how to build Django sites that scale and are fast. You can get a copy at highperformancejango.com and there's little coupons in your swag bag as well. So today's talk is Django deployments done right. But really, I probably could have called it Python deployments done right. There's not a whole lot of Django specific stuff in here. And honestly, deployments done right probably would have fit most of it as well. Deployments for web applications all look kind of similar. And so, you know, whether we were doing Rails or Node or Python or Django or whatever, a lot of these things will still apply. So first off, what is deployment? I think of it in two ways. One is the noun. Like, the deployment is kind of all your stuff you have set up, your servers, your software, how it all interacts, and then the verb, like the actual act of getting all that stuff onto your servers and keeping it up to date. So today we're mostly going to be talking about the verb, but I may interchange these a little bit. So why do you want a good deployment? First off, uptime. This is obvious. Like, you don't want deployments to take down your site. That's a problem. You don't want to be scared to do deployments because they're going to take down your site. But maybe less obvious is the humans that are working with the deployments. One thing I've noticed is having bad deployments sort of feels like this. It's like showing up at a messy office. You can't find anything. It's stressful. You know, there's probably one person that knows where the thing that you need is. And there's these gatekeepers. And it's really bad for people. Like, I do a lot of consulting work and I work on lots of different sites. And I notice the difference at the end of the day when I'm working with a site that's got a really poor process and not well managed and kind of a mess versus one that's kind of clean and tidy and does what I expect. It's stressful. It tends to lead to burnout and burnout leads to people leaving your company. So there's a case that, you know, the humans are very important to the deployment process. And for me, this is a good deployment. You know, it's kind of neat, tidy. You know where everything is. There aren't a lot of moving parts. It's just simple. Good deployments are non-events. It shouldn't be, you know, hand-ringing, palm-sweating, like, oh gosh, I hope nothing breaks event when you do a deployment. It should just happen and nobody really notices. And good deployments are also empowering. Good deployments let your, all your users, in the sense of developers, get software onto servers where, you know, it's in staging or production. That's our job as developers. You know, we build software and we want the world to see it. And sort of not letting people manage that process of getting their software into production and having gatekeepers in the middle is really kind of limiting. So, you know, empowering developers to do this stuff is pretty powerful. And then finally, there's a business case for good deployments. So if, you know, you can't convince your boss that burnout and stress is a bad thing, you know, good deployments, you ship faster. You get new features out to production quicker. You fix bugs quicker. And you recover from failure more quickly. In my notes, I had something from the State of DevOps report by Puppet Labs. And they have, you know, they did a study on the differences between high-performing IT organizations and low-performing IT organizations. And the numbers are really fascinating. If I had my notes, I would tell you what they were. But like, it's like, you know, they deploy 30 times more per day and they are, you know, 60 times faster to fail or to recover from failures and all this stuff. So there's a really big business use case here. And then also having happy people that are at your company. Happy people are more productive and they're going to leave less and you're not going to have brain drain and be training people all the time. So there's a great business case for good deployments. Now, what makes a good deployment? First off, it works. It's surprising how many times we see people get this wrong. Like, you know, they've got a deployment process and they deploy and, you know, we had an issue a while ago where we made a bunch of changes to somebody's salary worker queue and they deployed it and things broke and nothing worked. And we said, well, you know, what happened? And, you know, the one guy who was in charge of the deployment said, oh, I have to go in and restart salary manually after we do deployments. Nobody else knew that. But the one person did. And so this kind of stuff is really common. So, you know, one, it works. And it's reliable. It works every time. And, you know, in the off event, it doesn't work. It doesn't break things in the process. Next up, it's boring. This is kind of the part where I go on a little rant about Docker. Docker is really cool technology. And I think it's interesting for a lot of stuff. If you're just getting started with deployments, it's very likely not the best choice for you. Docker solves a lot of interesting problems, but it also kind of brings in a lot of problems of its own. So I prefer kind of using technology that's tried and true and proven. And Docker, you know, you end up using a lot of kind of bleeding edge technology. It's great for a lot of use cases, but for your standard, hey, I just need to get a Django site out on the web, it's probably overkill for your purposes. Next up, it's user friendly. So, you know, everybody should know how to use your deployment system. That doesn't mean they need to know all the internals of how your system works, but they should know how to get the code from their laptop to get to a server. Next up, it's fast. I'll give you an anecdote here as well. So we worked with somebody who had a deployment system, and the idea was all their tests had to pass every time before they could deploy their software. Sounds like a really good idea. You know, we want to make sure our test suite passes before we put it in production. Their test suite took something like 45 minutes to an hour to run. So when they wanted to do a deployment, they needed to wait an hour before their servers ever got updated. Well, a problem snuck through their continuous integration. You know, it didn't get caught in their tests, and they deployed a change that took down their sites. They knew what the fix was. If they could push it out, it would have been a 30-second or a minute long issue. But instead, it had to go through this hour-long pipeline to get into production. So fast deployments are really important for recovery, and they're also important for closing the feedback loop with developers. You know, we write code, we test it locally, we write tests, we put it on staging environments, but truly, we never really know how it's going to operate until we put it into production. So being able to give your developers that quick cycle of, hey, I built something, it's in production, it works, and I can move on to the next thing is really important to productivity. Next up, it's not disruptive. You shouldn't have to deploy after hours. Your deployments shouldn't take down the site. You know, even for a few seconds, people shouldn't get 500 errors when deployments happen. And then finally, a couple of $5 words here. It's adeptant and deterministic. In deployment, these words are kind of blended into each other. But basically, it's adeptant. You can run it over and over again, and you get the same thing. And it's deterministic. Given the same inputs, you get the same output. So when you deploy, you should expect that if you deploy on one server, you're going to get the exact same thing on another server. So to me, that's a good deployment. Now, how do we build all this stuff? So first off, use configuration management. It's kind of key to this all working. If you are not using configuration management today, look at Salt or Ansible. They are Python based, and they also, yeah, you're going to probably be most comfortable with those unless somebody in your shop has experience with Chef or Pop-It or one of these other systems. Fabric is not a configuration management tool. It's a remote execution tool, and you can do configuration management like things with it, but you're going to paint yourself into a corner, and you're going to have this really messy system. There's better tools out there, so don't use it for configuration management. And then use it for everything. You shouldn't configure your servers with your configuration management system and then update your software with other thing and then do deployments with another thing because that's kind of violating the don't repeat yourself principle, and you're going to end up in situations where those systems diverge and one stomps over the other. So once you kind of get everything codified into configuration management, that's what you want to use to sort of drive all this stuff. That includes setup, updates, deploys, everything. Next up, pin your dependencies. This sort of is that deterministic thing. When we deploy our software, we want to know exactly what we're getting. So that doesn't mean in your PIP requirements you put, you know, install Django or even install Django less than 1.9. There's a chance that a minor version update, you know, if there's a security issue, could be backwards incompatible. So make sure you pin to specific versions. One thing I would consider is actually just using set up that pie and putting your requirements in the install requires there. That sort of gives you a constraint that you can't link to crazy third party GitHub repos. You don't want to count on some random person on the internet's code or repo being up or existing when you're doing deployments. Now there's certain scenarios where you can't use PyPI. Maybe you've got proprietary code that's shared among multiple projects. Maybe there's, you know, you need to fork something and it's not available on PyPI. In that scenario, consider setting up your own PyPI. It's not that difficult to do a simple index, which basically is going to get you what you need. There's a project called PIP 2.PI that will take a requirements file and dump out a directory structure that's compatible with PIP and you just point PIP to that and you can install stuff from it. So copying that to a server that has Nginx or pushing it up to S3 is a pretty viable solution. Next up, if you've got code that is, you know, you've forked some project and you're diverging significantly or the other project is, you know, dead in the water and not getting updates anymore, consider just vending it. Put it in your project. The fewer external things you need to manage, the better. So just drop the code right in there and, you know, take ownership of it. And then finally, if you insist on installing out of Git repo or something like that, make sure it's when you own. Don't count on other people's stuff. Next up, this may seem obvious, but I see a lot of people that don't do this. When you deploy, you should reload and not restart your services. So the difference is, during a restart, you basically tell your WISGI server or whatever to shut down and come back up. And you're going to have a couple seconds during that time where all the requests coming into your application are going to get dropped. So what I would, you know, recommend doing instead is graceful reloads. And what that does is typically your WISGI worker is going to have multiple processes that are serving requests. It's going to let the old processes finish. It's going to bring new processes up alongside them. Once the old ones are finished, they die. And the new ones are already serving new incoming requests. Your front end proxies probably support this as well. Nginx, Farnish, HA proxy. And that just looks like a service Nginx reload on most Linux variants. It works out of the box with upstart if you're using the current Ubuntu LTS. And if you're on system D, putting this line in there, the exact reload, which is going to send the HUP signal to your WISGI server, which most of the WISGI servers will do the graceful reload when they get that signal. That'll cover it. So next up, making it user-friendly. If you're using configuration management, you might have a line like this if you're using Salt that is going to SSH into your Saltmaster server and update all your web servers. That's not good. And Ansible, you know, if you're doing Ansible, it might look like this. Also not good. It's kind of exposing all the details of your configuration management system to your developers. And you don't need to do that. Much better is something like this. I said fabric is not a configuration management system, but it is really great at remote execution. So write a deploy function. And all your deploy function does is call these other ones. And if your users, you know, are coming in and have used anything like Heroku or something like that, they're going to be much more comfortable with a script like this or a fabric command like this other than instead of the other options. While you're in there, consider adding some other stuff for your users. And most of these are wrappers around really simple, you know, kind of bash command line options. So, you know, hey, I can deploy to production. Can I easily get a branch out to a server for other people to look at it? That's a super common scenario. You know, hey, I'm working on something. Can you guys look it over? Making that scenario really easy is great for them. Another common thing, you know, hey, I need to just check the logs on this server. Something weird is happening. They shouldn't have to SSH in. Anytime somebody is SSH-ing into a server, consider that an opportunity for improvement. And, you know, the status. Hey, what version of the code is running on the server and, you know, is all our services up and stuff like that. So, fabric has this really nice dash L flag that will list all the fabric commands. Here's an example from our Lincoln Loop website. And it has a bunch of handy stuff. And we went through most of them. It also has like, hey, I need to open up a Django shell or I need to see what releases happened or roll back to a previous release. When you run those, you get something like this as a user. There's the server status. It shows you, like, the load and who's logged in and what version of the code is running and when last time you wasgiv restarted. So, next up on how we do a good deployment is isolating each build. So, this was something that wasn't really feasible a couple years ago and wasn't super easy up until a few months ago. So, what we're talking about is basically every time you build a new or you do a new release, you're building a fresh virtual M. You're putting all your dependencies in it. You're putting a copy of your code in it. Probably even a copy of your static files in it. And you sort of have this bundled, you know, directory that has everything to run this version of code. It's really nice for a few reasons. One is if you do have something that breaks down during the deployment, you're not doing it in your running version of the code. So, you know, the typical process is, hey, I've got my code on the server. I'm just going to update it on the fly and install any new requirements I have and then reload my server. If something breaks during that process, you've now broken your production site and, you know, are scrambling to fix it. So, this way, you can kind of build one in the background. Once it's all ready, you switch over to it. It helps with that determinism. You don't have any craft left over from previous builds. Has anybody here been bitten by like a stray PYC file that, you know, you deleted everything, but the PYC file is still there and imports aren't working? Yeah. So, you know, stuff like that, old versions of stuff that you've now removed from your requirements but are still in there. All those types of things can kind of bite you and prevent you from having kind of a true build. So, the other nice thing that this creates is it's really easy to roll back quickly to previous versions in the event of an issue. So, I'll show you that in a second. Here's again an example from our linkinloop.com website. It's really simple but the same thing applies for, you know, larger sites. So, we have this sort of home directory for our site. It's got a data directory in it. In that goes any sort of user generated content, stuff that isn't versioned. So, you know, user uploads and stuff like that. It's got a deploym.json file. I'm not really going to go into this, but that's kind of a 12 factor type application thing where we store secrets like API keys or your, you know, database IP and password and all that. So, that comes in from the configuration system and then the app reads it. There's a package cache which is sort of the secret to making, being able to create new virtual ends every time really fast. So, the change that happened a few months ago is that PIP, now when you install dependencies, will build Python wheels. Wheels are like binary kind of blobs of Python modules. So, if you have C extensions like common ones, Psycho, PG2, the Postgres driver or Pillow, your image manipulator, those usually take really long time to build like on the, you know, minutes. So that made it really hard to build fresh every time because your deployment got really, really slow. The nice thing is that PIP by default now creates these wheels and sticks them into a cache and we'll check there first for, if you have an updated versions, it's almost an instant install. So, using that package cache and keeping kind of outside of our versioned repo, we can reinstall everything really quickly. There's a checkout of our repo. Uwizgi cache, you can ignore it's, we're actually using Uwizgi's caching mechanism. There's our Uwizgi configuration file which tells Uwizgi how to run and then there's a virtual Ms directory. So this is what the virtual Ms directory looks like. Those are commit hashes of our code and the date, you know, each one's a directory. It's actually a virtual M and you can see the date on there that shows the time when that one was deployed. And the trick to kind of making this all work is you don't run out of a commit hash, you run out of this current directory. The current directory is just a sim link to whatever is currently running. Uwizgi points to that. Any sort of like static file serving points to that. And so updating to a new one or rolling back to a previous one, all it is is you change the sim link, you reload your services and you're on a new version. So this is really nice for a bunch of reasons. One is if you have, here's an example I ran into. So we had a site and it had dependencies and those dependencies have dependencies and one of the sub dependencies got updated and broke one of our dependencies. And that dependency wasn't pinned down the chain and so we had a working version of the code. We deployed a new version and things broke and we weren't sure why. And once we finally pinned it down, you know, you try to like release a new version or roll back to the previous version and it's still broken. You know, you're using the same code but something else down the stack is broken. So using this, in the event when those things happen, you can flip back to a previous working version of the code even if your dependency tree has broken, even if, you know, PyPI went down or GitHub went down or whatever, you always know that you have some good working version that you can quickly flip back to and that can be priceless. You know, other scenarios you run into like, hey, you deploy code and somebody put something in it that broke and that person's not available right now and you don't have time to dig in and troubleshoot it. Hey, let's just punt and roll back and, you know, we know we know we'll be safe and we don't have to muck around with our get history or anything like that. So the other interesting thing that this kind of gives you is the ability to build once and deploy everywhere and that's kind of one of the big things that people like about Docker, you know, the promise of Docker that you build and you know exactly what's in there and you can ship that image to multiple servers. So what you can do is have a build server, maybe it's Jenkins or something like that. A new commit comes in and it starts building all your dependencies into these wheels. You can build it all into, throw it all into a tarball and ship that archive off to all your servers which simply unpack it and run it. You don't need to install all your development headers on your production servers. They just live in one spot and everybody else gets pre-compiled wheels. There's a project by Armin Honahair from Did Flask and JINJA. He's over here with Century today called Platter and it's a really small Python script. I don't know, it's probably four or five hundred lines that does all this for you. So you can also stuff all your static files in there and it's a really nice way to kind of isolate Python builds and be able to ship, you know, a single file out to your servers which you can easily unpack and run. Next up is database migrations. So some of you may have noticed like if we're going to easily roll back to previous versions, how do database migrations play into that? You know, what about backwards migrations and all that? Quite frankly, I've never seen people really use backwards migrations effectively. The idea that you can, you know, move forward and then move backwards. What I would recommend instead is just maintaining backwards compatibility for a little bit until you know that your current code is good code. So if you think you're going to delete a column, don't release the code that doesn't depend on that column and delete the column in the same deploy. Release the code first and once you're sure everything's working, then release a migration which is going to delete that column. So you always maintain kind of a buffer that you can roll back. If you have a really large database and get a sufficient amount of traffic, you can run into the risk of locking your database with a migration. So, you know, maybe you're adding a new column and it needs to have a default value or something like that and, you know, your database starts ripping through all the columns. It puts a lock on the table and nobody else can write to the table until that migration completes. So if you have a scenario like that, there's a post that describes it all in detail much better than I could. It's there. It's by also in my notes Ludwig Hahn, I think is his name, and it basically gives you techniques on how to identify migrations that are going to be problematic, how to split those migrations up into multiple migrations that will allow you to do the same thing without locking your database, and it's a really great resource. And I think this is the last one, tracking cases. So you want to, when you release new code, know that new code was released and know when it was released and what was released. This is, like, hugely important for regression testing. It's really common on big high-traffic sites that, you know, you're moving fast and you're releasing new code, and a week later somebody checks your, you know, app metrics and holy cow, we're 30% slower than we were a week ago. What the heck happened? You know, and you've put in tons of new stuff in the interim. Being able to go back and see, you know, hey, on this date we released this commit and performance degraded immediately after that is huge. And it's frequent that it's not immediate. It's like, hey, I need to know from a week ago or two weeks ago or something like that. So getting this data really close to where you're tracking performance is helpful. If you pay New Relic a bunch of money, you can get them to, you can do this. Op-Beed includes it, and, you know, you can even just, like, push it to a Slack channel. It might not get it close to your performance, but at least you have that data somewhere that you can reference it later. And usually these are just like a curl API call and really easy to add into your configuration management system kind of at the last step, like, hey, new code's deployed, make a curl call to this REST API, and it'll handle all the release tracking for you. So to wrap up, the thing I've noticed with building our own configuration management system and building it for clients over the years is probably the biggest difference, the biggest, you know, thing that helped us was that we were always chipping away and making it better. If you're not using configuration management today, you don't need to sit down and stop everything and spend a week or two to, you know, build out the perfect configuration management system. Start small. Like, add, you know, next time you need to add a new user to a server, do it with a configuration management system. Next time you need to change a configuration file, you know, put that into your configuration management system and slowly chip away over time. Think of the humans that need to use this system. You know, that anecdote I gave earlier about, hey, you know, oh, yeah, we have a configuration management system, but it doesn't restart salary when you deploy. That's going to come back to bite somebody, you know, soon. So when you run into those issues, fix them. Figure out how you can, you know, anytime you do have a failure, figure out what went wrong and how you can fix it and make that fix. It's not something that you need to invest tons and tons of time into, but you need to kind of chip away at it and make things better and eventually you'll get to a system that's really, really nice and really reliable and has all those features I talked about earlier. If it's slow, look into why it's slow. Can you make it faster? Can you, you know, do some caching or something of some files like we talked about with the wheels? You know, that's one thing we ran into with our deployments. We were doing some node stuff for front-end files, you know, building stuff with Gulp and it took, I don't know, 20 or 30 seconds and I spent an hour looking into it and figured out how to, you know, cut that time in half. All that time adds up and if nobody ever looks at it, you're going to end up with a system that's slow and people don't want to use. And then finally, make it easier. When people ask you questions, when people are confused, figure out why and figure out how you can improve and make it better so, you know, people don't have questions in the future that they can empower themselves to do this, to build their own software and to put it onto your systems and that will make people happier both on the sysadmin side, that they're not, you know, constantly answering the same questions and on the development side that they're, you know, they don't need to have a gatekeeper to get this code out. So that's it. Thanks. I have a few minutes left so I'd be happy to field a few questions and I also am at the booth for Lincoln Loop over here. And yeah, feel free to come talk to me. Thanks. Thanks for a great talk. I come from a sysadmin background and in the sysadmin world, we're starting to learn all the kind of similar stuff that you've just been describing that we're coming from a perspective of, you know, things, systems that don't change for a couple of years and oh, hang on, you want to deploy twice a week or twice a day or twice an hour, whatever. The big movers and changes in the sysadmin world are people like Gene Kim who has written a couple of things, the Phoenix Project, I'm just wondering if you're familiar with that. I'll come and talk to you later. And Tom Lim and Shelley who along with a couple of co-authors has written a book which is cloud administration and they don't use the word cloud anywhere except on the front cover which is great. But basically, again, coming from a sysadmin perspective, these similar kind of things. So I was trying to ask a question rather than just deliver a comment but, you know, if you've heard of those things or not. No, I'm not familiar with them and I do have a, it was originally a sysadmin, probably not a very strong one and then became a developer. So I do have some empathy from the sysadmin side of things but, you know, I wasn't like a down in the trenches for 15 years type guy so I'll have to check those out. Thanks. Thanks. Can you talk about the SimLinks used for the virtual ends that you talked about using the current SimLinks seems like a really good way to solve a lot of problems. Is there something similar you use for the actual, for the code also so that you mentioned not leaving craft like the FTC files. Is that how you do that with the code also? So yeah, there's this repo directory here and that's a directory above the virtual ends. That is our get checkout. So we literally do a get pull on that and then we copy like a flat copy of that code directly into the virtual end and we are, all our projects we put a setup.py file in, it's like super simple but then we pip install that copy of the code into that virtual end. So it's all sort of baked in there and it's independent of the repo even if you do like a force push and blow out the commit like that is sort of, you know, baked in there and you can't mess with it. So yeah. Thank you. Hi. Again with the separate virtual ends, does that mean that your Python processes need to be restarted or can you reload those when you're switching virtual? So what we're trying to do is, yeah, that's an issue. If you're, you know, your Uwizki process or your Celery process or something like that is running from within that virtual end, yeah, you can't reload it, you know, because it's still that copy. So what we've done is sort of taken a different philosophy a little bit and looked at Uwizki and Celery more like we're looking at Nginx or Postgres. Those are like system level services. So Uwizki and Celery are going to get installed kind of on the system at a higher level and then they just point into those directories and those virtual ends which lets you kind of do all that stuff. So we don't have long running processes that are getting kicked off from Manage.py. They're getting kicked off, you know, by external kind of Python. So this morning I did the Azure Microsoft challenge and I pushed to GitHub and boom, it went live. And I was able to do that. And this feels like a really big, you know, learning curve. So at what point will I not want to just do the Microsoft Azure or Heroku or whatever method? As a beginner or as just like a kind of when you're doing small scale, smaller scale development, platform as a service is amazing. I strongly recommend it. Like you don't want to do this stuff. They probably do this stuff in the background for you. But at some point, if you know you're learning more and you're using new services or you're starting to get lots of traffic and you're starting, you know, those services can get expensive quickly when you're, you know, running large scale stuff. And at some point you start to kind of feel pain. Like you're, you know, I'm too constricted using this. I want to do something that's not possible on here or holy cow, I'm paying a ton of money and I think I could do this, you know, cheaper on my own. At that point, you know, that's when you start looking at, you know, how do I kind of manage all this stuff myself? But I'm very, it's kind of sad to me how much stuff there is that developers need to know to get a website online these days. And anything that kind of makes that smaller and prevents people from like just getting so overwhelmed that they give up is a good thing. So I'm a big fan of Pass and all that stuff. And it works great at a certain point and there's points where it doesn't. This might be a kind of out there question. If you have multiple projects that sort of depend on each other and depend on versioning. So what you were talking about like being able to easily roll back, but then what would be your suggestion for if you have that where you're deploying multiple projects completely separate, but maybe one of them messes up, you know, how would you handle that? What would be the best way to do that? So if they're not dependent on each other, you just create like two of these directories and serve them separately. The challenging part is when they start to depend on each other and there's a lot of talk on how to kind of gracefully, you know, you start talking about them like you're in a service oriented architecture and there's multiple services and how do I ensure that this service, you know, is talking the same, same, you know, messages that this service understands. That's really hard. And my take on it is like the whole microservices stuff is don't do it until you like really, really feel the pain of the big monolithic project and stick with, you know, there's, you know, really big organizations that have I think like Twitter and maybe Google too have like monolithic repos that are huge, that encompass like multiple services and everything, but then they still have like, you know, hey, there's one commit that represents a certain state of the world and, you know, this gets really hard. You know, I've done this sort of system, sorry. Stop. I've done this sort of system, you know, with like multiple projects and yeah, you can do like two commit hashes as your unique identifier, but that breaks down pretty quickly. So I don't know, hopefully that answers your question.
 Cool, thanks everybody. Sorry for the issues getting the display going. I am going without  my notes here, so hopefully I remember everything. Really quick about me. I'm the founder at  Lincoln Loop. Here's a shot of a few of us at PyCon last year, this year. And at Lincoln  Loop we do Django development and consulting and everything that goes along with building  Django sites like DevOps, JavaScript, UX design. Today we'll talk about deployment,  which is something, a topic I'm kind of interested in at the moment. I'm also the co-author of  this book, High Performance Django, and it's all about how to build Django sites that scale  and are fast. You can get a copy at highperformancejango.com and there's little coupons in your swag  bag as well. So today's talk is Django deployments done right. But really, I probably could have  called it Python deployments done right. There's not a whole lot of Django specific stuff in  here. And honestly, deployments done right probably would have fit most of it as well.  Deployments for web applications all look kind of similar. And so, you know, whether  we were doing Rails or Node or Python or Django or whatever, a lot of these things will still  apply. So first off, what is deployment? I think of it in two ways. One is the noun.  Like, the deployment is kind of all your stuff you have set up, your servers, your software,  how it all interacts, and then the verb, like the actual act of getting all that stuff onto  your servers and keeping it up to date. So today we're mostly going to be talking about  the verb, but I may interchange these a little bit. So why do you want a good deployment?  First off, uptime. This is obvious. Like, you don't want deployments to take down your  site. That's a problem. You don't want to be scared to do deployments because they're  going to take down your site. But maybe less obvious is the humans that are working with  the deployments. One thing I've noticed is having bad deployments sort of feels like  this. It's like showing up at a messy office. You can't find anything. It's stressful. You  know, there's probably one person that knows where the thing that you need is. And there's  these gatekeepers. And it's really bad for people. Like, I do a lot of consulting work  and I work on lots of different sites. And I notice the difference at the end of the  day when I'm working with a site that's got a really poor process and not well managed  and kind of a mess versus one that's kind of clean and tidy and does what I expect.  It's stressful. It tends to lead to burnout and burnout leads to people leaving your company.  So there's a case that, you know, the humans are very important to the deployment process.  And for me, this is a good deployment. You know, it's kind of neat, tidy. You know where  everything is. There aren't a lot of moving parts. It's just simple. Good deployments are  non-events. It shouldn't be, you know, hand-ringing, palm-sweating, like, oh gosh, I hope nothing  breaks event when you do a deployment. It should just happen and nobody really notices.  And good deployments are also empowering. Good deployments let your, all your users,  in the sense of developers, get software onto servers where, you know, it's in staging  or production. That's our job as developers. You know, we build software and we want the  world to see it. And sort of not letting people manage that process of getting their software  into production and having gatekeepers in the middle is really kind of limiting. So,  you know, empowering developers to do this stuff is pretty powerful.  And then finally, there's a business case for good deployments. So if, you know, you  can't convince your boss that burnout and stress is a bad thing, you know, good deployments,  you ship faster. You get new features out to production quicker. You fix bugs quicker.  And you recover from failure more quickly. In my notes, I had something from the State  of DevOps report by Puppet Labs. And they have, you know, they did a study on the differences  between high-performing IT organizations and low-performing IT organizations. And the numbers  are really fascinating. If I had my notes, I would tell you what they were. But like,  it's like, you know, they deploy 30 times more per day and they are, you know, 60 times  faster to fail or to recover from failures and all this stuff. So there's a really big  business use case here. And then also having happy people that are at your company. Happy  people are more productive and they're going to leave less and you're not going to have  brain drain and be training people all the time. So there's a great business case for  good deployments. Now, what makes a good deployment? First off, it works. It's surprising how  many times we see people get this wrong. Like, you know, they've got a deployment process  and they deploy and, you know, we had an issue a while ago where we made a bunch of changes  to somebody's salary worker queue and they deployed it and things broke and nothing worked.  And we said, well, you know, what happened? And, you know, the one guy who was in charge  of the deployment said, oh, I have to go in and restart salary manually after we do deployments.  Nobody else knew that. But the one person did. And so this kind of stuff is really common.  So, you know, one, it works. And it's reliable. It works every time. And, you know, in the  off event, it doesn't work. It doesn't break things in the process. Next up, it's boring.  This is kind of the part where I go on a little rant about Docker. Docker is really cool technology.  And I think it's interesting for a lot of stuff. If you're just getting started with  deployments, it's very likely not the best choice for you. Docker solves a lot of interesting  problems, but it also kind of brings in a lot of problems of its own. So I prefer kind  of using technology that's tried and true and proven. And Docker, you know, you end  up using a lot of kind of bleeding edge technology. It's great for a lot of use cases, but for  your standard, hey, I just need to get a Django site out on the web, it's probably overkill  for your purposes. Next up, it's user friendly. So, you know, everybody should know how to  use your deployment system. That doesn't mean they need to know all the internals of how  your system works, but they should know how to get the code from their laptop to get to  a server. Next up, it's fast. I'll give you an anecdote here as well. So we worked with  somebody who had a deployment system, and the idea was all their tests had to pass every  time before they could deploy their software. Sounds like a really good idea. You know,  we want to make sure our test suite passes before we put it in production. Their test  suite took something like 45 minutes to an hour to run. So when they wanted to do a deployment,  they needed to wait an hour before their servers ever got updated. Well, a problem snuck through  their continuous integration. You know, it didn't get caught in their tests, and they  deployed a change that took down their sites. They knew what the fix was. If they could  push it out, it would have been a 30-second or a minute long issue. But instead, it had  to go through this hour-long pipeline to get into production. So fast deployments are  really important for recovery, and they're also important for closing the feedback loop  with developers. You know, we write code, we test it locally, we write tests, we put  it on staging environments, but truly, we never really know how it's going to operate  until we put it into production. So being able to give your developers that quick cycle  of, hey, I built something, it's in production, it works, and I can move on to the next thing  is really important to productivity. Next up, it's not disruptive. You shouldn't have  to deploy after hours. Your deployments shouldn't take down the site. You know, even for a few  seconds, people shouldn't get 500 errors when deployments happen. And then finally, a couple  of $5 words here. It's adeptant and deterministic. In deployment, these words are kind of blended  into each other. But basically, it's adeptant. You can run it over and over again, and you  get the same thing. And it's deterministic. Given the same inputs, you get the same output.  So when you deploy, you should expect that if you deploy on one server, you're going  to get the exact same thing on another server. So to me, that's a good deployment. Now, how  do we build all this stuff? So first off, use configuration management. It's kind of key  to this all working. If you are not using configuration management today, look at Salt  or Ansible. They are Python based, and they also, yeah, you're going to probably be most  comfortable with those unless somebody in your shop has experience with Chef or Pop-It  or one of these other systems. Fabric is not a configuration management tool. It's a remote  execution tool, and you can do configuration management like things with it, but you're  going to paint yourself into a corner, and you're going to have this really messy system.  There's better tools out there, so don't use it for configuration management. And then  use it for everything. You shouldn't configure your servers with your configuration management  system and then update your software with other thing and then do deployments with another  thing because that's kind of violating the don't repeat yourself principle, and you're  going to end up in situations where those systems diverge and one stomps over the other.  So once you kind of get everything codified into configuration management, that's what  you want to use to sort of drive all this stuff. That includes setup, updates, deploys,  everything. Next up, pin your dependencies. This sort of is that deterministic thing.  When we deploy our software, we want to know exactly what we're getting. So that doesn't  mean in your PIP requirements you put, you know, install Django or even install Django  less than 1.9. There's a chance that a minor version update, you know, if there's a security  issue, could be backwards incompatible. So make sure you pin to specific versions. One  thing I would consider is actually just using set up that pie and putting your requirements  in the install requires there. That sort of gives you a constraint that you can't link  to crazy third party GitHub repos. You don't want to count on some random person on the  internet's code or repo being up or existing when you're doing deployments. Now there's  certain scenarios where you can't use PyPI. Maybe you've got proprietary code that's  shared among multiple projects. Maybe there's, you know, you need to fork something and it's  not available on PyPI. In that scenario, consider setting up your own PyPI. It's not that difficult  to do a simple index, which basically is going to get you what you need. There's a project  called PIP 2.PI that will take a requirements file and dump out a directory structure that's  compatible with PIP and you just point PIP to that and you can install stuff from it.  So copying that to a server that has Nginx or pushing it up to S3 is a pretty viable  solution. Next up, if you've got code that is, you know, you've forked some project  and you're diverging significantly or the other project is, you know, dead in the water  and not getting updates anymore, consider just vending it. Put it in your project. The fewer  external things you need to manage, the better. So just drop the code right in there and,  you know, take ownership of it. And then finally, if you insist on installing out of Git repo  or something like that, make sure it's when you own. Don't count on other people's stuff.  Next up, this may seem obvious, but I see a lot of people that don't do this. When you  deploy, you should reload and not restart your services. So the difference is, during  a restart, you basically tell your WISGI server or whatever to shut down and come back up.  And you're going to have a couple seconds during that time where all the requests coming  into your application are going to get dropped. So what I would, you know, recommend doing  instead is graceful reloads. And what that does is typically your WISGI worker is going  to have multiple processes that are serving requests. It's going to let the old processes  finish. It's going to bring new processes up alongside them. Once the old ones are finished,  they die. And the new ones are already serving new incoming requests. Your front end proxies  probably support this as well. Nginx, Farnish, HA proxy. And that just looks like a service  Nginx reload on most Linux variants. It works out of the box with upstart if you're using  the current Ubuntu LTS. And if you're on system D, putting this line in there, the exact reload,  which is going to send the HUP signal to your WISGI server, which most of the WISGI servers  will do the graceful reload when they get that signal. That'll cover it.  So next up, making it user-friendly. If you're using configuration management, you might  have a line like this if you're using Salt that is going to SSH into your Saltmaster  server and update all your web servers. That's not good. And Ansible, you know, if you're  doing Ansible, it might look like this. Also not good. It's kind of exposing all the details  of your configuration management system to your developers. And you don't need to do  that. Much better is something like this. I said fabric is not a configuration management  system, but it is really great at remote execution. So write a deploy function. And all your deploy  function does is call these other ones. And if your users, you know, are coming in and  have used anything like Heroku or something like that, they're going to be much more comfortable  with a script like this or a fabric command like this other than instead of the other  options. While you're in there, consider adding some other stuff for your users. And most  of these are wrappers around really simple, you know, kind of bash command line options.  So, you know, hey, I can deploy to production. Can I easily get a branch out to a server  for other people to look at it? That's a super common scenario. You know, hey, I'm working  on something. Can you guys look it over? Making that scenario really easy is great for them.  Another common thing, you know, hey, I need to just check the logs on this server. Something  weird is happening. They shouldn't have to SSH in. Anytime somebody is SSH-ing into a  server, consider that an opportunity for improvement. And, you know, the status. Hey, what version  of the code is running on the server and, you know, is all our services up and stuff  like that. So, fabric has this really nice dash L flag that will list all the fabric  commands. Here's an example from our Lincoln Loop website. And it has a bunch of handy  stuff. And we went through most of them. It also has like, hey, I need to open up a Django  shell or I need to see what releases happened or roll back to a previous release. When you  run those, you get something like this as a user. There's the server status. It shows  you, like, the load and who's logged in and what version of the code is running and when  last time you wasgiv restarted. So, next up on how we do a good deployment is isolating  each build. So, this was something that wasn't really feasible a couple years ago and wasn't  super easy up until a few months ago. So, what we're talking about is basically every  time you build a new or you do a new release, you're building a fresh virtual M. You're  putting all your dependencies in it. You're putting a copy of your code in it. Probably  even a copy of your static files in it. And you sort of have this bundled, you know, directory  that has everything to run this version of code. It's really nice for a few reasons.  One is if you do have something that breaks down during the deployment, you're not doing  it in your running version of the code. So, you know, the typical process is, hey, I've  got my code on the server. I'm just going to update it on the fly and install any new  requirements I have and then reload my server. If something breaks during that process, you've  now broken your production site and, you know, are scrambling to fix it. So, this way, you  can kind of build one in the background. Once it's all ready, you switch over to it. It  helps with that determinism. You don't have any craft left over from previous builds.  Has anybody here been bitten by like a stray PYC file that, you know, you deleted everything,  but the PYC file is still there and imports aren't working? Yeah. So, you know, stuff  like that, old versions of stuff that you've now removed from your requirements but are  still in there. All those types of things can kind of bite you and prevent you from  having kind of a true build. So, the other nice thing that this creates is it's really  easy to roll back quickly to previous versions in the event of an issue. So, I'll show you  that in a second. Here's again an example from our linkinloop.com website. It's really  simple but the same thing applies for, you know, larger sites. So, we have this sort  of home directory for our site. It's got a data directory in it. In that goes any sort  of user generated content, stuff that isn't versioned. So, you know, user uploads and  stuff like that. It's got a deploym.json file. I'm not really going to go into this, but  that's kind of a 12 factor type application thing where we store secrets like API keys  or your, you know, database IP and password and all that. So, that comes in from the configuration  system and then the app reads it. There's a package cache which is sort of the secret  to making, being able to create new virtual ends every time really fast. So, the change  that happened a few months ago is that PIP, now when you install dependencies, will build  Python wheels. Wheels are like binary kind of blobs of Python modules. So, if you have  C extensions like common ones, Psycho, PG2, the Postgres driver or Pillow, your image  manipulator, those usually take really long time to build like on the, you know, minutes.  So that made it really hard to build fresh every time because your deployment got really,  really slow. The nice thing is that PIP by default now creates these wheels and sticks  them into a cache and we'll check there first for, if you have an updated versions, it's  almost an instant install. So, using that package cache and keeping kind of outside  of our versioned repo, we can reinstall everything really quickly. There's a checkout of our  repo. Uwizgi cache, you can ignore it's, we're actually using Uwizgi's caching mechanism.  There's our Uwizgi configuration file which tells Uwizgi how to run and then there's  a virtual Ms directory. So this is what the virtual Ms directory looks like. Those are  commit hashes of our code and the date, you know, each one's a directory. It's actually  a virtual M and you can see the date on there that shows the time when that one was deployed.  And the trick to kind of making this all work is you don't run out of a commit hash, you  run out of this current directory. The current directory is just a sim link to whatever is  currently running. Uwizgi points to that. Any sort of like static file serving points  to that. And so updating to a new one or rolling back to a previous one, all it is is you change  the sim link, you reload your services and you're on a new version. So this is really  nice for a bunch of reasons. One is if you have, here's an example I ran into. So we  had a site and it had dependencies and those dependencies have dependencies and one of  the sub dependencies got updated and broke one of our dependencies. And that dependency  wasn't pinned down the chain and so we had a working version of the code. We deployed  a new version and things broke and we weren't sure why. And once we finally pinned it down,  you know, you try to like release a new version or roll back to the previous version and it's  still broken. You know, you're using the same code but something else down the stack is  broken. So using this, in the event when those things happen, you can flip back to a previous  working version of the code even if your dependency tree has broken, even if, you know, PyPI went  down or GitHub went down or whatever, you always know that you have some good working  version that you can quickly flip back to and that can be priceless. You know, other  scenarios you run into like, hey, you deploy code and somebody put something in it that  broke and that person's not available right now and you don't have time to dig in and  troubleshoot it. Hey, let's just punt and roll back and, you know, we know we know  we'll be safe and we don't have to muck around with our get history or anything like that.  So the other interesting thing that this kind of gives you is the ability to build once  and deploy everywhere and that's kind of one of the big things that people like about  Docker, you know, the promise of Docker that you build and you know exactly what's in there  and you can ship that image to multiple servers. So what you can do is have a build server,  maybe it's Jenkins or something like that. A new commit comes in and it starts building  all your dependencies into these wheels. You can build it all into, throw it all into a  tarball and ship that archive off to all your servers which simply unpack it and run  it. You don't need to install all your development headers on your production servers. They just  live in one spot and everybody else gets pre-compiled wheels. There's a project by  Armin Honahair from Did Flask and JINJA. He's over here with Century today called Platter  and it's a really small Python script. I don't know, it's probably four or five hundred lines  that does all this for you. So you can also stuff all your static files in there and it's  a really nice way to kind of isolate Python builds and be able to ship, you know, a single  file out to your servers which you can easily unpack and run.  Next up is database migrations. So some of you may have noticed like if we're going to  easily roll back to previous versions, how do database migrations play into that? You  know, what about backwards migrations and all that? Quite frankly, I've never seen people  really use backwards migrations effectively. The idea that you can, you know, move forward  and then move backwards. What I would recommend instead is just maintaining backwards compatibility  for a little bit until you know that your current code is good code. So if you think  you're going to delete a column, don't release the code that doesn't depend on that column  and delete the column in the same deploy. Release the code first and once you're sure  everything's working, then release a migration which is going to delete that column. So you  always maintain kind of a buffer that you can roll back. If you have a really large  database and get a sufficient amount of traffic, you can run into the risk of locking your  database with a migration. So, you know, maybe you're adding a new column and it needs to  have a default value or something like that and, you know, your database starts ripping  through all the columns. It puts a lock on the table and nobody else can write to the  table until that migration completes. So if you have a scenario like that, there's a  post that describes it all in detail much better than I could. It's there. It's by also  in my notes Ludwig Hahn, I think is his name, and it basically gives you techniques on how  to identify migrations that are going to be problematic, how to split those migrations  up into multiple migrations that will allow you to do the same thing without locking your  database, and it's a really great resource. And I think this is the last one, tracking  cases. So you want to, when you release new code, know that new code was released and  know when it was released and what was released. This is, like, hugely important for regression  testing. It's really common on big high-traffic sites that, you know, you're moving fast and  you're releasing new code, and a week later somebody checks your, you know, app metrics  and holy cow, we're 30% slower than we were a week ago. What the heck happened? You know,  and you've put in tons of new stuff in the interim. Being able to go back and see, you  know, hey, on this date we released this commit and performance degraded immediately after  that is huge. And it's frequent that it's not immediate. It's like, hey, I need to know  from a week ago or two weeks ago or something like that. So getting this data really close  to where you're tracking performance is helpful. If you pay New Relic a bunch of money, you  can get them to, you can do this. Op-Beed includes it, and, you know, you can even just, like,  push it to a Slack channel. It might not get it close to your performance, but at least  you have that data somewhere that you can reference it later. And usually these are  just like a curl API call and really easy to add into your configuration management  system kind of at the last step, like, hey, new code's deployed, make a curl call to this  REST API, and it'll handle all the release tracking for you.  So to wrap up, the thing I've noticed with building our own configuration management  system and building it for clients over the years is probably the biggest difference,  the biggest, you know, thing that helped us was that we were always chipping away and  making it better. If you're not using configuration management today, you don't need to sit down  and stop everything and spend a week or two to, you know, build out the perfect configuration  management system. Start small. Like, add, you know, next time you need to add a new  user to a server, do it with a configuration management system. Next time you need to change  a configuration file, you know, put that into your configuration management system and slowly  chip away over time. Think of the humans that need to use this system. You know, that anecdote  I gave earlier about, hey, you know, oh, yeah, we have a configuration management system,  but it doesn't restart salary when you deploy. That's going to come back to bite somebody,  you know, soon. So when you run into those issues, fix them. Figure out how you can,  you know, anytime you do have a failure, figure out what went wrong and how you can fix it  and make that fix. It's not something that you need to invest tons and tons of time into,  but you need to kind of chip away at it and make things better and eventually you'll get  to a system that's really, really nice and really reliable and has all those features  I talked about earlier. If it's slow, look into why it's slow. Can you make it faster?  Can you, you know, do some caching or something of some files like we talked about with the  wheels? You know, that's one thing we ran into with our deployments. We were doing some  node stuff for front-end files, you know, building stuff with Gulp and it took, I don't  know, 20 or 30 seconds and I spent an hour looking into it and figured out how to, you  know, cut that time in half. All that time adds up and if nobody ever looks at it, you're  going to end up with a system that's slow and people don't want to use. And then finally,  make it easier. When people ask you questions, when people are confused, figure out why and  figure out how you can improve and make it better so, you know, people don't have questions  in the future that they can empower themselves to do this, to build their own software and  to put it onto your systems and that will make people happier both on the sysadmin side,  that they're not, you know, constantly answering the same questions and on the development  side that they're, you know, they don't need to have a gatekeeper to get this code out.  So that's it. Thanks. I have a few minutes left so I'd be happy to field a few questions  and I also am at the booth for Lincoln Loop over here. And yeah, feel free to come talk  to me. Thanks.  Thanks for a great talk. I come from a sysadmin background and in the sysadmin world, we're  starting to learn all the kind of similar stuff that you've just been describing that  we're coming from a perspective of, you know, things, systems that don't change for a couple  of years and oh, hang on, you want to deploy twice a week or twice a day or twice an hour,  whatever. The big movers and changes in the sysadmin world are people like Gene Kim who  has written a couple of things, the Phoenix Project, I'm just wondering if you're familiar  with that. I'll come and talk to you later. And Tom Lim and Shelley who along with a couple  of co-authors has written a book which is cloud administration and they don't use the  word cloud anywhere except on the front cover which is great. But basically, again, coming  from a sysadmin perspective, these similar kind of things. So I was trying to ask a question  rather than just deliver a comment but, you know, if you've heard of those things or not.  No, I'm not familiar with them and I do have a, it was originally a sysadmin, probably  not a very strong one and then became a developer. So I do have some empathy from the sysadmin  side of things but, you know, I wasn't like a down in the trenches for 15 years type guy  so I'll have to check those out. Thanks. Thanks.  Can you talk about the SimLinks used for the virtual ends that you talked about using the  current SimLinks seems like a really good way to solve a lot of problems. Is there something  similar you use for the actual, for the code also so that you mentioned not leaving craft  like the FTC files. Is that how you do that with the code also?  So yeah, there's this repo directory here and that's a directory above the virtual  ends. That is our get checkout. So we literally do a get pull on that and then we copy like  a flat copy of that code directly into the virtual end and we are, all our projects we  put a setup.py file in, it's like super simple but then we pip install that copy of the code  into that virtual end. So it's all sort of baked in there and it's independent of the  repo even if you do like a force push and blow out the commit like that is sort of,  you know, baked in there and you can't mess with it. So yeah. Thank you.  Hi. Again with the separate virtual ends, does that mean that your Python processes  need to be restarted or can you reload those when you're switching virtual?  So what we're trying to do is, yeah, that's an issue. If you're, you know, your Uwizki  process or your Celery process or something like that is running from within that virtual  end, yeah, you can't reload it, you know, because it's still that copy. So what we've  done is sort of taken a different philosophy a little bit and looked at Uwizki and Celery  more like we're looking at Nginx or Postgres. Those are like system level services. So Uwizki  and Celery are going to get installed kind of on the system at a higher level and then  they just point into those directories and those virtual ends which lets you kind of  do all that stuff. So we don't have long running processes that are getting kicked off from  Manage.py. They're getting kicked off, you know, by external kind of Python.  So this morning I did the Azure Microsoft challenge and I pushed to GitHub and boom,  it went live. And I was able to do that. And this feels like a really big, you know, learning  curve. So at what point will I not want to just do the Microsoft Azure or Heroku or whatever  method? As a beginner or as just like a kind of when you're doing small scale, smaller  scale development, platform as a service is amazing. I strongly recommend it. Like you  don't want to do this stuff. They probably do this stuff in the background for you. But  at some point, if you know you're learning more and you're using new services or you're  starting to get lots of traffic and you're starting, you know, those services can get  expensive quickly when you're, you know, running large scale stuff. And at some point you start  to kind of feel pain. Like you're, you know, I'm too constricted using this. I want to  do something that's not possible on here or holy cow, I'm paying a ton of money and I  think I could do this, you know, cheaper on my own. At that point, you know, that's  when you start looking at, you know, how do I kind of manage all this stuff myself? But  I'm very, it's kind of sad to me how much stuff there is that developers need to know  to get a website online these days. And anything that kind of makes that smaller and prevents  people from like just getting so overwhelmed that they give up is a good thing. So I'm a  big fan of Pass and all that stuff. And it works great at a certain point and there's  points where it doesn't.  This might be a kind of out there question. If you have multiple projects that sort of  depend on each other and depend on versioning. So what you were talking about like being  able to easily roll back, but then what would be your suggestion for if you have that where  you're deploying multiple projects completely separate, but maybe one of them messes up,  you know, how would you handle that? What would be the best way to do that?  So if they're not dependent on each other, you just create like two of these directories  and serve them separately. The challenging part is when they start to depend on each  other and there's a lot of talk on how to kind of gracefully, you know, you start talking  about them like you're in a service oriented architecture and there's multiple services  and how do I ensure that this service, you know, is talking the same, same, you know,  messages that this service understands. That's really hard. And my take on it is like the  whole microservices stuff is don't do it until you like really, really feel the pain of the  big monolithic project and stick with, you know, there's, you know, really big organizations  that have I think like Twitter and maybe Google too have like monolithic repos that are huge,  that encompass like multiple services and everything, but then they still have like,  you know, hey, there's one commit that represents a certain state of the world and, you know,  this gets really hard. You know, I've done this sort of system, sorry. Stop. I've done  this sort of system, you know, with like multiple projects and yeah, you can do like two commit  hashes as your unique identifier, but that breaks down pretty quickly. So I don't know,  hopefully that answers your question.
DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_0_155.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_156_287.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_288_395.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_396_1787.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_1788_2687.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_2688_2783.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_2784_3359.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_3360_3479.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_3480_3755.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_3756_3959.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_3960_4391.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_4392_14219.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_14220_14999.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_15000_15239.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_15240_15419.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_15420_20843.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_20844_21071.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_21072_23579.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_23580_25979.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_25980_26615.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_26616_26939.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_26940_27227.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_27228_30143.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_30144_39647.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_39648_49031.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_49032_50015.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_50016_51095.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_51096_51575.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_51576_51671.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_51672_53783.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_53784_54023.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_54024_54191.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_54192_55175.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_55176_55799.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_55800_56747.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_56748_57347.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_57348_58415.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_58416_58619.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_58620_58751.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_58752_59771.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_59772_60095.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_60096_60659.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_60660_60983.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_60984_61031.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_61032_61247.png DjangoCon_US_2015_-_Django_Deployments_Done_Right_by_Peter_Baumgartner_SUczHTa7WmQ_61248_61250.png