Releasing Software with GitOps
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
McGinnis, Sean
Overview of the process the OpenStack community uses to manage all software releases through automation around Git commits. The OpenStack community has enabled a lot of automation around releasing their software. This presentation will give an overview of how code reviews are used to manage release activity. It will step through the use of Zuul CI jobs to perform validation of requests before they are accepted, and how commits are used to trigger jobs to tag and release the software, update documentation, trigger requirements updates, and other follow on work that needs to happen whenever new code is released.
 Hello, I'm Sean McGinnis. I run the open source office at Dell. But I'm here talking about things that I've been doing in the open stack community for the last several years. And not that I've been doing, I've been doing it recently. A lot of this has been stuff that's built up over years of people making improvements and making things better. So I just get to present all the great work that's been done. So in the open stack community, we need to release all the work that is being done. If you're not familiar with it, within open stack there's quite a few different projects. They each have different deliverables, usually several deliverables per project. And we need a way to package that up and make that available to downstream consumers or whoever needs that. So like typical release steps, we need to be able to compile or validate, make sure that that source code is good. We want to make sure whatever we're delivering and releasing actually is something that someone can pick up and use. We need to take the repos. We need to be able to know exactly at what point any given deliverable was released so that we can go back later. We need to create whatever artifacts and this varies depending on what the deliverable is. For us, most of the open stack is Python. So that's packages that PIP can install, our tar balls of the source itself. There's some node, NPM packages, things like that. We need to publish those somewhere. It doesn't do much good just to create them. We need to make them available so they need to get published on a website. They need to get pushed up to a package repository where I can get pulled down. And typically, even if you're just doing this as something within your own company, you need to send some kind of announcement, let people know that there's something new that is out there that they can pick up and use. So we've tried to automate all these steps. This is something that's evolved over time. Initially, we tried to just trigger on new takes being pushed, which was great. That took care of a lot of things. That does take care of a lot of manual steps. But there were some issues with that approach. There was inconsistent use of versioning. We didn't actually follow Simver for a while, but semantic versioning, if you're not aware of it, simver.org. Really great way to make sure that you're communicating by your version number if that version contains breaking changes or if there's just bug fixes. And if you leave that up to individual teams to do, you just kind of have to hope that they're aware of what those rules are and that they actually follow them. The packages that were being published weren't always what you wanted. They could maybe grab the wrong commit, miss something important. There's the times where the wrong commit gets tagged. And you end up releasing something that really, well, now it's out there. You can't really easily go back and take that down. And for stable branches, it's easy to go back and create branches. But if you know you're doing something for a past cycle, you want to make sure you have a branch there and that everything's in place and makes it easy to make sure that the right changes are going to the right place. And publishing that, making that information available, we have releases.opensack.org. We want to list everything on there. If that information can't easily be extracted, then it's a manual process of going through and finding what versions are where and getting that published. So we try to automate more of the things. And not only that, but make it a process that really fits into the processes that the developers already are comfortable with, with doing code reviews, pushing changes up just like they push code changes. So the process evolved where we actually are able to leverage that tooling we have in place for the development workflow. We use Garrett for code review. It makes it really easy to have anybody propose something and then you can review what changes. And we have Zool CI that is great at testing these changes and being able to have dependencies between things, making sure that if you have a dependency that one change merges before the next. And the nice thing about Zool for us being a mostly Python project, it's Ansible, very Zool friendly, very easy to extend and work with Python with that. So our typical workflow for just code changes is that we have a check queue. So when someone proposes a change, that goes through check, tests are run like any kind of CI, and those tests need to pass before you can even get beyond that step. And then once it's approved, it goes through the gate, really runs those tests again, makes sure that nothing has changed in the meantime when you initially test it and right before you merge that into the code base. And then it gets merged into the final repo, the code, where everybody else can pull that down. So we tried to do the same thing with the release process. And the way we do that is we have a YAML file for each deliverable. So each team, each deliverable out of that team, has a YAML file that is per cycle. So there's a nice folder structure. You can very easily go in and see each cycle exactly what is released, which commit hash it is, all the information. It contains release information. It contains any branching information. And then as a release team, we were able to kind of spread the load of doing this release because then anybody can propose changes to this YAML file. And the release team just needs to review that, just like reviewing a code change. And then Zoole really does a lot in there in what triggers that automation or what enables that automation of being able to run tests against these changes and go out and use the information that we capture in the YAML file to do a lot of validation. So what it really looks like is this. It's just YAML. We capture information like the name of the team that is responsible for that deliverable. We capture information about the deliverable itself that lets us programmatically evaluate certain things, knowing what type of thing it is, like knowing that this is a library. There's certain things that we want to enforce for something that's a library. We have information if there's anything unique about the repo that we can capture. This specific repo maybe needs to be called something else when we package it up and provide it as a deliverable. And then the actual release information itself, where we can say on this commit hash, I need this release version xyz. And then if there's anything to branch, we can also include that and what to call it. So it's all captured. Basically, the idea here is that everything that we need to know to make decisions about the proposed release and know what it is and what the release name is going to be and all of that is all captured with any YAML file. And I know everyone loves to edit YAML, but in that same spirit of trying to automate things, that also then allows us to provide some tooling on the proposer's side. They can just run this new release command and tell us what series and what deliverable and what type. And then that is able to go and add that release information into the YAML file so that they don't need to manually edit it. That has the smarts where, if you say type here is probably the very useful thing, I think type is, is it a major release? Is it a feature change? Is it a bug fix? And just by telling the tool this is a bug fix, it is smart enough to go and look, because we have all of that history already in the repo, in the release's repo, it can go back and read the existing deliverable file and see, the last version was 1.5.0. They're requesting a bug fix release. Therefore, I know this is 1.5.1. So it encapsulates a lot of that logic into the tooling. So anybody within the community that needs to do a release can run the command. And they don't need to know all the little details about how those little decisions that you need to decide what version number and which file and things like that. So here's an example. New release for the train cycle, for the sender deliverable, and it's a feature release. So this last little part there, if you can see that, it just added another version. It automatically goes, and because I told it it's in the train cycle, by default it'll look at the train branch and pick the most recent commits. So it gets the commit hash. And because I told it it's a feature release, last version was 1.1.0. And it knows, OK, now I'm going to bump that up and call it 1.2.0. Makes it nice and easy. So now that they've made that change, they're able to commit that code and submit it as a code review, just like they would do if they're editing a Python file or whatever, what have you. And then as the release team, that makes it really easy that we can go in and we can see here's the specific thing that's being added to this file, which honestly, because of the automation in place and the jobs that we were able to run in CI, don't really need to worry too much about this. Because usually, if the YAML is messed up, those jobs will find that. But it's still easy if we want to look at it exactly what's happening here. You can look at that. And because it's done as a code review, it highlights exactly the differences in the file that are being proposed. But we do leverage those CI jobs. Just like our development workflow, we have that check in gate queue that runs tests. And because of all of the information that we capture in the YAML file, there are quite a few things that we're able to automate in a validation job that looks at the change being made so that we know things can pass. And then we know if we approve that change, everything's good for the most part. But I'll go into some detail about some of the other things here that we can't. And then once it gets through check passes, we do a review, make sure everything's good, approve it. It does those tests one more time through the gate queue. And that's when that actually merges. It kicks off some of the automation that actually does all those tasks I mentioned in the beginning of packaging things up and then doing the release. So I can dig a little more into that. So this is what it looks like in the Garrett UI. It runs these jobs, reports them back. One of the useful things is building the docs. Like I briefly mentioned, because we have this information in the YAML file, we're able to extract that information and automatically use that to publish to the releases site so that we have all of the release details on the site. So because those docs are really integral to doing our release, it's part of our overall release process, we want to make sure that we can build those docs. So we use RST and Sphinx to generate those. There's a Sphinx extension that we have that will read in that YAML file. And from that, extract all the information it needs to plug into the actual documentation itself. And then that's what gets published on the website all automatically. So we want to make sure that those docs actually build. If there's some error in how they've entered information in the YAML file, likely the docs will fail, or there's some issue that we need to take a look at. So we don't want anything to be passing tests and look happy if it ends up that. If we actually approve that, there's going to be issues afterwards. It's all about catching the errors earlier in the process of making sure things are right. There are a bunch of validation checks. And this is where there's a lot of benefit. This is an area where we've been able to build up over time. Because there are certain things that we know we need to look for, we don't have to rely on the release team as a code review to know or to remember to go and look at all these things. So there's validation jobs just to make sure that all the information is good. The job will actually clone the repo of the deliverable and make sure things like the commit hash exists, which has actually been a very common issue that I typoed something if I didn't use that tool. Something's not quite right here, especially on stable branches. Maybe they want to make a change or a bug fix in a stable branch, but they merged it into the master branch, which will be the next release, and forgot that they'd need to take that extra step and actually backport it into the old stable branch. So this is a great way. Right away it sees, oh, no, this commit hash is not here. So really prevent some errors there. Make sure that the package can be built. Readme files don't have any errors, because that's used in the package, publish package information. Make sure that there's permissions. This was a common one. Everything's built. Everything package is fine. And then we go to publish the package, and oh, we actually don't have permissions to do that. And by that point, everything else is all done. So this just makes it easier. Right away we can validate, oh, yep, we can do this. And version numbers, things like that. So this has been an area, as we've run across different corner cases and different things that, oh, I never thought of that. We can add that into our validation, and then it's able to go and check that automatically. And we don't have to be as humans remembering, oh, I should check this, make sure this weird case isn't going to happen here. It's all built in. And this is, I think, one of the most useful things. This is all about automating how this process is done. But there are certain subjective things that need an actual human to take a look at. So rather than just leaving that up to us as a release team to go out and check these things, at least we can use the automation to pull all of the things that we need to look at into one place so that when it comes time to evaluate this release to see if everything is good, it's not a matter of going to five different places and pulling together information on our own. The automation has already pulled that into the output of the job. So it makes it much easier if we just pull up the results of that job. We're able to see all of the commits that are included, any requirements changes. And those two are really useful because one of the most common things that we might have to go back and say, no, I think you need to change this, is maybe they've called it a bugfix release. But we look at some of these changes here, and they've added a new dependency. We see that one of the commits that emerged into the repo really changes, adds a new feature, or even removes something that makes it backwards incompatible so that we have that information. And then we can say, with semantic versioning, you've requested this version number. Really with what you have in here, I think it makes more sense to go with 2.0 or something, which if we didn't have the automation pulling all this together for us, would really be time consuming for us to go out and look at the repo, look at the commit history, and pull all that information. We, it gets the lead of that project. And if there is one, I release the liaison. So that's part of decentralizing this process, is anybody in the community can propose a release. But if somebody just wants their bugfix available right away, we don't want to just do the releases all the time. We want to make sure that the people that are responsible for these deliverables actually can be aware that a release is being requested. And they need to be the ones to say, plus one on the review, because we don't want to release something and then have that team come back later and say, hey, why did you do this? Why is this out here? This way we can make sure that we know who we need to get feedback from to make sure that, yeah, OK, this is good. We're ready to go. And then things like generating a release announcement, we can see what exactly is going to be sent out to the community on the mailing list before it's actually sent out to the world. And sometimes there's errors or something like that that we want to make sure get caught. So after we approve the release request, so that YAML file change actually does get committed into the repo, that triggers some post jobs. That'll actually publish those things generated HTML docs to the release site. So then it'll be available out there. It'll show that this new release is available. And that's what this looks like here. If you can see that, it's a list of all of the deliverables, all the services with the first release of this release cycle, and the most recent release of this release cycle. There are links to release notes that we're able to capture in that YAML file. All this information that gets published out there, very extensive information. And we don't need someone to sit down and actually fill out all this information, because it can just pull it right out of the YAML files. Taking process will get details from the commit. And then it's able to add the tag to the repo and include all that metadata so that there's a lot of really good tracking in there. And because we know where bugs are being tracked for that individual project and what commits are included in that release, it's able to go and actually update the bug tracker and say this based on information in the commit message to say this is now released. And if it needs to, it'll create the branch. So no human is actually creating the branch. The automation does it all. We manage our requirements by, if there are any, libraries that we release updating. Before we have everybody start using that new release, we run some jobs and make sure that doesn't break anything, any unintended consequences. So it'll propose a change to our requirements process. It'll publish the package, whichever it is, and it'll send the release announcement. Basically, what I go through there. So to go back to what that looks like, so the check and gate queue, those are just the validation on the change in the deliverable YAML file itself where we're able to get that information and decide is this release correct? Is it using the right version number? After it's committed, this post-release process runs, that actually is what creates the tag in the repo. And then we'll publish the package and get all the information out there. So overall ideas of what we do and then overall ideas of what you can maybe think about of how you can use some of these ideas for how you manage your releases, whether it's a community open source project or something internally or your company. Automate those repetitive tasks. If you're doing releases for whatever deliverable it is, see what pieces of that process you can have a script to. Whether you can fully automate it and just have it taken care of, or our job that gets those changes have a job that can at least pull together information and minimize the amount of brain power that you need to expend on evaluating these releases. Automate security tests. What I mean about this is things like creating those branches, pushing those tags in the Git repo, signing tar balls, all of that we're able to offload to the tooling. So it's not a human that's accessing these secret keys and doing these things. The tool does it. And by the time the tool runs, we know we've vetted everything, and then the tool can access the secrets that it needs to do that. No human needs to do it. Automate even the things that are manual as much as you can, and then keep looking for things that'll make it better. The validation job that we have, we've just kept improving, adding new checks, adding new things that really get rid of all those surprises for us. And I think having, using Git and using the same workflow, or very, very similar workflow that we do for changes to the code of the project, makes it a lot easier for everyone or anyone to be able to step in and then also do releases. So that's it. You can always follow up afterwards and ask me any questions. Here's my Twitter handle. I think we have a couple minutes, five minutes. So if anybody has any questions, I can try to answer them. Yes? Hi. I was interested, because we've done a lot with GitOps, and we mostly think of the things as being item-potent. It feels like your releases will have a point where if something goes wrong, you're going to have to do it all again. It's not item-potent. It's item-potent up to a point, and then not. It should always be item-potent. Sorry, we should repeat the question. Sorry. So the question was, is it always that important? GitOps should be something that you can do over and over and not have different results every time. Yes, so if someone has put in a release and a version number and that YAML file gets merged, we can make other changes to that YAML file. And when that merges, the automation will see what has changed here. It will not attempt to take that commit again. It'll check, OK, I have hash ABC, version 123. OK, hash ABC has take 123. And then it'll move on. Same thing with branches. If a branch exists, it won't try to do that again. That's actually kind of bit us a couple of times where something, for whatever reason, someone had to manually delete a branch. And then we make a change that YAML file and we'll say, oh, I need to create this branch. So it'll always make sure that whatever is requested in the YAML file actually exists out there. If it already exists out there, it won't attempt to do any other operations on it. Yes? I have one question now because at the beginning, this YAML file, do these exist in their own separate repository? Oh, good. There's always in the repository where the product is. So the question is about where these YAML files exist, whether it's in the repo with the product or elsewhere. We have a releases repo. So it is a separate repo by itself, which is nice because then it contains a lot of the automation, not all of it, but a lot of the automation exists right there. And then there's within that repo, there is the directory structure of deliverables, the name of the cycle, and then the deliverable.yaml file, which makes it a really nice reference. Everything's self-contained in this releases repo. If you're someone out in the community, you don't care about releases, you don't have all these files in your repo, you need to know to go to this releases repo to make this change and propose the release. Yeah. Any other questions? Yes. Just rather a comment and a question. As a consumer of this process for the past few years, and I still remember the first release we were pushing about five years ago that I was involved into myself and the current release, PPL at the time, about an hour sitting together on IRC and figuring out every single step to get the package out there. And the last release, what we did on November, I think it took me about five minutes total time because the release team had the patch already. Only thing what I had to go and do was check the comment hash, I go, oh, this is not the right one, I know we still are pending merging one more hash. Comment, once we got that into the repo, we changed the hash in the reviews tooling. And my demo was done. I didn't need to worry anything after that. And I knew that everything was published and out there when the time hit the timeline. So it's amazing how much a bunch of automation and verification can help from the actual man hours spent on simple task, so-called simple task as a release. Which is not, but it definitely wasn't five years ago, I think it has to release the offer. Great to hear. Well, for the sake of the recording, to really summarize that, about a few years ago, it took about an hour of the project lead and the release manager together to get everything in place to do a release. And now this last one took maybe five minutes of time. So that's great to hear. And that's exactly the benefit of doing something like this that we're trying to accomplish. Good? Can you tell us how much of a learning curve is to start using these tools? Like somebody that's completely familiar with them? What is the learning curve to use to request a release? Or to? You start using the tools and the job, all the way should be proposed and then you release and push it out. OK. Well, I'll go through from both. So from the consuming side, on our releases.opensec.org, there's documentation, reference documentation, it says with using. And then that kind of goes through the steps as a consumer of this automation of here are the things that I need to do to edit the CMO file and request the release through requesting a code review. And then for the release team, getting people on board to participate as release managers, it's really important to have a good understanding of what is going on in the CMO. There's a bit of a learning curve because there, like I said, this is something that we've added more and more validation over time every time that we found something that, hey, we could have checked for this ahead of time. The overall concept is pretty easy to grasp. And then if anything goes wrong or if you need to dig into some kind of change, it's just a matter of knowing where in the automation in the scripts you need to go to look at it. You have to get repo and do some get commands and the rest is fairly basic Python code. Reviewing those release requests takes a little time just knowing for that semi-manual step of just knowing all of the information that's included in that job output, what do I need to look at, what does it mean, and just understanding things like semantic versioning. Good question. Thank you very much. Thanks. Thank you.
 Hello, I'm Sean McGinnis.  I run the open source office at Dell.  But I'm here talking about things that I've been doing in  the open stack community for the last several years.  And not that I've been doing,  I've been doing it recently.  A lot of this has been stuff that's built up over years of  people making improvements and making things better.  So I just get to present all the great work that's been done.  So in the open stack community, we need to release all the work  that is being done.  If you're not familiar with it, within open stack there's  quite a few different projects.  They each have different deliverables, usually several  deliverables per project.  And we need a way to package that up and make that available  to downstream consumers or whoever needs that.  So like typical release steps, we need to be able to compile  or validate, make sure that that source code is good.  We want to make sure whatever we're delivering and releasing  actually is something that someone can pick up and use.  We need to take the repos.  We need to be able to know exactly at what point any given  deliverable was released so that we can go back later.  We need to create whatever artifacts and this varies  depending on what the deliverable is.  For us, most of the open stack is Python.  So that's packages that PIP can install,  our tar balls of the source itself.  There's some node, NPM packages, things like that.  We need to publish those somewhere.  It doesn't do much good just to create them.  We need to make them available so they need to get published  on a website.  They need to get pushed up to a package repository where I can  get pulled down.  And typically, even if you're just doing this as something  within your own company, you need to send some kind of  announcement, let people know that there's something new  that is out there that they can pick up and use.  So we've tried to automate all these steps.  This is something that's evolved over time.  Initially, we tried to just trigger on new takes being  pushed, which was great.  That took care of a lot of things.  That does take care of a lot of manual steps.  But there were some issues with that approach.  There was inconsistent use of versioning.  We didn't actually follow Simver for a while, but  semantic versioning, if you're not aware of it, simver.org.  Really great way to make sure that you're communicating by  your version number if that version contains breaking  changes or if there's just bug fixes.  And if you leave that up to individual teams to do, you  just kind of have to hope that they're aware of what those  rules are and that they actually follow them.  The packages that were being published weren't always what  you wanted.  They could maybe grab the wrong commit, miss something  important.  There's the times where the wrong commit gets tagged.  And you end up releasing something that really, well,  now it's out there.  You can't really easily go back and take that down.  And for stable branches, it's easy to go back and create  branches.  But if you know you're doing something for a past cycle,  you want to make sure you have a branch there and that  everything's in place and makes it easy to make sure that  the right changes are going to the right place.  And publishing that, making that information available, we  have releases.opensack.org.  We want to list everything on there.  If that information can't easily be extracted, then it's a  manual process of going through and finding what versions  are where and getting that published.  So we try to automate more of the things.  And not only that, but make it a process that really fits  into the processes that the developers already are  comfortable with, with doing code reviews, pushing changes  up just like they push code changes.  So the process evolved where we actually are able to leverage  that tooling we have in place for the development workflow.  We use Garrett for code review.  It makes it really easy to have anybody propose something  and then you can review what changes.  And we have Zool CI that is great at testing these  changes and being able to have dependencies between things,  making sure that if you have a dependency that one change  merges before the next.  And the nice thing about Zool for us being a mostly Python  project, it's Ansible, very Zool friendly, very easy to  extend and work with Python with that.  So our typical workflow for just code changes is that we  have a check queue.  So when someone proposes a change, that goes through  check, tests are run like any kind of CI, and those tests  need to pass before you can even get beyond that step.  And then once it's approved, it goes through the gate,  really runs those tests again, makes sure that nothing has  changed in the meantime when you initially test it and  right before you merge that into the code base.  And then it gets merged into the final repo, the code, where  everybody else can pull that down.  So we tried to do the same thing with the release process.  And the way we do that is we have a YAML file for each  deliverable.  So each team, each deliverable out of that team, has a YAML  file that is per cycle.  So there's a nice folder structure.  You can very easily go in and see each cycle exactly what is  released, which commit hash it is, all the information.  It contains release information.  It contains any branching information.  And then as a release team, we were able to kind of spread  the load of doing this release because then anybody can  propose changes to this YAML file.  And the release team just needs to review that, just like  reviewing a code change.  And then Zoole really does a lot in there in what triggers  that automation or what enables that automation of being able  to run tests against these changes and go out and use the  information that we capture in the YAML file to do a lot of  validation.  So what it really looks like is this.  It's just YAML.  We capture information like the name of the team that is  responsible for that deliverable.  We capture information about the deliverable itself that lets  us programmatically evaluate certain things, knowing what  type of thing it is, like knowing that this is a library.  There's certain things that we want to enforce for something  that's a library.  We have information if there's anything unique about the  repo that we can capture.  This specific repo maybe needs to be called something else  when we package it up and provide it as a deliverable.  And then the actual release information itself, where we  can say on this commit hash, I need this release version xyz.  And then if there's anything to branch, we can also include  that and what to call it.  So it's all captured.  Basically, the idea here is that everything that we need to  know to make decisions about the proposed release and know  what it is and what the release name is going to be and all  of that is all captured with any YAML file.  And I know everyone loves to edit YAML, but in that same  spirit of trying to automate things, that also then allows  us to provide some tooling on the proposer's side.  They can just run this new release command and tell us  what series and what deliverable and what type.  And then that is able to go and add that release information  into the YAML file so that they don't need to manually edit  it.  That has the smarts where, if you say type here is probably  the very useful thing, I think type is, is it a major  release?  Is it a feature change?  Is it a bug fix?  And just by telling the tool this is a bug fix, it is smart  enough to go and look, because we have all of that history  already in the repo, in the release's repo, it can go  back and read the existing deliverable file and see,  the last version was 1.5.0.  They're requesting a bug fix release.  Therefore, I know this is 1.5.1.  So it encapsulates a lot of that logic into the tooling.  So anybody within the community that needs to do a release  can run the command.  And they don't need to know all the little details about how  those little decisions that you need to decide what version  number and which file and things like that.  So here's an example.  New release for the train cycle, for the sender  deliverable, and it's a feature release.  So this last little part there, if you can see that, it just  added another version.  It automatically goes, and because I told it it's in the  train cycle, by default it'll look at the train branch and  pick the most recent commits.  So it gets the commit hash.  And because I told it it's a feature release, last version  was 1.1.0.  And it knows, OK, now I'm going to bump that up and call it  1.2.0.  Makes it nice and easy.  So now that they've made that change, they're able to  commit that code and submit it as a code review, just like  they would do if they're editing a Python file or  whatever, what have you.  And then as the release team, that makes it really easy  that we can go in and we can see here's the specific thing  that's being added to this file, which honestly, because of  the automation in place and the jobs that we were able to  run in CI, don't really need to worry too much about this.  Because usually, if the YAML is messed up, those jobs will  find that.  But it's still easy if we want to look at it exactly what's  happening here.  You can look at that.  And because it's done as a code review, it highlights  exactly the differences in the file that are being proposed.  But we do leverage those CI jobs.  Just like our development workflow, we have that check  in gate queue that runs tests.  And because of all of the information that we capture in  the YAML file, there are quite a few things that we're able  to automate in a validation job that looks at the change  being made so that we know things can pass.  And then we know if we approve that change, everything's  good for the most part.  But I'll go into some detail about some of the other  things here that we can't.  And then once it gets through check passes, we do a review,  make sure everything's good, approve it.  It does those tests one more time through the gate queue.  And that's when that actually merges.  It kicks off some of the automation that actually does  all those tasks I mentioned in the beginning of packaging  things up and then doing the release.  So I can dig a little more into that.  So this is what it looks like in the Garrett UI.  It runs these jobs, reports them back.  One of the useful things is building the docs.  Like I briefly mentioned, because we have this information  in the YAML file, we're able to extract that information and  automatically use that to publish to the releases site so  that we have all of the release details on the site.  So because those docs are really integral to doing our  release, it's part of our overall release process, we  want to make sure that we can build those docs.  So we use RST and Sphinx to generate those.  There's a Sphinx extension that we have that will read in  that YAML file.  And from that, extract all the information it needs to plug  into the actual documentation itself.  And then that's what gets published on the website all  automatically.  So we want to make sure that those docs actually build.  If there's some error in how they've entered information  in the YAML file, likely the docs will fail, or there's  some issue that we need to take a look at.  So we don't want anything to be passing tests and look happy  if it ends up that.  If we actually approve that, there's going to be issues  afterwards.  It's all about catching the errors earlier in the process  of making sure things are right.  There are a bunch of validation checks.  And this is where there's a lot of benefit.  This is an area where we've been able to build up over time.  Because there are certain things that we know we need to  look for, we don't have to rely on the release team as a  code review to know or to remember to go and look at  all these things.  So there's validation jobs just to make sure that all the  information is good.  The job will actually clone the repo of the deliverable and  make sure things like the commit hash exists, which has  actually been a very common issue that I typoed something  if I didn't use that tool.  Something's not quite right here, especially on stable  branches.  Maybe they want to make a change or a bug fix in a stable  branch, but they merged it into the master branch, which  will be the next release, and forgot that they'd need to  take that extra step and actually backport it into  the old stable branch.  So this is a great way.  Right away it sees, oh, no, this commit hash is not here.  So really prevent some errors there.  Make sure that the package can be built.  Readme files don't have any errors, because that's used in  the package, publish package information.  Make sure that there's permissions.  This was a common one.  Everything's built.  Everything package is fine.  And then we go to publish the package, and oh, we actually  don't have permissions to do that.  And by that point, everything else is all done.  So this just makes it easier.  Right away we can validate, oh, yep, we can do this.  And version numbers, things like that.  So this has been an area, as we've run across different  corner cases and different things that, oh, I never  thought of that.  We can add that into our validation, and then it's  able to go and check that automatically.  And we don't have to be as humans remembering, oh, I  should check this, make sure this weird case isn't going to  happen here.  It's all built in.  And this is, I think, one of the most useful things.  This is all about automating how this process is done.  But there are certain subjective things that need an  actual human to take a look at.  So rather than just leaving that up to us as a release team  to go out and check these things, at least we can use  the automation to pull all of the things that we need to  look at into one place so that when it comes time to  evaluate this release to see if everything is good, it's  not a matter of going to five different places and pulling  together information on our own.  The automation has already pulled that into the  output of the job.  So it makes it much easier if we just pull up the results of  that job.  We're able to see all of the commits that are included,  any requirements changes.  And those two are really useful because one of the most  common things that we might have to go back and say, no, I  think you need to change this, is maybe they've called it a  bugfix release.  But we look at some of these changes here, and they've  added a new dependency.  We see that one of the commits that emerged into the repo  really changes, adds a new feature, or even removes  something that makes it backwards incompatible so that  we have that information.  And then we can say, with semantic versioning, you've  requested this version number.  Really with what you have in here, I think it makes more  sense to go with 2.0 or something, which if we  didn't have the automation pulling all this together for  us, would really be time consuming for us to go out and  look at the repo, look at the commit history, and pull all  that information.  We, it gets the lead of that project.  And if there is one, I release the liaison.  So that's part of decentralizing this process,  is anybody in the community can propose a release.  But if somebody just wants their bugfix  available right away, we don't want to just do the  releases all the time.  We want to make sure that the people that are responsible  for these deliverables actually can be aware that a  release is being requested.  And they need to be the ones to say, plus one on the review,  because we don't want to release something and then have  that team come back later and say, hey, why did you do this?  Why is this out here?  This way we can make sure that we know who we need to get  feedback from to make sure that, yeah, OK, this is good.  We're ready to go.  And then things like generating a release announcement, we  can see what exactly is going to be sent out to the  community on the mailing list before it's actually sent out  to the world.  And sometimes there's errors or something like that that we  want to make sure get caught.  So after we approve the release request, so that  YAML file change actually does get committed into the  repo, that triggers some post jobs.  That'll actually publish those things generated HTML docs to  the release site.  So then it'll be available out there.  It'll show that this new release is available.  And that's what this looks like here.  If you can see that, it's a list of all of the deliverables,  all the services with the first release of this release  cycle, and the most recent release of this release cycle.  There are links to release notes that we're able to  capture in that YAML file.  All this information that gets published out there, very  extensive information.  And we don't need someone to sit down and actually fill out  all this information, because it can just pull it right out  of the YAML files.  Taking process will get details from the commit.  And then it's able to add the tag to the repo and include all  that metadata so that there's a lot of really good tracking  in there.  And because we know where bugs are being tracked for that  individual project and what commits are included in that  release, it's able to go and actually update the bug tracker  and say this based on information in the commit  message to say this is now released.  And if it needs to, it'll create the branch.  So no human is actually creating the branch.  The automation does it all.  We manage our requirements by, if there are any, libraries  that we release updating.  Before we have everybody start using that new release, we  run some jobs and make sure that doesn't break anything, any  unintended consequences.  So it'll propose a change to our requirements process.  It'll publish the package, whichever it is, and it'll  send the release announcement.  Basically, what I go through there.  So to go back to what that looks like, so the check and  gate queue, those are just the validation on the change in  the deliverable YAML file itself where we're able to get  that information and decide is this release correct?  Is it using the right version number?  After it's committed, this post-release process runs,  that actually is what creates the tag in the repo.  And then we'll publish the package and get all the  information out there.  So overall ideas of what we do and then overall ideas of  what you can maybe think about of how you can use some of  these ideas for how you manage your releases, whether it's  a community open source project or something  internally or your company.  Automate those repetitive tasks.  If you're doing releases for whatever deliverable it is,  see what pieces of that process you can have a script to.  Whether you can fully automate it and just have it taken  care of, or our job that gets those changes have a job that  can at least pull together information and minimize the  amount of brain power that you need to expend on evaluating  these releases.  Automate security tests.  What I mean about this is things like creating those  branches, pushing those tags in the Git repo,  signing tar balls, all of that we're able to offload to the  tooling.  So it's not a human that's accessing these secret keys  and doing these things.  The tool does it.  And by the time the tool runs, we know we've vetted  everything, and then the tool can access the secrets that it  needs to do that.  No human needs to do it.  Automate even the things that are manual as much as you can,  and then keep looking for things that'll make it better.  The validation job that we have, we've just kept  improving, adding new checks, adding new things that really  get rid of all those surprises for us.  And I think having, using Git and using the same workflow, or  very, very similar workflow that we do for changes to the  code of the project, makes it a lot easier for everyone or  anyone to be able to step in and then also do releases.  So that's it.  You can always follow up afterwards and ask me any  questions.  Here's my Twitter handle.  I think we have a couple minutes, five minutes.  So if anybody has any questions, I can try to  answer them.  Yes?  Hi.  I was interested, because we've done a lot with GitOps, and  we mostly think of the things as being item-potent.  It feels like your releases will have a point where if  something goes wrong, you're going to have to do it all  again.  It's not item-potent.  It's item-potent up to a point, and then not.  It should always be item-potent.  Sorry, we should repeat the question.  Sorry.  So the question was, is it always that important?  GitOps should be something that you can do over and over and  not have different results every time.  Yes, so if someone has put in a release and a version number  and that YAML file gets merged, we can make other changes  to that YAML file.  And when that merges, the automation will see what has  changed here.  It will not attempt to take that commit again.  It'll check, OK, I have hash ABC, version 123.  OK, hash ABC has take 123.  And then it'll move on.  Same thing with branches.  If a branch exists, it won't try to do that again.  That's actually kind of bit us a couple of times where  something, for whatever reason, someone had to manually  delete a branch.  And then we make a change that YAML file and we'll say, oh,  I need to create this branch.  So it'll always make sure that whatever is requested in the  YAML file actually exists out there.  If it already exists out there, it won't attempt to do any  other operations on it.  Yes?  I have one question now because at the beginning, this YAML  file, do these exist in their own separate repository?  Oh, good.  There's always in the repository where the product is.  So the question is about where these YAML files exist,  whether it's in the repo with the product or elsewhere.  We have a releases repo.  So it is a separate repo by itself, which is nice because  then it contains a lot of the automation, not all of it, but  a lot of the automation exists right there.  And then there's within that repo, there is the directory  structure of deliverables, the name of the cycle, and then  the deliverable.yaml file, which makes it a really nice  reference.  Everything's self-contained in this releases repo.  If you're someone out in the community, you don't care  about releases, you don't have all these files in your repo,  you need to know to go to this releases repo to make this  change and propose the release.  Yeah.  Any other questions?  Yes.  Just rather a comment and a question.  As a consumer of this process for the past few years, and I  still remember the first release we were pushing about five  years ago that I was involved into myself and the current  release, PPL at the time, about an hour sitting together on  IRC and figuring out every single step to get the package  out there.  And the last release, what we did on November, I think it  took me about five minutes total time because the release  team had the patch already.  Only thing what I had to go and do was check the comment  hash, I go, oh, this is not the right one, I know we still  are pending merging one more hash.  Comment, once we got that into the repo, we changed the  hash in the reviews tooling.  And my demo was done.  I didn't need to worry anything after that.  And I knew that everything was published and out there when  the time hit the timeline.  So it's amazing how much a bunch of automation and  verification can help from the actual man hours spent on  simple task, so-called simple task as a release.  Which is not, but it definitely wasn't five years  ago, I think it has to release the offer.  Great to hear.  Well, for the sake of the recording, to really summarize  that, about a few years ago, it took about an hour of the  project lead and the release manager together to get  everything in place to do a release.  And now this last one took maybe five minutes of time.  So that's great to hear.  And that's exactly the benefit of doing something like this  that we're trying to accomplish.  Good?  Can you tell us how much of a learning curve is to start  using these tools?  Like somebody that's completely familiar with them?  What is the learning curve to use to request a release?  Or to?  You start using the tools and the job, all the way  should be proposed and then you release and push it out.  OK.  Well, I'll go through from both.  So from the consuming side, on our releases.opensec.org,  there's documentation, reference documentation,  it says with using.  And then that kind of goes through the steps as a  consumer of this automation of here are the things that I  need to do to edit the CMO file and request the release  through requesting a code review.  And then for the release team, getting people on board to  participate as release managers, it's really  important to have a good understanding of what  is going on in the CMO.  There's a bit of a learning curve because there, like I  said, this is something that we've added more and more  validation over time every time that we found something  that, hey, we could have checked for this ahead of time.  The overall concept is pretty easy to grasp.  And then if anything goes wrong or if you need to dig into  some kind of change, it's just a matter of knowing where in the  automation in the scripts you need to go to look at it.  You have to get repo and do some get commands and the rest is  fairly basic Python code.  Reviewing those release requests takes a little time  just knowing for that semi-manual step of just  knowing all of the information that's included in that job  output, what do I need to look at, what does it mean, and  just understanding things like semantic versioning.  Good question.  Thank you very much.  Thanks.  Thank you.
testing_releasing_software_gitops_0_131.png testing_releasing_software_gitops_132_983.png testing_releasing_software_gitops_984_1079.png testing_releasing_software_gitops_1080_3791.png testing_releasing_software_gitops_3792_3995.png testing_releasing_software_gitops_3996_4367.png testing_releasing_software_gitops_4368_4451.png testing_releasing_software_gitops_4452_5639.png testing_releasing_software_gitops_5640_6899.png testing_releasing_software_gitops_6900_6995.png testing_releasing_software_gitops_6996_7511.png testing_releasing_software_gitops_7512_7883.png testing_releasing_software_gitops_7884_8639.png testing_releasing_software_gitops_8640_9707.png testing_releasing_software_gitops_9708_11495.png testing_releasing_software_gitops_11496_11615.png testing_releasing_software_gitops_11616_11759.png testing_releasing_software_gitops_11760_12563.png testing_releasing_software_gitops_12564_12911.png testing_releasing_software_gitops_12912_13127.png testing_releasing_software_gitops_13128_13619.png testing_releasing_software_gitops_13620_16043.png testing_releasing_software_gitops_16044_17063.png testing_releasing_software_gitops_17064_17519.png testing_releasing_software_gitops_17520_18539.png testing_releasing_software_gitops_18540_20075.png testing_releasing_software_gitops_20076_20291.png testing_releasing_software_gitops_20292_22271.png testing_releasing_software_gitops_22272_25619.png testing_releasing_software_gitops_25620_30455.png testing_releasing_software_gitops_30456_31115.png testing_releasing_software_gitops_31116_31871.png testing_releasing_software_gitops_31872_31931.png testing_releasing_software_gitops_31932_32855.png testing_releasing_software_gitops_32856_33527.png testing_releasing_software_gitops_33528_33599.png testing_releasing_software_gitops_33600_33683.png testing_releasing_software_gitops_33684_33755.png testing_releasing_software_gitops_33756_35699.png testing_releasing_software_gitops_35700_36623.png testing_releasing_software_gitops_36624_46499.png testing_releasing_software_gitops_46500_46607.png testing_releasing_software_gitops_46608_46859.png testing_releasing_software_gitops_46860_48215.png testing_releasing_software_gitops_48216_48368.png