reno: A New Way to Manage Release Notes
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Hellmann, Doug
reno is a tool for managing release notes in projects that support multiple branches of development, and releases, simultaneously. It solves the problem of managing release notes within patches that fix bugs, and makes it easier to cherry-pick changes between branches (allowing backports or forward ports). This talk will cover the requirements, and constraints, that led us to design and build reno. I will also show how to use it to create notes and publish them in your packages and via Sphinx-based documentation. Then I will talk about the impact reno, and the release notes publishing automation, had on our project and release processes.
 Thank you, Martin. Yeah, so as Martin said, my name is Doug Hellman. I'm here to talk to you today about a tool called Reno, which we built as part of the OpenStack community to manage our release notes. It's not really relevant to the talk itself, but for those of you who might not know what OpenStack is, I'll give you sort of the elevator pitch. It is the Cloud Management Platform, Infrastructure Management Platform, written in Python, that gives you features like you might find in AWS or Azure or things like that. And it's all open source, and you can run it on your hardware in your own data center. We consider release notes to be an important aspect of communicating with our users. Those of you who build libraries are probably used to explaining about what you've done in each release, or if you've got an application that you distribute to other users and don't just host yourself, you're probably used to doing the same thing, sort of explaining what kinds of challenges you might run into to do an upgrade, or what new features are present, or what other kinds of changes are there. With OpenStack in particular, because of the complexity, release notes are a very important part of our documentation. And so I'm going to talk a little bit about a tool that we created to help us manage those in a way that let us maintain them more sustainably. We created it early in 2015, as our community was growing and we were seeing not just an increase in the number of different services and components to OpenStack itself, but the contributors and the contributions that were coming in. So we started out with six or seven service projects, each managing an API and a Python client library to talk to that API. And then over time, we grew. But in the early days, each project was sort of responsible for doing their own thing with release notes. So we had a wiki, and people would just go and manually put notes into the wiki. Maybe they would keep up to date with what they were doing, and frequently they really wouldn't. So at the end of a release cycle, we would spend a bunch of time going through all of the work that we had done. And our release cycles are every six months. So that meant going back through six months worth of work for all of those projects and trying to understand which ones were user-facing changes, what added new configuration options, what might impact the upgrade, and things like that. That turned into a real challenge for us as we grew, in particular because of the amount of changes that we were seeing. So even at a point where we were still doing just six different projects, the changes ramped up very quickly. And it also turned into a problem when we were backboarding changes. So our development model is that we do all of our development on the master branch. And then we produce a stable release every six months. And the stable release branch will see updates over the period of six to 18 months, depending on how long it's maintained. And those changes are limited to bug fixes. So we call it stable because we don't backport features. And we hope that that makes it a little more stable than it would otherwise be. But those bug fixes typically come with changes that need release notes as well. So as we saw more and more of those branches created and more and more patches and fixes going into them, that turned into a problem for us too. And then we really hit a stride where we were growing as a community. So we grew from six projects, teams, to over 60 different teams now. And we have those 60 teams managing 300 different or more deliverable objects. So we talk about deliverables, meaning an artifact that we build and ship to someone. So that's usually a library, but it's often a service in some sort of package as well. And the manual processes that we were using for six teams just were not going to scale to 60 teams and 300 different deliverable objects. We did scale out the release team. So we originally had one person managing all of the releases. And he was doing a lot of work by hand. The team doubled in size when I joined it. But I'm extremely lazy. So that really didn't actually double the amount of work that we were able to do. Instead of trying to do all of that work by hand, I started thinking about ways that we could apply automation. So as a good programmer, I immediately tried to write a program to solve the problem that we had. So I have not yet been able to figure out a way to automate writing for release notes. So I set that aside. And instead, focused on making it easier for other people to do that work. So making it easier to manage the release notes throughout the course of developing the project over a release cycle, as well as automating the publishing of those release notes. So we include them within the package that we distribute. But we also want to publish them on our website so that they're easy to find and Google-able and that sort of thing. When we sat down as a team to look at our different requirements for this project, we kind of divided them into two different groups. So we had some content requirements about how we wanted to manage content and some process requirements about how we wanted to make our process sustainable as we grew. Peer review is a very important aspect of our community culture. Everything that we do within OpenStack is peer reviewed. So every bit of code is posted for review and normally reviewed by two reviewers at least before it gets approved and merged. We also wanted to be able to organize the content. So I mentioned several different kinds of release notes, like upgrades, impacts, bug fixes, and incompatibilities and things like that. We wanted to be able to organize our release notes as they were published into those different kinds of notes so that you could go and read all about what you needed to know to do an upgrade in one section and not necessarily worry about new features in that section. We also wanted to be able to change the release notes. So we don't assume, even with the peer review process, that we're going to catch any mistakes or not miss something or have some other kind of error. So we knew we wanted to be able to go back later and change notes in old releases. And then it's very important due to the pace that we work at that we want to avoid merge conflicts. So it can typically take a week or more to get a patch reviewed and merged even at a fast review period. Some of them take much longer than that. And every time you encounter a merge conflict, you basically have to start over because you have to rebase the patch and resubmit it. It has to pass all of the tests again. And then you have to get reviewers to take a look at it again. So we wanted to design a system that just completely avoided having to deal with merge conflicts as a contributor. On the process side, we had some other kinds of similar requirements. We use two different kinds of versioning for our deliverables. So libraries are versioned using semantic versioning, which means that you increment the version number based on the kinds of changes that are in the new version, not the number of changes that are in the new version. For those kinds of things, we don't know in advance when you write a release note what the next version is going to be. So we didn't want release note authors to even have to think about version numbers. And we wanted them to be able to just contribute a note, and it would automatically be applied to the correct version when the release was created. I mentioned our stable branch policy. We wanted to make sure that not just that we avoided merge conflicts on the master branch, but that we also avoided merge conflicts in the backported patches. Right now, we use Garrett as a review tool, and you can backport a patch basically clicking a couple of buttons in a web page, and it will create the new patch on the new branch for you and apply it automatically. And as soon as you start talking about merge conflicts there, you raise the bar for backporting a bug fixed. It makes it much harder to do that. So we wanted to avoid merge conflicts there as well. And then, of course, since the release team is very lazy, we didn't want any manual processes involved in publishing the notes at the end of a release. So we didn't want to have the author of the release note have to worry about versioning. We didn't want the release team to have to sit down and mark version numbers on a bunch of release notes or anything like that either. We just wanted all of that stuff to be handled automatically. We went through three different designs in the course of trying to figure out how to meet all of those requirements. So the first one was sort of the obvious thing. We were already using Sphinx for documentation. We could just use Sphinx for release notes. You could just write your release notes in a document and include it in Sphinx. That has a couple of different problems. So either the release note author has to know the version number so they put it in the right place in the right file, or someone has to go back later and apply the version number. So that didn't meet either of those requirements for us. And it's also messy for cherry picking changes, particularly if you organize things in a directory structure in some way. When you cherry pick a change back, you then have to edit that change to put the note in the correct place for the older branch because the version number system is different. We also thought about using the get commit messages. So just having the patch author write the release note write in the get commit message as they were contributing the patch. That would work mostly, but the audience for a commit message is very different from the audience for a release note. So for a library, it might be similar. But really, we wanted people to be focused on writing commit messages for the reviewers to understand what was going on in the patch and not necessarily the user having to go and read the get history to understand everything that had changed. So we wanted them to have either a condensed version or an expanded version that explained in more detail for the correct audience for those release notes. The commit messages are also largely immutable. So once it's in that public repository, your notes are basically set, and that's done. We also looked at the get notes feature, which is, I don't know a lot of the detail about this one, but it's sort of a parallel thing to the get repository. And you can do a little bit of extra setup and have notes attached to commits, but not be part of the commit. And that was an interesting approach until we realized that you needed to do a bunch of extra setup with Garrett in order to make it work, and the permissions were different, and it wasn't really a reviewable thing. So it fell down on our peer review criteria there. And so we finally ended up with a system that we implemented as Reno, where we used data files inside of the commit to hold the release note content. But they're not Sphinx files. They're just independent data files. And then we do some integration work to make those publishable. Because they're inside the patch, they're reviewable. So the reviewers can comment on them and reject them, and you can make changes and submit a new version. We do read the release notes data out of the get objects. So it doesn't matter where they are on the file system. They're all basically in one directory together. And we look at the get history to figure out which versions they go in, because they're part of a commit. We can tell which version the commit is in. And so that's what Reno does. Reno uses those data files and basically uses the get history as a database, reading those particular files, and then assembling them in whatever form you've asked to build the release notes report. It's a command line tool. So you start out with the Reno add command, and you give it a slug just to sort of be able to identify what the file is. Frequently, people will use bug numbers, or they'll use a short abbreviation of the feature. We have a specification process, which gives everything a sort of unique name. So sometimes the notes files will have those as part of the names as well. And then Reno adds a unique identifier to that slug that you give it, so that it can track the history of that file across all of the branches and through all of the revisions that you have. The files are placed in a subdirectory of the project. Like I said, they're all just placed in one directory together. They can be organized a little bit under that directory. Some teams have divided them up into different kinds of organizations. It doesn't really make any difference for that. Reno, because it's not reading the file system, it doesn't really care what files the notes are actually in. And the data file themselves are YAML files. So this is an abbreviated version of what one of those would look like. The section, the keys, there are section names, and those are predefined but configurable. So if you have a project, and we have several projects outside of OpenStack that are using Reno that have a different set of features that they want to use or sections within their release notes that they want to use, you can change what those are. And then Reno knows what order to apply the sections and then which sections each note goes in. Within the section, within the data file, you have a list of restructured text blobs that are just inserted in the order that they're presented. And using restructured text lets us integrate easily with Sphinx. It also lets us do things like link off to bug reports or feature descriptions or specifications or that sort of thing. So you get all of that nice markup. And we use restructured text rather than HTML or something like that, because we do actually generate PDF documentation for some of our translated sites, especially those behind the Great Wall in China. It's easier for them to download APDF and share it than it is sometimes to get to websites that are outside of China. So the output from Reno, we can take a look at an example repository. So this is a typical graph showing the history with the beginning at the bottom and the most current commit at the top there. And I have one stable branch that I've created just to be able to show what a cherry pick is going to look like. There's two different versions are tagged. So we have a 2.0.0 version that has four commits. And if we assume that we're all good developers and care about our users, we have a release note in each of those. We're going to have four release notes for 2.0.0. And then version 1.0 is the first two commits there. And that's going to include two release notes. So if we look at the output of the report command for the 2.0 version, we can see those four release notes. I just included the letter names there so that you can tell that they're from each of those commits. And it does that. I've included them all in the new features section just because that made it easier to fit on the slide. But of course, you can organize those into different sections as appropriate. Reno figures out which notes to include by starting at the tag that you've given it as a start point. And it reads the git history one commit at a time looking for release note changes in those files. And then it assembles those into its database for that version. And it emits them in the order that they've been presented. And it knows that those notes apply to that version because they appear either on the commit that is tagged or before the commit that is tagged with the next version number or the previous version number, I suppose. So as it's scanning backwards in history, it's going to find the 2.0 version first. And then it will find the 1.0 version following that. So Reno does the scanning using a pure Python library called Dullich, which is an implementation of the git library. We chose that one because it's pip installable without any other dependencies, which makes it easier to manage and work on release notes on various different platforms. So there are a bunch of libraries that actually interface with git. But this one, because you can just pip install it, it means folks that are on a Mac or on a Windows system, they don't have to figure out how to install a C library in order to do the work with Reno. And using Dullich has proved to be considerably more reliable than using the porcelain output from git. So the first version of Reno actually ran git as a command and then read the output and tried to parse it. And then somewhere in there, the porcelain formatting changed. And we were no longer getting some of the data that we needed in order to track changes to files. So we had to rewrite it using a library instead. So if we look at the stable branch, if we ask it to scan the history of a branch, instead of asking for a specific version number, we get two versions as output. So using git's sort of standard version nomenclature of tacking on a number of patches after a tag, we get a 1.0.0-1 as a version. And that's actually an unreleased version that represents the change that's on that branch after the tag. And then a 1.0 version that includes the a and the b commits. So scanning a branch doesn't necessarily stop at the previous release. So you can actually ask it, give me the entire history of a series of releases on a given branch. And that's useful, particularly in the way that we publish releases. So we name these stable branches. So Rocky is the current branch that we're in. And they're alphabetical order. So based on what the naming convention doesn't matter. But they're in alphabetical order. So we have a page for each project, for each series. And we can say, show me all of the notes for that series altogether. And you get all the versions that have been released, all the patch updates, and everything. So I talked about backporting changes from one branch to another. So if we take that commit e and we copy it using cherry pick from the master branch onto the stable branch, we get the note that goes with the code change. And it ends up on that stable branch. And then if we tag a new version at the end of that stable branch, 1.0.1, meaning that it's a patch update with a bug fix in it, we get the release notes for 1.0.1 will include a copy of the note that was in the 2.0 release that's in that e commit. And we can do that without making any changes to e. Because it has a unique file name for the release note. And assuming the rest of the code backports cleanly, you can just backport and approve the patch. Now, that doesn't always happen. But we didn't want Reno to introduce reasons to have backport problems. So we're still not 100% there. But we're doing the best we can. And the notes for 1.0.1 will look like this. So that includes that e release. And then the previous commit that was there that hadn't been tagged yet. So we get both notes. When we publish our notes, we publish them from one job that happens when we tag a commit. And so we actually have, as I mentioned, we have a different page for each release series. And we publish all of that together at one time. When a user reads the release notes, though, they don't read all of them. They go and look at the version that they're installing. And they want to see the release notes for that version or that series of versions. And so having a note copied from one branch to another like this means that the user doesn't have to jump around and find the release notes in different places based on where the fix was originally applied and then backported to. That was a real problem for us with the Wiki approach, because people would backport the code and then forget to go and copy the release note around. And so after the initial release on a stable branch, basically the release notes were not really being updated very often. In addition to doing a backport like that, I mentioned that we needed to be able to change a patch, change a release note in case there was a typo or something like that. So the way that works is you change the release note file at any point in the history of the branch after it's been added. And as Reno scans through the history, it finds the change first because it's starting with the newest patch in the branch and going backwards in history. But it doesn't actually insert the content from the release note until it finds the add event where that file was initially created. So it takes the content from the newest version of the file and inserts it at the location in the history of the original file. And that lets us change the published version of the notes for an older release after that release has been cut. We can't obviously change the content of the package for the 2.0 release. So if you're looking at the release notes that are inside the package, we can't do anything about that. But we can change them what we actually publish online. And so this is the output for looking at the 2.0 release after that change has been applied. You can see that there's no typo in the release note anymore. And it's been changed outside of that. It also understands that if you delete a release note file, if for some reason the note doesn't apply at all and you want to delete it, when it sees that delete, it basically ignores the file. So it doesn't add it later. So if you have a completely incorrect release note that somehow passed through all of the peer review, you can just delete the whole thing and it will be ignored. All right, I've shown you basic command line output for Reno. It does also integrate with Sphinx. That was one of the key benefits for using restructured text. So there's a Sphinx directive to insert the release notes for a given series or version into the output of whatever your Sphinx documentation is. This particular directive, without any extra arguments, is going to insert the full history of the current branch. So that's really useful on the master branch because you basically get everything that's happening in the development series. On to publish the documentation for a stable branch, you would specify that stable branch name. Or if you just want to have a version or a couple of versions, you can include a range or a set of version numbers in there as well. So it's easy to organize your published release notes in whatever way you want. Basically, any arguments you can pass to the report command, you can pass here through the Sphinx directive. And then the output, depending on what your styles are going to look like, your CSS styles, it's going to look something like this. So you get version numbers inserted with headers, and you get all of your sections with titles. And then the release notes are inserted in order. And we do maintain a consistent order, so it's not necessarily predictable as you're writing the release notes, but we're going to insert them in the order that I think it's sorted based on those unique IDs, so that it's consistent, so that if you publish a new version, you don't suddenly scramble all of your release notes into a different order. Right, so that's the tool. The effect that had on the community was pretty impressive, even beyond what we had anticipated. So we've definitely increased the quantity of release notes that we're writing. We are at something like 13,000 notes for those 300 different projects now, which is far more than we had, even relative to the six projects at the beginning. More importantly, though, we're changing the culture. So in addition to encouraging people to write release notes, some of the teams are actually requiring that new features come with release notes, new configuration options come with release notes. Less so for bug fixes, unless there's a significant upgrade impact or something like that. But that's not a pervasive thing throughout the community. It's a few of the teams that are a little more interested and have a little more impact in the sort of thing that a release note would document. And I consider that a win for my own laziness, because I don't have to deal with release notes at all at this point. It's basically every once in a while, there's a bug report against the tool, and I go work on that. But I don't have to deal with the actual documentation. So if you're interested in trying Reno out, we would love to hear from you. It is set up by default to work using the OpenStack naming conventions and version conventions and patterns and that sort of thing. But it's very configurable. I think almost everything that would be a useful behavior has a configuration option now. But if it's not, then we're interested in making it more usable for more projects. If you're not interested in Reno, I'm still interested in having you write better release notes and actually write release notes. You might be interested in looking at a couple of other tools. TownCryor is a tool out of the Twisted community, which I didn't know about at the time that we built Reno. But I believe it has the problem of requiring you to know either know in advance the version numbers or do something at the release point to deal with version numbers. And so it wouldn't have met our needs anyway, but it's still a good tool. And then Blurb is a tool that Larry Hastings wrote for the Python Dev community. And it has a similar sort of problem. And both of those teams are releasing one thing, and so it's not really a big deal for their release managers to take a little bit of action at the point of the release. But for us, with the scale of the number of things we're touching, we didn't want to have to deal with any of those kinds of manual steps. The documentation for Reno is on the OpenStack website, so docs.openstack.org slash Reno. And this sample repository that I used for this presentation is on my GitHub account, if you want to play with that. And I'll be posting the slides for the presentation online as part of the conference website. And that's all I have, so if anybody has any questions, I'd be happy to talk. All right. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thanks. Sorry. Ah, if anybody has any questions, please raise the hand. Thank you for on this presentation. Just a quick question, if I understand. For now, it's only on forget, but do you have any plan for any other VCS? And second part of the question, would it be easy to contribute and well for other VCS? Is it just like callbacks plus settings, or is it like really forking the whole stuff? Yeah, so I had not seriously considered other version control tools yet, but I'm interested in talking to you about what, which one in particular are you interested in? Perforce. OK. I don't know anything about that, so it'd be up to you to do that. I could help you. There is, I think, a point at which we could add some logic that would let you switch back and forth between those two things in the scanner in particular. I don't know whether Perforce's data model is similar to Git's. It really relies on being able to walk through the history and see the topological sort of the commits and that sort of thing. But yeah, I'd be happy to talk to you after about some of the details for that. That would be interesting. Sure. OK, next question, please. Basically the same, but for Mercurial. Basically the same answer. OK, let's see. Oh, over there. It might be easier for Mercurial because I think the data model is closer there. I didn't quite understand how it knew when to stop going back in history. So you gave an example where you ran it on Rocky. Yes. And I went back and then got MB, which were from Master. How did it not go back to, for example, the commits that came before Queens or something previous? Sure. Yeah. What's the termination condition, basically? OK, so I very drastically simplified the example in the presentation. So the open stack, we're on the Rocky release cycle right now. That means we have branches all the way back through A, potentially for some projects. And it does a little bit of work to figure out where the previous stable branch came from, and it stops before it gets to that point. So that's not really, it's very difficult to represent that in a slide. And the hand waving is probably indicative of that. But yeah, it does have some smarts for doing that. And you can turn that off. So projects that don't have that kind of stable branch structure could basically ignore that feature and disable that. But yeah. OK, we have time for one more final question. Thank you for the talk. I understand that individual release nodes are committed also to Master. So my question is, if you make a mistake in your release node, how do you commit without force pushing to Master? Yeah, so we don't do any force pushing ever. All the changes are iterative. So you apply another patch on top. And then Reno, as it's scanning that history, it sees a change to a file because it's scanning backwards in history. It'll see the change before it sees the ad. So to hold on to the contents from the patch where you've changed it, and when it gets to the point where you've added the file six months ago or a year ago or whatever, then it will insert the node at that point in the history using the newer contents. And it can do that because it's not reading the files that you've commit off the file system. It's reading them out of the Git objects in the Git history, if that makes sense. OK, thank you very much for the presentation. Let's have a round of applause for Ben. Thank you all very much for coming. Thanks.
 Thank you, Martin.  Yeah, so as Martin said, my name is Doug Hellman.  I'm here to talk to you today about a tool called Reno,  which we built as part of the OpenStack community  to manage our release notes.  It's not really relevant to the talk itself,  but for those of you who might not know what OpenStack is,  I'll give you sort of the elevator pitch.  It is the Cloud Management Platform,  Infrastructure Management Platform, written in Python,  that gives you features like you might find in AWS  or Azure or things like that.  And it's all open source, and you can run it  on your hardware in your own data center.  We consider release notes to be an important aspect  of communicating with our users.  Those of you who build libraries are probably  used to explaining about what you've done in each release,  or if you've got an application that you distribute to other  users and don't just host yourself,  you're probably used to doing the same thing,  sort of explaining what kinds of challenges  you might run into to do an upgrade,  or what new features are present,  or what other kinds of changes are there.  With OpenStack in particular, because of the complexity,  release notes are a very important part of our documentation.  And so I'm going to talk a little bit about a tool  that we created to help us manage those in a way that  let us maintain them more sustainably.  We created it early in 2015, as our community was growing  and we were seeing not just an increase  in the number of different services and components  to OpenStack itself, but the contributors and the contributions  that were coming in.  So we started out with six or seven service projects,  each managing an API and a Python client library  to talk to that API.  And then over time, we grew.  But in the early days, each project  was sort of responsible for doing their own thing  with release notes.  So we had a wiki, and people would just  go and manually put notes into the wiki.  Maybe they would keep up to date with what they were doing,  and frequently they really wouldn't.  So at the end of a release cycle,  we would spend a bunch of time going through all of the work  that we had done.  And our release cycles are every six months.  So that meant going back through six months worth of work  for all of those projects and trying  to understand which ones were user-facing changes, what  added new configuration options, what  might impact the upgrade, and things like that.  That turned into a real challenge for us as we grew,  in particular because of the amount of changes  that we were seeing.  So even at a point where we were still doing just  six different projects, the changes ramped up very quickly.  And it also turned into a problem  when we were backboarding changes.  So our development model is that we do all of our development  on the master branch.  And then we produce a stable release every six months.  And the stable release branch will  see updates over the period of six to 18 months,  depending on how long it's maintained.  And those changes are limited to bug fixes.  So we call it stable because we don't backport features.  And we hope that that makes it a little more stable  than it would otherwise be.  But those bug fixes typically come with changes that need  release notes as well.  So as we saw more and more of those branches created  and more and more patches and fixes going into them,  that turned into a problem for us too.  And then we really hit a stride where  we were growing as a community.  So we grew from six projects, teams,  to over 60 different teams now.  And we have those 60 teams managing 300 different or more  deliverable objects.  So we talk about deliverables, meaning an artifact  that we build and ship to someone.  So that's usually a library, but it's often  a service in some sort of package as well.  And the manual processes that we were using for six teams  just were not going to scale to 60 teams and 300  different deliverable objects.  We did scale out the release team.  So we originally had one person managing all of the releases.  And he was doing a lot of work by hand.  The team doubled in size when I joined it.  But I'm extremely lazy.  So that really didn't actually double the amount of work  that we were able to do.  Instead of trying to do all of that work by hand,  I started thinking about ways that we could apply automation.  So as a good programmer, I immediately  tried to write a program to solve the problem that we had.  So I have not yet been able to figure out a way  to automate writing for release notes.  So I set that aside.  And instead, focused on making it easier for other people  to do that work.  So making it easier to manage the release notes  throughout the course of developing the project  over a release cycle, as well as automating the publishing  of those release notes.  So we include them within the package that we distribute.  But we also want to publish them on our website  so that they're easy to find and Google-able  and that sort of thing.  When we sat down as a team to look at our different  requirements for this project, we kind of divided them  into two different groups.  So we had some content requirements  about how we wanted to manage content  and some process requirements about how  we wanted to make our process sustainable as we grew.  Peer review is a very important aspect  of our community culture.  Everything that we do within OpenStack is peer reviewed.  So every bit of code is posted for review  and normally reviewed by two reviewers at least  before it gets approved and merged.  We also wanted to be able to organize the content.  So I mentioned several different kinds of release notes,  like upgrades, impacts, bug fixes,  and incompatibilities and things like that.  We wanted to be able to organize our release notes  as they were published into those different kinds of notes  so that you could go and read all about what you needed  to know to do an upgrade in one section  and not necessarily worry about new features in that section.  We also wanted to be able to change the release notes.  So we don't assume, even with the peer review process,  that we're going to catch any mistakes or not miss something  or have some other kind of error.  So we knew we wanted to be able to go back later  and change notes in old releases.  And then it's very important due to the pace that we work at  that we want to avoid merge conflicts.  So it can typically take a week or more  to get a patch reviewed and merged  even at a fast review period.  Some of them take much longer than that.  And every time you encounter a merge conflict,  you basically have to start over  because you have to rebase the patch and resubmit it.  It has to pass all of the tests again.  And then you have to get reviewers to take a look at it again.  So we wanted to design a system that just completely avoided  having to deal with merge conflicts as a contributor.  On the process side, we had some other kinds  of similar requirements.  We use two different kinds of versioning for our deliverables.  So libraries are versioned using semantic versioning,  which means that you increment the version number based  on the kinds of changes that are in the new version,  not the number of changes that are in the new version.  For those kinds of things, we don't know in advance  when you write a release note what the next version is  going to be.  So we didn't want release note authors to even  have to think about version numbers.  And we wanted them to be able to just contribute a note,  and it would automatically be applied to the correct version  when the release was created.  I mentioned our stable branch policy.  We wanted to make sure that not just that we avoided  merge conflicts on the master branch,  but that we also avoided merge conflicts  in the backported patches.  Right now, we use Garrett as a review tool,  and you can backport a patch basically clicking  a couple of buttons in a web page,  and it will create the new patch on the new branch for you  and apply it automatically.  And as soon as you start talking about merge conflicts there,  you raise the bar for backporting a bug fixed.  It makes it much harder to do that.  So we wanted to avoid merge conflicts there as well.  And then, of course, since the release team is very lazy,  we didn't want any manual processes involved  in publishing the notes at the end of a release.  So we didn't want to have the author of the release note  have to worry about versioning.  We didn't want the release team to have to sit down  and mark version numbers on a bunch of release notes  or anything like that either.  We just wanted all of that stuff to be handled automatically.  We went through three different designs  in the course of trying to figure out  how to meet all of those requirements.  So the first one was sort of the obvious thing.  We were already using Sphinx for documentation.  We could just use Sphinx for release notes.  You could just write your release notes in a document  and include it in Sphinx.  That has a couple of different problems.  So either the release note author has to know the version  number so they put it in the right place in the right file,  or someone has to go back later and apply the version number.  So that didn't meet either of those requirements for us.  And it's also messy for cherry picking changes,  particularly if you organize things in a directory structure  in some way.  When you cherry pick a change back,  you then have to edit that change  to put the note in the correct place for the older branch  because the version number system is different.  We also thought about using the get commit messages.  So just having the patch author write the release note  write in the get commit message as they were contributing  the patch.  That would work mostly, but the audience for a commit message  is very different from the audience for a release note.  So for a library, it might be similar.  But really, we wanted people to be  focused on writing commit messages for the reviewers  to understand what was going on in the patch  and not necessarily the user having to go and read the get  history to understand everything that had changed.  So we wanted them to have either a condensed version  or an expanded version that explained in more detail  for the correct audience for those release notes.  The commit messages are also largely immutable.  So once it's in that public repository,  your notes are basically set, and that's done.  We also looked at the get notes feature, which is,  I don't know a lot of the detail about this one,  but it's sort of a parallel thing to the get repository.  And you can do a little bit of extra setup  and have notes attached to commits,  but not be part of the commit.  And that was an interesting approach  until we realized that you needed to do a bunch of extra setup  with Garrett in order to make it work,  and the permissions were different,  and it wasn't really a reviewable thing.  So it fell down on our peer review criteria there.  And so we finally ended up with a system  that we implemented as Reno, where  we used data files inside of the commit  to hold the release note content.  But they're not Sphinx files.  They're just independent data files.  And then we do some integration work  to make those publishable.  Because they're inside the patch, they're reviewable.  So the reviewers can comment on them and reject them,  and you can make changes and submit a new version.  We do read the release notes data out of the get objects.  So it doesn't matter where they are on the file system.  They're all basically in one directory together.  And we look at the get history to figure out  which versions they go in, because they're part of a commit.  We can tell which version the commit is in.  And so that's what Reno does.  Reno uses those data files and basically uses the get history  as a database, reading those particular files,  and then assembling them in whatever form  you've asked to build the release notes report.  It's a command line tool.  So you start out with the Reno add command,  and you give it a slug just to sort of be able to identify  what the file is.  Frequently, people will use bug numbers,  or they'll use a short abbreviation of the feature.  We have a specification process, which  gives everything a sort of unique name.  So sometimes the notes files will  have those as part of the names as well.  And then Reno adds a unique identifier to that slug  that you give it, so that it can track the history of that file  across all of the branches and through all of the revisions  that you have.  The files are placed in a subdirectory of the project.  Like I said, they're all just placed in one directory  together.  They can be organized a little bit under that directory.  Some teams have divided them up into different kinds  of organizations.  It doesn't really make any difference for that.  Reno, because it's not reading the file system,  it doesn't really care what files the notes are actually in.  And the data file themselves are YAML files.  So this is an abbreviated version of what  one of those would look like.  The section, the keys, there are section names,  and those are predefined but configurable.  So if you have a project, and we have several projects outside  of OpenStack that are using Reno that  have a different set of features that they want to use  or sections within their release notes that they want to use,  you can change what those are.  And then Reno knows what order to apply the sections  and then which sections each note goes in.  Within the section, within the data file,  you have a list of restructured text blobs  that are just inserted in the order that they're presented.  And using restructured text lets us integrate easily  with Sphinx.  It also lets us do things like link off to bug reports  or feature descriptions or specifications  or that sort of thing.  So you get all of that nice markup.  And we use restructured text rather than HTML or something  like that, because we do actually generate PDF documentation  for some of our translated sites, especially those  behind the Great Wall in China.  It's easier for them to download APDF and share it  than it is sometimes to get to websites that  are outside of China.  So the output from Reno, we can take a look at an example  repository.  So this is a typical graph showing the history  with the beginning at the bottom and the most current commit  at the top there.  And I have one stable branch that I've created just  to be able to show what a cherry pick is going to look like.  There's two different versions are tagged.  So we have a 2.0.0 version that has four commits.  And if we assume that we're all good developers  and care about our users, we have a release note in each  of those.  We're going to have four release notes for 2.0.0.  And then version 1.0 is the first two commits there.  And that's going to include two release notes.  So if we look at the output of the report command  for the 2.0 version, we can see those four release notes.  I just included the letter names there  so that you can tell that they're from each of those commits.  And it does that. I've included them all in the new features  section just because that made it easier to fit on the slide.  But of course, you can organize those  into different sections as appropriate.  Reno figures out which notes to include  by starting at the tag that you've given it as a start point.  And it reads the git history one commit at a time  looking for release note changes in those files.  And then it assembles those into its database for that version.  And it emits them in the order that they've been presented.  And it knows that those notes apply to that version  because they appear either on the commit that is tagged  or before the commit that is tagged with the next version  number or the previous version number, I suppose.  So as it's scanning backwards in history,  it's going to find the 2.0 version first.  And then it will find the 1.0 version following that.  So Reno does the scanning using a pure Python library called  Dullich, which is an implementation of the git library.  We chose that one because it's pip installable  without any other dependencies, which  makes it easier to manage and work on release notes  on various different platforms.  So there are a bunch of libraries that actually  interface with git.  But this one, because you can just pip install it,  it means folks that are on a Mac or on a Windows system,  they don't have to figure out how to install a C library in order  to do the work with Reno.  And using Dullich has proved to be considerably more  reliable than using the porcelain output from git.  So the first version of Reno actually ran git as a command  and then read the output and tried to parse it.  And then somewhere in there, the porcelain formatting changed.  And we were no longer getting some of the data  that we needed in order to track changes to files.  So we had to rewrite it using a library instead.  So if we look at the stable branch,  if we ask it to scan the history of a branch,  instead of asking for a specific version number,  we get two versions as output.  So using git's sort of standard version nomenclature  of tacking on a number of patches after a tag,  we get a 1.0.0-1 as a version.  And that's actually an unreleased version that  represents the change that's on that branch after the tag.  And then a 1.0 version that includes the a and the b  commits.  So scanning a branch doesn't necessarily  stop at the previous release.  So you can actually ask it, give me  the entire history of a series of releases on a given branch.  And that's useful, particularly in the way  that we publish releases.  So we name these stable branches.  So Rocky is the current branch that we're in.  And they're alphabetical order.  So based on what the naming convention doesn't matter.  But they're in alphabetical order.  So we have a page for each project, for each series.  And we can say, show me all of the notes  for that series altogether.  And you get all the versions that  have been released, all the patch updates, and everything.  So I talked about backporting changes from one branch  to another.  So if we take that commit e and we copy it using cherry pick  from the master branch onto the stable branch,  we get the note that goes with the code change.  And it ends up on that stable branch.  And then if we tag a new version at the end  of that stable branch, 1.0.1, meaning  that it's a patch update with a bug fix in it,  we get the release notes for 1.0.1  will include a copy of the note that was in the 2.0 release  that's in that e commit.  And we can do that without making any changes to e.  Because it has a unique file name for the release note.  And assuming the rest of the code backports cleanly,  you can just backport and approve the patch.  Now, that doesn't always happen.  But we didn't want Reno to introduce reasons  to have backport problems.  So we're still not 100% there.  But we're doing the best we can.  And the notes for 1.0.1 will look like this.  So that includes that e release.  And then the previous commit that was there  that hadn't been tagged yet.  So we get both notes.  When we publish our notes, we publish them from one job that  happens when we tag a commit.  And so we actually have, as I mentioned,  we have a different page for each release series.  And we publish all of that together at one time.  When a user reads the release notes,  though, they don't read all of them.  They go and look at the version that they're installing.  And they want to see the release notes for that version  or that series of versions.  And so having a note copied from one branch to another like  this means that the user doesn't have to jump around and find  the release notes in different places based on where  the fix was originally applied and then backported to.  That was a real problem for us with the Wiki approach,  because people would backport the code  and then forget to go and copy the release note around.  And so after the initial release on a stable branch,  basically the release notes were not really  being updated very often.  In addition to doing a backport like that,  I mentioned that we needed to be able to change a patch,  change a release note in case there was a typo or something  like that.  So the way that works is you change the release note file  at any point in the history of the branch  after it's been added.  And as Reno scans through the history,  it finds the change first because it's  starting with the newest patch in the branch  and going backwards in history.  But it doesn't actually insert the content from the release note  until it finds the add event where that file was initially  created.  So it takes the content from the newest version of the file  and inserts it at the location in the history  of the original file.  And that lets us change the published version of the notes  for an older release after that release has been cut.  We can't obviously change the content of the package  for the 2.0 release.  So if you're looking at the release notes that  are inside the package, we can't do anything about that.  But we can change them what we actually publish online.  And so this is the output for looking at the 2.0 release  after that change has been applied.  You can see that there's no typo in the release note anymore.  And it's been changed outside of that.  It also understands that if you delete a release note file,  if for some reason the note doesn't apply at all  and you want to delete it, when it sees that delete,  it basically ignores the file.  So it doesn't add it later.  So if you have a completely incorrect release note  that somehow passed through all of the peer review,  you can just delete the whole thing and it will be ignored.  All right, I've shown you basic command line output for Reno.  It does also integrate with Sphinx.  That was one of the key benefits for using restructured text.  So there's a Sphinx directive to insert the release notes  for a given series or version into the output of whatever  your Sphinx documentation is.  This particular directive, without any extra arguments,  is going to insert the full history of the current branch.  So that's really useful on the master branch  because you basically get everything that's  happening in the development series.  On to publish the documentation for a stable branch,  you would specify that stable branch name.  Or if you just want to have a version or a couple of versions,  you can include a range or a set of version numbers  in there as well.  So it's easy to organize your published release notes  in whatever way you want.  Basically, any arguments you can pass to the report command,  you can pass here through the Sphinx directive.  And then the output, depending on what your styles are  going to look like, your CSS styles,  it's going to look something like this.  So you get version numbers inserted with headers,  and you get all of your sections with titles.  And then the release notes are inserted in order.  And we do maintain a consistent order,  so it's not necessarily predictable as you're writing  the release notes, but we're going to insert them in the order  that I think it's sorted based on those unique IDs,  so that it's consistent, so that if you publish a new version,  you don't suddenly scramble all of your release notes  into a different order.  Right, so that's the tool.  The effect that had on the community  was pretty impressive, even beyond what we had anticipated.  So we've definitely increased the quantity of release  notes that we're writing.  We are at something like 13,000 notes for those 300  different projects now, which is far more than we had,  even relative to the six projects at the beginning.  More importantly, though, we're changing the culture.  So in addition to encouraging people to write release notes,  some of the teams are actually requiring  that new features come with release notes,  new configuration options come with release notes.  Less so for bug fixes, unless there's  a significant upgrade impact or something like that.  But that's not a pervasive thing throughout the community.  It's a few of the teams that are a little more interested  and have a little more impact in the sort of thing  that a release note would document.  And I consider that a win for my own laziness,  because I don't have to deal with release notes at all  at this point.  It's basically every once in a while,  there's a bug report against the tool,  and I go work on that.  But I don't have to deal with the actual documentation.  So if you're interested in trying Reno out,  we would love to hear from you.  It is set up by default to work using the OpenStack naming  conventions and version conventions and patterns  and that sort of thing.  But it's very configurable.  I think almost everything that would be a useful behavior  has a configuration option now.  But if it's not, then we're interested in making it more  usable for more projects.  If you're not interested in Reno,  I'm still interested in having you write better release notes  and actually write release notes.  You might be interested in looking  at a couple of other tools.  TownCryor is a tool out of the Twisted community,  which I didn't know about at the time that we built Reno.  But I believe it has the problem of requiring  you to know either know in advance the version numbers  or do something at the release point  to deal with version numbers.  And so it wouldn't have met our needs anyway,  but it's still a good tool.  And then Blurb is a tool that Larry Hastings wrote  for the Python Dev community.  And it has a similar sort of problem.  And both of those teams are releasing one thing,  and so it's not really a big deal for their release managers  to take a little bit of action at the point of the release.  But for us, with the scale of the number of things  we're touching, we didn't want to have  to deal with any of those kinds of manual steps.  The documentation for Reno is on the OpenStack website,  so docs.openstack.org slash Reno.  And this sample repository that I used for this presentation  is on my GitHub account, if you want to play with that.  And I'll be posting the slides for the presentation online  as part of the conference website.  And that's all I have, so if anybody has any questions,  I'd be happy to talk.  All right.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thanks.  Sorry.  Ah, if anybody has any questions, please raise the hand.  Thank you for on this presentation.  Just a quick question, if I understand.  For now, it's only on forget, but do you have any plan  for any other VCS?  And second part of the question, would it be easy  to contribute and well for other VCS?  Is it just like callbacks plus settings, or is it like  really forking the whole stuff?  Yeah, so I had not seriously considered other version  control tools yet, but I'm interested in talking to you  about what, which one in particular are you  interested in?  Perforce.  OK.  I don't know anything about that, so it'd be up to you  to do that.  I could help you.  There is, I think, a point at which we could add some logic  that would let you switch back and forth between those two  things in the scanner in particular.  I don't know whether Perforce's data model is similar to  Git's.  It really relies on being able to walk through the history  and see the topological sort of the commits and that sort  of thing.  But yeah, I'd be happy to talk to you after about some of the  details for that.  That would be interesting.  Sure.  OK, next question, please.  Basically the same, but for Mercurial.  Basically the same answer.  OK, let's see.  Oh, over there.  It might be easier for Mercurial because I think the data  model is closer there.  I didn't quite understand how it knew when to stop going back  in history.  So you gave an example where you ran it on Rocky.  Yes.  And I went back and then got MB, which were from Master.  How did it not go back to, for example, the commits that came  before Queens or something previous?  Sure.  Yeah.  What's the termination condition, basically?  OK, so I very drastically simplified the example in the  presentation.  So the open stack, we're on the Rocky release cycle right  now.  That means we have branches all the way back through A,  potentially for some projects.  And it does a little bit of work to figure out where the  previous stable branch came from, and it stops before it  gets to that point.  So that's not really, it's very difficult to represent that  in a slide.  And the hand waving is probably indicative of that.  But yeah, it does have some smarts for doing that.  And you can turn that off.  So projects that don't have that kind of stable branch  structure could basically ignore that feature and  disable that.  But yeah.  OK, we have time for one more final question.  Thank you for the talk.  I understand that individual release nodes are committed  also to Master.  So my question is, if you make a mistake in your release  node, how do you commit without force pushing to Master?  Yeah, so we don't do any force pushing ever.  All the changes are iterative.  So you apply another patch on top.  And then Reno, as it's scanning that history, it sees a  change to a file because it's scanning backwards in history.  It'll see the change before it sees the ad.  So to hold on to the contents from the patch where you've  changed it, and when it gets to the point where you've added  the file six months ago or a year ago or whatever, then it  will insert the node at that point in the history using  the newer contents.  And it can do that because it's not reading the files that  you've commit off the file system.  It's reading them out of the Git objects in the Git history,  if that makes sense.  OK, thank you very much for the presentation.  Let's have a round of applause for Ben.  Thank you all very much for coming.  Thanks.
Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_0_131.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_132_1943.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_1944_3635.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_3636_5111.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_5112_6443.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_6444_7523.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_7524_12647.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_12648_13943.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_13944_15275.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_15276_16043.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16044_16259.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16260_16391.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16392_16511.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16512_16739.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16740_16895.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_16896_17543.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_17544_18995.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_18996_20831.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_20832_21815.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_21816_22427.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_22428_24971.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_24972_26699.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_26700_28151.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_28152_29783.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_29784_31295.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_31296_32207.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32208_32267.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32268_32315.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32316_32363.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32364_32411.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32412_32543.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32544_32759.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_32760_33023.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_33024_33155.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_33156_33527.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_33528_34427.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_34428_36107.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_36108_44603.png Doug_Hellmann_-_reno_-_A_New_Way_to_Manage_Release_Notes_tEOGJ_h0Lx0_44604_44704.png