"RSpec no longer works with ActiveRecord"
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Phippen, Sam
Sometimes an email appears in front of you in your inbox that immediately grabs your attention. For me, this was the case with a particularly scarily titled RSpec Mocks bug. In this talk, you'll hear a story of investigation and fixing of what could have been a day ruining bug for all RSpec users. You'll come away with some deeper knowledge about RSpec's mocking library. You'll learn some protips on good practise when making an open source bug report. If you've ever used with RSpec's mocking library and would like to learn more about deep Ruby metaprogramming this talk is for you.
 We're not quite starting yet. We have a minute before the timer officially kicks in. But before I do start, I just want to say thank you to everyone for coming to this talk. Instead of going and watching Aaron Patterson talk about garbage collection and his cats, that's absolutely 100% where I would be right now. As an aspect maintainer, knowing how the garbage collector relocates pages is actually super important to my day-to-day work. But I suspect most people in this room are actually application developers, and being able to actually test your code is maybe more important than knowing what shiny new under the hood features the Ruby VM has, I hope. Or maybe you just like Aspect a bunch, or you care about open source. Anyway, so, yeah, I guess they've closed the doors, which means I should probably start. So this is Aspect, no longer works with ActiveRecord. My name is Sam Phippen. Just to be absolutely 100% clear, this is not a feature announcement. Sometimes when you wake up in the morning and you're working through your email drinking that first cup of coffee, a line comes in, a subject of the email that fills your heart with such dread and terror that everything else stops, and you must immediately focus on it. For me, that was this Aspect mocks issue, number 972, allow no longer works with ActiveRecord classes. And so while the title of this talk is slightly hyperbole in that Aspect was not completely broken with ActiveRecord, this basically is one of the most core and important functional pieces of the Aspect gem. And if Aspect doesn't work with ActiveRecord classes, then, you know, there's much gnashing of teeth, maintainers get sad, everything kind of breaks and explodes, and, you know, no one is happy. Another way of saying this is that when Aspect doesn't work with Rails, my email inbox suddenly explodes. And I think nowhere in my head is there a more poignant example of this than a different Aspect issue, undefined method cache in tests for Rails 4.2.5.1, which if you zoom in had 37 comments come in overnight while I was sleeping. You see, this version of Rails got released late Pacific time while I was still living in the UK, and that just means I was in bed, minding my own business, and I wake up to discover that all of your tests aren't working. Every single person in this room who did the security upgrade had non-functional tests, and I had an inbox full of very sad email people sending pull requests, issues, debugging, trying to work out what was going on. I was just like, everyone, please calm down. It will be fine. And it may sound like I'm complaining, but I'm really, really not. I'm glad people care that Aspect works. I think it's the sign of an incredibly mature open source project that people care enough about how the Aspect gem works, that the moment there's even the slightest tremor of an issue and a very specific component that applies to them, be it mocking, stubbing, expectations, or Rails, that they will come in and file an issue. But this is a gif of me at the end of the day after work most days, and you have to remember that open source projects are mostly worked on in evenings and weekends, and maintainers have to divide their attention very, very carefully amongst their projects, or all will be laid to waste. And so support a maintainer, hug a maintainer, all of that good stuff. So anyway, let's get back to our issue and actually focus in on what we're talking about here. The Rails issue that I've mentioned affected expressions like this, where you say, allow user to receive, and then some kind of scope method, in this case we're using confirmed, but really this could be anything, and return something. Executing this simple test would raise an exception, and I don't know about you, but when I look at that, I face screaming in fear. That has to work. That is a very, very simple expression in Aspect, and active record objects are one of the core things that people want to mock on. This is really bad. Like when I saw this issue, I was like, oh, this is not going to be fun. Before we get too far into debugging, I wanted to actually do a little bit of an examination of the bug report, and think about whether it's good, bad, how it might be improved, and what sort of things we could do as contributors to actually make this issue better. If you look at the full text of the issue as submitted, it says, while updating a Rails 4.2.1 project from aspect 3.2.0 to 3.3.0, I got a lot of failing specs. I noticed the issue can be produced simply, and then the user provides this test that we just looked at. There are a few really, really great things about this bug report. The user provided a Rails version and an aspect version, and as a maintainer, that means that I can hone in very quickly on which exact things I'm going to need to begin debugging what's going on here. So if you're filing issues on projects, if you're having a bug report with an actual problem in an open source project, these are really, really great things to provide. In terms of things that aren't really great, we don't have much in the way of steps to reproduce. We only have a failing test. We don't have any of the production code. We don't have any of the Rails configuration. We don't have any of the other surrounding stuff that that application is going to be providing along with that test to make sure that it's not some other gem, some other piece of tooling, something the user has done that is outside of RSpec. And we also don't have a Ruby version. In the modern times, basically if you're on Ruby, I don't know, say 2.2 plus, variances between Ruby versions don't matter that much anymore. They're mostly stable and work all the time at this point. But providing that information can be really, really helpful, especially if you're on an alternative interpreter like JRuby, or just if you think something really weird is going on. So because we're missing a Ruby version here, we might have a slightly harder time debugging. Now, for this particular issue, it's not the case, but I do think that is a really helpful thing to provide if you do have the time and ability to do so. In terms of things that could absolutely be improved with this issue, I don't have a back trace. I don't know what the exception the user is seeing is. I don't know where in RSpec the exception comes from. I really don't know what's going on. As a maintainer, I'm going to have to go into this, like, thing and do a bunch of investigation before I'm able to actually work out where my bug is coming from. And whilst I applauded this issue for providing Rails and RSpec versions, we don't have extended dependency information. I don't know all the other gems. I don't know what was in the lock file. I can't really begin to know whether or not this is an RSpec bug, a Rails bug, or something in a gem, which is monkey patching one of those things, or monkey patching user objects. These days, I think a really, really good standard to aim for is providing a full application, a full reproduction case that the maintainer can clone and immediately execute to be able to begin debugging this issue. So basically what that would mean is instead of filing an issue and providing a single spec, you file an issue and provide a link to a Git repository that I can clone, run bundle exact RSpec, and see exactly what is going on. That will provide me a gem file, gem file.loc, all of your tests, all of your code, and everything I need to see in order to begin to understand what is going on. One of the things that's really great about RSpec is it's a multi-maintainer project. And so what happened immediately after this user filed this issue is another RSpec maintainer, Myron, comes in and is like, thanks for filing this issue. Please, could you provide us a complete back trace in order so that we can begin debugging this? And then the thing that happens next is kind of like one of those amazing beautiful platonic ideal open source things where a different user to the one who initially files the issue comes in and provides all of that information. And like this is how open source is literally supposed to work. Multiple people are experiencing the same bug and they're like literally collaborating on the issue. And this is like an amazing thing that we can only do because our code is open and our tools are collaborative. And so like if you are using an open source framework and you're encountering an issue in it and you go to the issue tracker and you see that the issue is immediately at the top of the list because it's a new bug, usually they're new, you can collaborate, right? If you can provide more information than the person who initially did and that information is helpful, that's really, really great. And no maintainer is ever going to tell you off for providing extra data points. It's a really helpful thing and I super want to encourage you all to do that. But we're still missing something from this issue. We still don't quite have enough information to be able to begin debugging it and really that's the ability to reproduce this issue in an automated, repeatable, easy to use fashion which is basically what you need to debug any real issue in open source these days. So a third aspect maintainer comes in and provides us with exactly that. Here we have complete steps to reproduce Rails new, what to put in the gem file, which models to generate and how to write a test. And so let's actually work out what this issue is. Let's actually debug this issue together live on stage and work out what's going on. So I'm going to go ahead and new up a Rails application at version 4.2.0. The reason for that is that this issue was filed against that exact version of Rails and we're now in the future and Rails 5.1 exists and so I need to be able to go back to that older version of Rails in order to do so. Once our application is created, we can then begin getting ready to debug this. The first thing that we're going to need to do is put our spec actually into our Rails application so that we can begin testing our bug. And here what you'll notice I've done is actually specified path dependencies to the aspect version instead of like the actual official gem versions. The reason that I've done this is that I have all of the aspect repositories locally checked out on my computer so that I can do debugging, insert binding.pry, insert print statements, do everything I'm going to need to do to actually debug this. This also makes switching versions really, really easy. I can check out a specific git tag and get the exact version that I need in order to do so. You might also notice I've specified rake 10 here. That is because the current version of rake is rake 12 and the older versions of rspec aren't compatible with rake 12, but that's just sort of a side note. So now that we've updated our gem file, let's go ahead and bundle update and bundle install so that we have everything ready to go. And then we'll actually begin debugging this issue and making sure everything's good. And to do that, we'll basically do this very standard Rails generate rspec install. And that will take a while because bundler is slow. Sorry, is Andre in the room? I'm sorry, Andre. I love you. And then we'll go ahead and generate a user model which is beginning to get us towards the ability to reproduce our bug. And once we've generated that user model, we're having some video troubles. I had to rebuild all these videos like half an hour ago because they were a bit blurry, so please bear with me. So that's going to go ahead and generate our user model. And then we'll migrate our database because you have to do that, otherwise you won't be able to run your tests. So because we didn't rspec install on our Rails application, it's automatically going to have generated a spec model's user spec for us, which we can then pop open and actually replace the default generated test with the one that's going to reproduce our bug. Now, because we didn't actually create a scope method on this user object, I'm going to go ahead and stub new instead of stubbing a specific scope method, which should actually allow us to reproduce this bug exactly as we expected. If I do that and run my tests, you should be able to see that this test passes. And the reason for that is that we've checked out aspect of version 3.2.0 to confirm that this did actually ever work. The user has reported that this is failing, and we want to make sure that it did work on the previous version as they are claiming. So what I'm going to do here is go ahead and change into my rspec development tooling repository, which is a sort of meta repository used by rspec maintainers to make working with various versions of rspec more easy. What I did there is run a command rake get checkout, version 3.3.0, moving us from 3.2.0 to 3.3.0, then I changed back into my Rails application and ran my tests. And you can see that they are now failing. So we know for a fact somewhere between 3.2.0 and 3.3.0, our tests broke. Reproduction case, check, and we have a good commit range. So at this point, we have a commit range, we have a passing and failing test, and that is sufficient, I think, for most people to start doing debugging. Because this issue is small and probably is resulting from a small change, it wouldn't be that difficult to just go ahead and stick a debugger in and work out what's going on. But we can do a little bit better. The year is 2017 and we have great tools for debugging systems now. And for when a regression gets introduced like this, something used to work, and now it doesn't, there's one tool in particular that is better than all the others for working out what's going on, and that's Git by sect. For those of you who aren't familiar, Git by sect is a tool where you tell Git where the thing was working, where the thing was not working, and Git will move you in exactly hard of the commit range, helping you find where the problem occurs. The important commands that you need to know, a Git by sect start, which basically tells Git you're going to do this, Git by sect good, which labels a particular commit as functional, and Git by sect bad, which labels a particular commit as non-functional. And by using these commands, Git will interactively walk you through a series of commits and help you debug and find the correct commit in order to identify where the issue came from. Unfortunately, we are slightly complicated in our ability to find the specific commit, because there are potentially two repositories at play here, RSpec Rails and RSpec Mox. For those of you who aren't used to working with the internals of RSpec, RSpec is broken down into multiple different repositories to allow us to ship features independently from each other. RSpec Rails manages all of RSpec's integration with the Rails gems, and RSpec Mox is the gem that provides these mocking methods, allow, receive, and so on. And at the moment, we don't know where in either of these two gems it could be. Now, I sort of take the lead on maintaining RSpec Rails, so I tend to assume that bugs exist in that repository first. So, let's actually bisect across RSpec Rails and take a look at what's going on here. So here I've moved into the RSpec Rails repository, Git by sect starts, label version 3.3.0 is bad, check out 3.2.0, label it as good, and now we can begin debugging. You can see here Git actually moved us to a specific revision, and then we go back to our Rails application, we type bundle exec RSpec, and we explode. The reason for this explosion is that Bundler can't actually resolve RSpec core equal to 3.3.0 with RSpec Rails equal to 3.3.0 pre. So far, so cryptic. Let me explain what's going on here. All of the RSpec gems are released at the same version numbers so that we can maintain compatibility. While RSpec provides a stable and semantically versioned public API, our internal APIs don't have to follow that same compatibility guarantee. And RSpec individual RSpec gems are allowed to call each other's private APIs where it makes sense, because after all these are sort of one combined project. So if I have RSpec Rails checked out to 3.2.0, I must also have all the other RSpec gems checked out to a 3.2x version, and in this case, they're all checked out to 3.2.0. The same is true for version 3.3.0, they must all also be checked out to that same version. But what's happened when we started the bisection is that the version tag for RSpec Rails has changed to 3.3.0 pre. This version tag doesn't represent a single commit of RSpec, but it's instead the version number that we use in the master branch of the repository while we're developing the next minor version of RSpec. But earlier, we checked out all of the RSpec gems to version 3.3.0, and 3.3.0 pre is not equal to 3.3.0 because trust me, and so this won't bundle. There is a very, very simple solution to this, which is this command, git check out head carrot. What the carrot character says is git, please give me the revision that's immediately before the one I'm specifying, and head is the revision we're currently checked out to. So what I'm going to go ahead and do is change to every single one of my RSpec repositories and type git check out head carrot in what is a furious bunch of manual typing in order to get them all onto the 3.3.0 pre versions of the RSpec gem so that we can bundle our application. Once I've done all of this, I can go back into my Rails application, and now if I run my tests, they will actually run because the bundle is now compatible on those RSpec versions that are checked out. In this case, my tests are failing, so I go back into the RSpec Rails git repository, git by sec bad, go back to my Rails application, run the tests, they fail, go back to Rails, git by sec bad, and this happens basically in a loop, marking revisions as bad every single time because the tests are going to fail for a while. Coffee. This is what open source is like. This is my life. So anyway, I think we're nearly done. Zero revisions left to test after this one, so we run our tests one more time, they fail, we go back to RSpec Rails, we try to run the test but we can't because we moved and I'm bad at typing, we mark as bad, we run our tests. Really, this is, you too could be a maintainer who gets to do this. Okay, and git has given us the revision where we started failing, and this says something really interesting, it says bunk to 3.3.0 pre. Well, that's kind of curious because that would be a commit immediately after we released a version of RSpec, so here we look at the git log and we see indeed the thing immediately before is releasing 3.2, and we can see here that our patch has absolutely nothing to do with active record scope objects and mocking and any of that stuff, which means our bisection failed. Oh, no. You know what this means? Oh, yeah. Oh, yeah. Time to do open source people. So what we're going to do is run our tests in our Rails app, make sure they're failing. Here I've reset everything to the 3.3.0 versions of RSpec, and then go back into my RSpec mox repository this time and reset bisection because I was doing some earlier, check out the 3.2.0 release, market is bad, sorry, 3.3.0 release, market is bad, check out 3.2.0, market is good, and now we're going to do the same thing, changing back and forth between our Rails application and running our tests. This will be accelerated because I like my time, so this time our tests passed, we type get bisect good, run our tests, and this time we get a blow up. We don't actually get output from the test suite. This is interesting, right? It says cannot load such file, RSpec support object inspector. Here's what's going on. Bisecting old code across multiple repositories is a dangerous and scary game. Because we've been actively developing in all of these repositories, classes have been appearing and disappearing as that development has happened. If we go into the repository that has that file object inspector and take a look at what's going on, we can quickly identify the issue. Here we're looking at RSpec support, which is an internal RSpec library to help us develop code. You can see here there's a commit, which literally deletes this object, which my tests depend on during bisection because, again, this is active development. What I'm going to do is grab the commit before the object was deleted, check it out, and now go back into my Rails application and here we can begin bisection again. Here, where the tests fail, I'm going to type get bisect bad, where the tests pass, I'm going to type get bisect good, and so on and so on, like, through all of this process. Basically, what will happen is the same thing, git will identify a specific revision where the project started breaking, and once it's done that, we can actually go ahead and understand what's going on. I'm going to go ahead and skip like a bunch of bisection here, and sort of see, good, bad, good, bad, and eventually the commit gets popped out at the end. What we go ahead and do is grab that commit number and head on over to GitHub and see what's going on, and that can't be seen at all because the contrast on these projectors isn't the best. Can anyone read that? No. All right. Basically, what's happened here is a patch has been introduced that causes our spec to call, do a callback every time you try and verify methods on an expectation on all of the subclasses of this object. As we sort of debug through various lines of GitHub, we find these methods which do the things, we search for them, we discover they don't exist, we search for different ones, we discover they do, and eventually we're led back to this piece of code, the important line of which is this one. If your object responds to a method called define methods, which is defined on Active Record Base and everything that inherits from Active Record Base implements, we call that before attempting to do any R-spec work on it. The reason we do that is that when you have an Active Record Object for the first time before it's ever spoken to a database, all of the methods for all of the attributes don't exist, right? If you have a database called name and you try and call the method name, that method isn't defined yet, and it's the method called define methods which actually define attribute methods which actually does that. I can show you this by going back into my terminal and firing up a Rails console and trying to call it. If I call Active Record Base define attribute methods, it's going to explode. And the reason for that is that Active Record Base is abstract. It doesn't represent any specific table, and so there are no attribute methods to be defined. It doesn't have any attributes. And this is identical to the blow-up that the user who provided us with a backtrace reported in their issue. So we've done it. We've found the cause of the problem. It's that when we invoke this method, R-spec explodes, and this method is being called because R-spec is calling this method on all subclasses that respond to it. The problem here is that Active Record Base is a subclass, but it's not valid to call this method on Active Record Base. We only want to call it for the subclasses and not Active Record Base itself. We solve this by implementing a check that we're not going to invoke this on the thing that explodes. And also, because R-spec is a testing framework, we add a bunch of tests that actually verify this behavior exists and works. There's a little bit of aftermath to this issue. Whilst this solution works for Active Record, the generic idea of being able to do this kind of crazy callback chain isn't the best. And so a different patch gets applied to R-spec mocks, which is more generic and doesn't only work with Rails. Well, we like to fix the immediate issue. It's also useful to come up with generic solutions if you have the time. So let's do a quick retrospective on this issue. How did we do? How was this? Well, it wasn't great that we shipped a version of R-spec Rails that was critically broken. I think a few people need R-spec to work with Rails applications, just sort of a hunch. But we had a really, really great debugging strategy, right? We had a backtrace, we had version information, we had steps to reproduce, and then we did very, very carefully calibrated bisection. Bisection is a really great method for finding specific bugs and exactly where they were introduced in the system. It's also worth noting that initially we bisected the wrong repository, but that was just based on intuition. If we'd spent a few more minutes looking at the backtrace that the user provided, it would have been obvious which repository had the issue in it, but it's great because we actually eventually ended up finding the commit. Before I go too far, I think it's also worth noting that for this bug report, 99% of the time that was spent is actually in trying to find out where the bug came in. The actual fix was trivial. It took about five minutes once it was obvious what the bug was. And that's so common and so true. I think debugging is most of the time and then just implementing the fix. It's not hard. Let me give you a few pro tips. Let me summarize all of the wisdom that I've been trying to distill into this talk. There are ways that you can make your bug reports infinitely better with just a few small tweaks. For one star, the most basic level of filing a bug report is just telling a maintainer there is a bug. This thing broke. Here is some code. It's fine, but it doesn't really provide us with enough information to be able to begin to assess and triage the issue. That's where we started with this issue. And then our maintainer has got to work actually working with the community to begin to have enough information to solve this problem. A second star I can sort of award to you for providing a backtrace because backtraces are rich with information. They don't just tell us where the bug is, but if you look carefully, you can also see that there's version information baked into these lines here. They actually tell us more or less exactly what stuff you had checked out and what it was doing when the problem occurred. The next most useful thing is dependency information. If I know every piece of code that was running in the system at the time a bug occurred, I'm much more likely to be able to quickly and suddenly identify your issue. If there's a gem which is monkey patching what RSpec does or monkey patching what Rails does, I'm able to tell you it's probably an issue with that gem and probably not an issue with RSpec or Rails. The big one, the thing that I wish everybody did is providing a reproduction case that I can just clone, just type bundle install, bundle exec RSpec and see exactly what the bug is. This kind of automated reproducible bug reporting is kind of like the golden standard these days. Many maintainers won't even look at your bug report unless you do that now because it takes so much time. The Ruby ecosystem is really, really complicated these days and if we don't have this, it can be almost impossible to work out what's going on. And then like for five stars, but really this should be more like infinity stars is doing the bisection for us. If you can come not just with an application that I can clone, run a thing and see it fail, but also then spend the time to see if this is a regression to actually bisect across many, many revisions of Rails or RSpec to determine what's going on, that's gold because then you've done most of the work and that five minute fix that I talked about earlier, that can happen straight away. It saves me all the time in the world to not have to bisect your repo on your behalf. And in fact, like I have a sort of pithy quote about this from Sean Griffin, not Sean's baby, just to be clear. A few years ago, actually I think last year, I had a bug in Rails that was badly affecting RSpec and I had actually bisected down to a single commit in Rails and I was like, Sean, the bug came in here and it was just like bam, six hours later, the issue's gone. And he said to me, if everyone who filed an issue on Rails gave me an exact commit where the bug was introduced, there would be no more open bugs in Rails. Now there is a whole army of you and there aren't that many people with a Rails commit bit. So if you all want there to be no more bugs in Rails, this is a thing you can do. It's not that specialized. So that's approximately where I finish talking about bug reports. It behooves me to just say one more thing, which is you've probably noticed DigitalOcean has been all over this conference. I work there. Who enjoyed our party? See, that's just easy pandering to the audience. Here's not pandering to the audience. Very specifically, my team at DigitalOcean is hiring, not just DigitalOcean is hiring, my team at DigitalOcean is hiring. I have, I think, one open headcount for this year and four open headcount in 2018 and I want to build an amazing team filled with Rubyists who can ship great software. We work on internal tools but also we write a lot of aspect tests and if you want to see how aspect maintainer writes aspect tests, this is an open opportunity. Please come grab me afterwards. We have socks. I am wearing them. I will give socks out until there are no socks left. First come, first serve. Please don't tackle each other. So that's it. I'm done. I'm Sam Phippin everywhere on the Internet. That's my email address.
 We're not quite starting yet. We have a minute before the timer officially kicks in. But  before I do start, I just want to say thank you to everyone for coming to this talk. Instead  of going and watching Aaron Patterson talk about garbage collection and his cats, that's  absolutely 100% where I would be right now. As an aspect maintainer, knowing how the garbage  collector relocates pages is actually super important to my day-to-day work. But I suspect  most people in this room are actually application developers, and being able to actually test  your code is maybe more important than knowing what shiny new under the hood features the  Ruby VM has, I hope. Or maybe you just like Aspect a bunch, or you care about open source.  Anyway, so, yeah, I guess they've closed the doors, which means I should probably start.  So this is Aspect, no longer works with ActiveRecord. My name is Sam Phippen. Just to be absolutely  100% clear, this is not a feature announcement. Sometimes when you wake up in the morning  and you're working through your email drinking that first cup of coffee, a line comes in,  a subject of the email that fills your heart with such dread and terror that everything  else stops, and you must immediately focus on it. For me, that was this Aspect mocks  issue, number 972, allow no longer works with ActiveRecord classes. And so while the title  of this talk is slightly hyperbole in that Aspect was not completely broken with ActiveRecord,  this basically is one of the most core and important functional pieces of the Aspect  gem. And if Aspect doesn't work with ActiveRecord classes, then, you know, there's much gnashing  of teeth, maintainers get sad, everything kind of breaks and explodes, and, you know,  no one is happy. Another way of saying this is that when Aspect doesn't work with Rails,  my email inbox suddenly explodes. And I think nowhere in my head is there a more poignant  example of this than a different Aspect issue, undefined method cache in tests for Rails  4.2.5.1, which if you zoom in had 37 comments come in overnight while I was sleeping. You  see, this version of Rails got released late Pacific time while I was still living in the  UK, and that just means I was in bed, minding my own business, and I wake up to discover  that all of your tests aren't working. Every single person in this room who did the security  upgrade had non-functional tests, and I had an inbox full of very sad email people sending  pull requests, issues, debugging, trying to work out what was going on. I was just like,  everyone, please calm down. It will be fine. And it may sound like I'm complaining, but  I'm really, really not. I'm glad people care that Aspect works. I think it's the sign of  an incredibly mature open source project that people care enough about how the Aspect gem  works, that the moment there's even the slightest tremor of an issue and a very specific component  that applies to them, be it mocking, stubbing, expectations, or Rails, that they will come  in and file an issue. But this is a gif of me at the end of the day after work most days,  and you have to remember that open source projects are mostly worked on in evenings and weekends,  and maintainers have to divide their attention very, very carefully amongst their projects,  or all will be laid to waste. And so support a maintainer, hug a maintainer, all of that  good stuff. So anyway, let's get back to our issue and  actually focus in on what we're talking about here. The Rails issue that I've mentioned  affected expressions like this, where you say, allow user to receive, and then some kind  of scope method, in this case we're using confirmed, but really this could be anything,  and return something. Executing this simple test would raise an exception, and I don't  know about you, but when I look at that, I face screaming in fear. That has to work.  That is a very, very simple expression in Aspect, and active record objects are one  of the core things that people want to mock on. This is really bad. Like when I saw this  issue, I was like, oh, this is not going to be fun.  Before we get too far into debugging, I wanted to actually do a little bit of an examination  of the bug report, and think about whether it's good, bad, how it might be improved,  and what sort of things we could do as contributors to actually make this issue better. If you  look at the full text of the issue as submitted, it says, while updating a Rails 4.2.1 project  from aspect 3.2.0 to 3.3.0, I got a lot of failing specs. I noticed the issue can be  produced simply, and then the user provides this test that we just looked at.  There are a few really, really great things about this bug report. The user provided a  Rails version and an aspect version, and as a maintainer, that means that I can hone in  very quickly on which exact things I'm going to need to begin debugging what's going on  here. So if you're filing issues on projects, if you're having a bug report with an actual  problem in an open source project, these are really, really great things to provide. In  terms of things that aren't really great, we don't have much in the way of steps to  reproduce. We only have a failing test. We don't have any of the production code. We  don't have any of the Rails configuration. We don't have any of the other surrounding  stuff that that application is going to be providing along with that test to make sure  that it's not some other gem, some other piece of tooling, something the user has done that  is outside of RSpec. And we also don't have a Ruby version. In the modern times, basically  if you're on Ruby, I don't know, say 2.2 plus, variances between Ruby versions don't matter  that much anymore. They're mostly stable and work all the time at this point. But providing  that information can be really, really helpful, especially if you're on an alternative interpreter  like JRuby, or just if you think something really weird is going on. So because we're  missing a Ruby version here, we might have a slightly harder time debugging. Now, for  this particular issue, it's not the case, but I do think that is a really helpful thing  to provide if you do have the time and ability to do so. In terms of things that could absolutely  be improved with this issue, I don't have a back trace. I don't know what the exception  the user is seeing is. I don't know where in RSpec the exception comes from. I really  don't know what's going on. As a maintainer, I'm going to have to go into this, like, thing  and do a bunch of investigation before I'm able to actually work out where my bug is  coming from. And whilst I applauded this issue for providing Rails and RSpec versions, we  don't have extended dependency information. I don't know all the other gems. I don't  know what was in the lock file. I can't really begin to know whether or not this is an RSpec  bug, a Rails bug, or something in a gem, which is monkey patching one of those things, or  monkey patching user objects. These days, I think a really, really good standard to  aim for is providing a full application, a full reproduction case that the maintainer  can clone and immediately execute to be able to begin debugging this issue. So basically  what that would mean is instead of filing an issue and providing a single spec, you  file an issue and provide a link to a Git repository that I can clone, run bundle exact  RSpec, and see exactly what is going on. That will provide me a gem file, gem file.loc,  all of your tests, all of your code, and everything I need to see in order to begin to understand  what is going on. One of the things that's really great about RSpec is it's a multi-maintainer  project. And so what happened immediately after this user filed this issue is another  RSpec maintainer, Myron, comes in and is like, thanks for filing this issue. Please, could  you provide us a complete back trace in order so that we can begin debugging this? And then  the thing that happens next is kind of like one of those amazing beautiful platonic ideal  open source things where a different user to the one who initially files the issue comes  in and provides all of that information. And like this is how open source is literally  supposed to work. Multiple people are experiencing the same bug and they're like literally collaborating  on the issue. And this is like an amazing thing that we can only do because our code  is open and our tools are collaborative. And so like if you are using an open source framework  and you're encountering an issue in it and you go to the issue tracker and you see that  the issue is immediately at the top of the list because it's a new bug, usually they're  new, you can collaborate, right? If you can provide more information than the person who  initially did and that information is helpful, that's really, really great. And no maintainer  is ever going to tell you off for providing extra data points. It's a really helpful thing  and I super want to encourage you all to do that. But we're still missing something from  this issue. We still don't quite have enough information to be able to begin debugging  it and really that's the ability to reproduce this issue in an automated, repeatable, easy  to use fashion which is basically what you need to debug any real issue in open source  these days. So a third aspect maintainer comes in and provides us with exactly that. Here  we have complete steps to reproduce Rails new, what to put in the gem file, which models  to generate and how to write a test. And so let's actually work out what this issue is.  Let's actually debug this issue together live on stage and work out what's going on. So  I'm going to go ahead and new up a Rails application at version 4.2.0. The reason for that is that  this issue was filed against that exact version of Rails and we're now in the future and Rails  5.1 exists and so I need to be able to go back to that older version of Rails in order  to do so. Once our application is created, we can then begin getting ready to debug this.  The first thing that we're going to need to do is put our spec actually into our Rails  application so that we can begin testing our bug. And here what you'll notice I've  done is actually specified path dependencies to the aspect version instead of like the  actual official gem versions. The reason that I've done this is that I have all of the aspect  repositories locally checked out on my computer so that I can do debugging, insert binding.pry,  insert print statements, do everything I'm going to need to do to actually debug this.  This also makes switching versions really, really easy. I can check out a specific git  tag and get the exact version that I need in order to do so. You might also notice I've  specified rake 10 here. That is because the current version of rake is rake 12 and the  older versions of rspec aren't compatible with rake 12, but that's just sort of a side  note. So now that we've updated our gem file, let's go ahead and bundle update and bundle  install so that we have everything ready to go. And then we'll actually begin debugging  this issue and making sure everything's good. And to do that, we'll basically do this very  standard Rails generate rspec install. And that will take a while because bundler is  slow. Sorry, is Andre in the room? I'm sorry, Andre. I love you. And then we'll go ahead  and generate a user model which is beginning to get us towards the ability to reproduce  our bug. And once we've generated that user model, we're having some video troubles. I  had to rebuild all these videos like half an hour ago because they were a bit blurry,  so please bear with me. So that's going to go ahead and generate our user model. And  then we'll migrate our database because you have to do that, otherwise you won't be able  to run your tests. So because we didn't rspec install on our Rails application, it's automatically  going to have generated a spec model's user spec for us, which we can then pop open and  actually replace the default generated test with the one that's going to reproduce our  bug. Now, because we didn't actually create a scope method on this user object, I'm going  to go ahead and stub new instead of stubbing a specific scope method, which should actually  allow us to reproduce this bug exactly as we expected. If I do that and run my tests,  you should be able to see that this test passes. And the reason for that is that we've checked  out aspect of version 3.2.0 to confirm that this did actually ever work. The user has  reported that this is failing, and we want to make sure that it did work on the previous  version as they are claiming. So what I'm going to do here is go ahead and change into  my rspec development tooling repository, which is a sort of meta repository used by rspec  maintainers to make working with various versions of rspec more easy. What I did there is run  a command rake get checkout, version 3.3.0, moving us from 3.2.0 to 3.3.0, then I changed  back into my Rails application and ran my tests. And you can see that they are now failing.  So we know for a fact somewhere between 3.2.0 and 3.3.0, our tests broke. Reproduction case,  check, and we have a good commit range. So at this point, we have a commit range, we have  a passing and failing test, and that is sufficient, I think, for most people to start doing debugging.  Because this issue is small and probably is resulting from a small change, it wouldn't  be that difficult to just go ahead and stick a debugger in and work out what's going on.  But we can do a little bit better. The year is 2017 and we have great tools for debugging  systems now. And for when a regression gets introduced like this, something used to work,  and now it doesn't, there's one tool in particular that is better than all the others for working  out what's going on, and that's Git by sect. For those of you who aren't familiar, Git  by sect is a tool where you tell Git where the thing was working, where the thing was  not working, and Git will move you in exactly hard of the commit range, helping you find  where the problem occurs. The important commands that you need to know, a Git by sect start,  which basically tells Git you're going to do this, Git by sect good, which labels a  particular commit as functional, and Git by sect bad, which labels a particular commit  as non-functional. And by using these commands, Git will interactively walk you through a  series of commits and help you debug and find the correct commit in order to identify where  the issue came from. Unfortunately, we are slightly complicated in our ability to find  the specific commit, because there are potentially two repositories at play here, RSpec Rails and  RSpec Mox. For those of you who aren't used to working with the internals of RSpec, RSpec  is broken down into multiple different repositories to allow us to ship features independently  from each other. RSpec Rails manages all of RSpec's integration with the Rails gems,  and RSpec Mox is the gem that provides these mocking methods, allow, receive, and so on.  And at the moment, we don't know where in either of these two gems it could be. Now,  I sort of take the lead on maintaining RSpec Rails, so I tend to assume that bugs exist  in that repository first. So, let's actually bisect across RSpec Rails and take a look  at what's going on here. So here I've moved into the RSpec Rails repository, Git by sect  starts, label version 3.3.0 is bad, check out 3.2.0, label it as good, and now we can  begin debugging. You can see here Git actually moved us to a specific revision, and then  we go back to our Rails application, we type bundle exec RSpec, and we explode. The reason  for this explosion is that Bundler can't actually resolve RSpec core equal to 3.3.0 with RSpec  Rails equal to 3.3.0 pre. So far, so cryptic. Let me explain what's going on here. All of  the RSpec gems are released at the same version numbers so that we can maintain compatibility.  While RSpec provides a stable and semantically versioned public API, our internal APIs don't  have to follow that same compatibility guarantee. And RSpec individual RSpec gems are allowed  to call each other's private APIs where it makes sense, because after all these are sort  of one combined project. So if I have RSpec Rails checked out to 3.2.0, I must also have  all the other RSpec gems checked out to a 3.2x version, and in this case, they're all checked  out to 3.2.0. The same is true for version 3.3.0, they must all also be checked out to  that same version. But what's happened when we started the bisection is that the version  tag for RSpec Rails has changed to 3.3.0 pre. This version tag doesn't represent a single  commit of RSpec, but it's instead the version number that we use in the master branch of  the repository while we're developing the next minor version of RSpec. But earlier, we  checked out all of the RSpec gems to version 3.3.0, and 3.3.0 pre is not equal to 3.3.0  because trust me, and so this won't bundle. There is a very, very simple solution to this,  which is this command, git check out head carrot. What the carrot character says is  git, please give me the revision that's immediately before the one I'm specifying, and head is  the revision we're currently checked out to. So what I'm going to go ahead and do is change  to every single one of my RSpec repositories and type git check out head carrot in what  is a furious bunch of manual typing in order to get them all onto the 3.3.0 pre versions  of the RSpec gem so that we can bundle our application. Once I've done all of this, I  can go back into my Rails application, and now if I run my tests, they will actually  run because the bundle is now compatible on those RSpec versions that are checked out.  In this case, my tests are failing, so I go back into the RSpec Rails git repository,  git by sec bad, go back to my Rails application, run the tests, they fail, go back to Rails,  git by sec bad, and this happens basically in a loop, marking revisions as bad every  single time because the tests are going to fail for a while. Coffee. This is what open  source is like. This is my life. So anyway, I think we're nearly done. Zero revisions  left to test after this one, so we run our tests one more time, they fail, we go back  to RSpec Rails, we try to run the test but we can't because we moved and I'm bad at typing,  we mark as bad, we run our tests. Really, this is, you too could be a maintainer who  gets to do this. Okay, and git has given us the revision where we started failing, and  this says something really interesting, it says bunk to 3.3.0 pre. Well, that's kind  of curious because that would be a commit immediately after we released a version of  RSpec, so here we look at the git log and we see indeed the thing immediately before  is releasing 3.2, and we can see here that our patch has absolutely nothing to do with  active record scope objects and mocking and any of that stuff, which means our bisection  failed. Oh, no. You know what this means? Oh, yeah. Oh, yeah. Time to do open source  people. So what we're going to do is run our tests in our Rails app, make sure they're  failing. Here I've reset everything to the 3.3.0 versions of RSpec, and then go back  into my RSpec mox repository this time and reset bisection because I was doing some earlier,  check out the 3.2.0 release, market is bad, sorry, 3.3.0 release, market is bad, check  out 3.2.0, market is good, and now we're going to do the same thing, changing back and forth  between our Rails application and running our tests. This will be accelerated because  I like my time, so this time our tests passed, we type get bisect good, run our tests, and  this time we get a blow up. We don't actually get output from the test suite. This is interesting,  right? It says cannot load such file, RSpec support object inspector. Here's what's going  on. Bisecting old code across multiple repositories is a dangerous and scary game. Because we've  been actively developing in all of these repositories, classes have been appearing and disappearing  as that development has happened. If we go into the repository that has that file object  inspector and take a look at what's going on, we can quickly identify the issue. Here  we're looking at RSpec support, which is an internal RSpec library to help us develop  code. You can see here there's a commit, which literally deletes this object, which my tests  depend on during bisection because, again, this is active development. What I'm going  to do is grab the commit before the object was deleted, check it out, and now go back  into my Rails application and here we can begin bisection again. Here, where the tests  fail, I'm going to type get bisect bad, where the tests pass, I'm going to type get bisect  good, and so on and so on, like, through all of this process. Basically, what will happen  is the same thing, git will identify a specific revision where the project started breaking,  and once it's done that, we can actually go ahead and understand what's going on. I'm  going to go ahead and skip like a bunch of bisection here, and sort of see, good, bad,  good, bad, and eventually the commit gets popped out at the end. What we go ahead and  do is grab that commit number and head on over to GitHub and see what's going on, and  that can't be seen at all because the contrast on these projectors isn't the best. Can anyone  read that? No. All right. Basically, what's happened here is a patch has been introduced  that causes our spec to call, do a callback every time you try and verify methods on an  expectation on all of the subclasses of this object. As we sort of debug through various  lines of GitHub, we find these methods which do the things, we search for them, we discover  they don't exist, we search for different ones, we discover they do, and eventually  we're led back to this piece of code, the important line of which is this one. If your  object responds to a method called define methods, which is defined on Active Record  Base and everything that inherits from Active Record Base implements, we call that before  attempting to do any R-spec work on it. The reason we do that is that when you have an  Active Record Object for the first time before it's ever spoken to a database, all of the  methods for all of the attributes don't exist, right? If you have a database called name  and you try and call the method name, that method isn't defined yet, and it's the method  called define methods which actually define attribute methods which actually does that.  I can show you this by going back into my terminal and firing up a Rails console and  trying to call it. If I call Active Record Base define attribute methods, it's going  to explode. And the reason for that is that Active Record Base is abstract. It doesn't  represent any specific table, and so there are no attribute methods to be defined. It  doesn't have any attributes. And this is identical to the blow-up that the user who  provided us with a backtrace reported in their issue. So we've done it. We've found  the cause of the problem. It's that when we invoke this method, R-spec explodes, and this  method is being called because R-spec is calling this method on all subclasses that respond  to it. The problem here is that Active Record Base is a subclass, but it's not valid to  call this method on Active Record Base. We only want to call it for the subclasses and  not Active Record Base itself. We solve this by implementing a check that we're not going  to invoke this on the thing that explodes. And also, because R-spec is a testing framework,  we add a bunch of tests that actually verify this behavior exists and works. There's a  little bit of aftermath to this issue. Whilst this solution works for Active Record, the  generic idea of being able to do this kind of crazy callback chain isn't the best. And  so a different patch gets applied to R-spec mocks, which is more generic and doesn't only  work with Rails. Well, we like to fix the immediate issue. It's also useful to come  up with generic solutions if you have the time. So let's do a quick retrospective on  this issue. How did we do? How was this? Well, it wasn't great that we shipped a version  of R-spec Rails that was critically broken. I think a few people need R-spec to work with  Rails applications, just sort of a hunch. But we had a really, really great debugging  strategy, right? We had a backtrace, we had version information, we had steps to reproduce,  and then we did very, very carefully calibrated bisection. Bisection is a really great method  for finding specific bugs and exactly where they were introduced in the system. It's also  worth noting that initially we bisected the wrong repository, but that was just based  on intuition. If we'd spent a few more minutes looking at the backtrace that the user provided,  it would have been obvious which repository had the issue in it, but it's great because  we actually eventually ended up finding the commit. Before I go too far, I think it's  also worth noting that for this bug report, 99% of the time that was spent is actually  in trying to find out where the bug came in. The actual fix was trivial. It took about  five minutes once it was obvious what the bug was. And that's so common and so true.  I think debugging is most of the time and then just implementing the fix. It's not hard.  Let me give you a few pro tips. Let me summarize all of the wisdom that I've been trying to  distill into this talk. There are ways that you can make your bug reports infinitely better  with just a few small tweaks. For one star, the most basic level of filing a bug report  is just telling a maintainer there is a bug. This thing broke. Here is some code. It's  fine, but it doesn't really provide us with enough information to be able to begin to  assess and triage the issue. That's where we started with this issue. And then our maintainer  has got to work actually working with the community to begin to have enough information to solve  this problem. A second star I can sort of award to you for providing a backtrace because  backtraces are rich with information. They don't just tell us where the bug is, but if  you look carefully, you can also see that there's version information baked into these  lines here. They actually tell us more or less exactly what stuff you had checked out  and what it was doing when the problem occurred. The next most useful thing is dependency information.  If I know every piece of code that was running in the system at the time a bug occurred,  I'm much more likely to be able to quickly and suddenly identify your issue. If there's  a gem which is monkey patching what RSpec does or monkey patching what Rails does, I'm  able to tell you it's probably an issue with that gem and probably not an issue with RSpec  or Rails. The big one, the thing that I wish everybody did is providing a reproduction  case that I can just clone, just type bundle install, bundle exec RSpec and see exactly  what the bug is. This kind of automated reproducible bug reporting is kind of like the golden  standard these days. Many maintainers won't even look at your bug report unless you do  that now because it takes so much time. The Ruby ecosystem is really, really complicated  these days and if we don't have this, it can be almost impossible to work out what's going  on. And then like for five stars, but really this should be more like infinity stars is  doing the bisection for us. If you can come not just with an application that I can clone,  run a thing and see it fail, but also then spend the time to see if this is a regression  to actually bisect across many, many revisions of Rails or RSpec to determine what's going  on, that's gold because then you've done most of the work and that five minute fix  that I talked about earlier, that can happen straight away. It saves me all the time in  the world to not have to bisect your repo on your behalf. And in fact, like I have a  sort of pithy quote about this from Sean Griffin, not Sean's baby, just to be clear.  A few years ago, actually I think last year, I had a bug in Rails that was badly affecting  RSpec and I had actually bisected down to a single commit in Rails and I was like, Sean,  the bug came in here and it was just like bam, six hours later, the issue's gone. And  he said to me, if everyone who filed an issue on Rails gave me an exact commit where the  bug was introduced, there would be no more open bugs in Rails. Now there is a whole army  of you and there aren't that many people with a Rails commit bit. So if you all want  there to be no more bugs in Rails, this is a thing you can do. It's not that specialized.  So that's approximately where I finish talking about bug reports. It behooves me to just say  one more thing, which is you've probably noticed DigitalOcean has been all over this conference.  I work there. Who enjoyed our party? See, that's just easy pandering to the audience. Here's  not pandering to the audience. Very specifically, my team at DigitalOcean is hiring, not just  DigitalOcean is hiring, my team at DigitalOcean is hiring. I have, I think, one open headcount  for this year and four open headcount in 2018 and I want to build an amazing team filled  with Rubyists who can ship great software. We work on internal tools but also we write  a lot of aspect tests and if you want to see how aspect maintainer writes aspect tests,  this is an open opportunity. Please come grab me afterwards. We have socks. I am wearing  them. I will give socks out until there are no socks left. First come, first serve. Please  don't tackle each other. So that's it. I'm done. I'm Sam Phippin everywhere on the Internet.  That's my email address.
RSpec_no_longer_works_with_ActiveRecord_0_47.png RSpec_no_longer_works_with_ActiveRecord_48_3575.png RSpec_no_longer_works_with_ActiveRecord_3576_3839.png RSpec_no_longer_works_with_ActiveRecord_3840_5003.png RSpec_no_longer_works_with_ActiveRecord_5004_5711.png RSpec_no_longer_works_with_ActiveRecord_5712_5939.png RSpec_no_longer_works_with_ActiveRecord_5940_6203.png RSpec_no_longer_works_with_ActiveRecord_6204_6491.png RSpec_no_longer_works_with_ActiveRecord_6492_7787.png RSpec_no_longer_works_with_ActiveRecord_7788_8315.png RSpec_no_longer_works_with_ActiveRecord_8316_12659.png RSpec_no_longer_works_with_ActiveRecord_12660_13499.png RSpec_no_longer_works_with_ActiveRecord_13500_15059.png RSpec_no_longer_works_with_ActiveRecord_15060_15359.png RSpec_no_longer_works_with_ActiveRecord_15360_15551.png RSpec_no_longer_works_with_ActiveRecord_15552_17399.png RSpec_no_longer_works_with_ActiveRecord_17400_17963.png RSpec_no_longer_works_with_ActiveRecord_17964_18455.png RSpec_no_longer_works_with_ActiveRecord_18456_20411.png RSpec_no_longer_works_with_ActiveRecord_20412_24611.png RSpec_no_longer_works_with_ActiveRecord_24612_25691.png RSpec_no_longer_works_with_ActiveRecord_25692_28079.png RSpec_no_longer_works_with_ActiveRecord_28080_30539.png RSpec_no_longer_works_with_ActiveRecord_30540_30707.png RSpec_no_longer_works_with_ActiveRecord_30708_30947.png RSpec_no_longer_works_with_ActiveRecord_30948_31247.png RSpec_no_longer_works_with_ActiveRecord_31248_31427.png RSpec_no_longer_works_with_ActiveRecord_31428_32663.png RSpec_no_longer_works_with_ActiveRecord_32664_33131.png RSpec_no_longer_works_with_ActiveRecord_33132_34799.png RSpec_no_longer_works_with_ActiveRecord_34800_36911.png RSpec_no_longer_works_with_ActiveRecord_36912_37799.png RSpec_no_longer_works_with_ActiveRecord_37800_38303.png RSpec_no_longer_works_with_ActiveRecord_38304_38723.png RSpec_no_longer_works_with_ActiveRecord_38724_38987.png RSpec_no_longer_works_with_ActiveRecord_38988_39383.png RSpec_no_longer_works_with_ActiveRecord_39384_42683.png RSpec_no_longer_works_with_ActiveRecord_42684_43103.png RSpec_no_longer_works_with_ActiveRecord_43104_48863.png RSpec_no_longer_works_with_ActiveRecord_48864_48995.png RSpec_no_longer_works_with_ActiveRecord_48996_49018.png