GDB: A Gentle Intro
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Clark, Jason
We love Ruby’s elegance, simplicity, and flexibility. But our favorite language perches atop a world of native code. When that other world intrudes on your peaceful Ruby, GDB, the venerable GNU debugger, is the tool to turn to. We’ll examine setting up Ruby to work with GDB. We’ll learn the fundamental commands, and soon you’ll be debugging with ease. We’ll even peer deep into Ruby object internals and face down crashes, deadlocks, and bugs. Whether you’re writing a native gem, hacking the Ruby VM, or just want a glimpse of the layers below, this talk is for you!
 Music Afternoon everyone. Thank you for hanging tough until the last session before the closing keynote and coming along. My name is Jason Clark. I work as an engineer at New Relic. And that informs a little bit of what we're going to talk about today. So I'm going to start off in this talk kind of sectioned into a couple of different examples that you may be able to kind of relate to. So the first one is, has anyone here ever had a process? I've got some Ruby program that's running and it's stopped. Like it's still running, it's still there, but it's not doing what it's supposed to do. It's deadlocked in some fashion, it's hung up somewhere and you don't really know what's going on. So you look, the process is there, but it's hardly turning any CPU. There's just no detectable activity. You look at the log file and the log file's just got nothing. And imagine that you're in this scenario and just to kind of up the ante a little, you know, recently at this particular place, maybe your coworkers had been talking about adding some multi-threading support into this service. You know, it'll make it way faster if we put some threads in there. And all of a sudden you have this deep and abiding dread because you think you might have a deadlock. Your Ruby process is stuck and you have no idea why and you don't know where this is happening in the code. Well, that's the sort of situation where GDB can come to your rescue. So GDB stands for the GNU debugger. It is actually a very old tool. The first version of it was released in 1986, but it is a C-level debugger among other things. There's lots of extensions to it, but it allows you to dig into the native layer of code below where your Ruby is executing. Now, we're at RubyConf. Like, why am I talking about a C debugger? Why do we care about this? Well, the majority of us use MRI. That's reference implementation for Ruby. Matt says Ruby. And the truth is that Matt's actually prefers that we call it C-Ruby. The Ruby VM that you are executing your code on is written in C. And so GDB as a tool that has visibility to that layer will let us see what's happening underneath. We can take a peek at what the Ruby VM is actually doing for us to try to solve these sorts of problems. So this is intended as a gentle introduction. You're not expected to know any C, but I'm going to step you through how you can use GDB as a tool to understand what's happening in these processes underneath. So with our stuck process, there's a couple of conventions that we'll follow in this talk and in the output. So when there's a dollar sign at the beginning of a line, that indicates a command that we're going to issue with our terminal. And so at our terminal, we're going to say GDB minus P and then give it the process ID of the process that we care about this stuck process running on our system. Now, GDB is going to throw a fair amount of output. It's going to tell you it's attaching to things that may spit out stuff about symbols that it's loading. But then eventually it lands you at this. So when there is a parentheses GDB in this as well, that is a prompt within GDB itself where you can issue commands. So GDB has taken the process that we were running. It's paused its execution and it's now asking us, well, what do you want to do? Well, okay, the first thing that you might want to know is where is this code that's executing? I seem to be stopped. Where am I actually stopped at? So the natural thing to ask for is to ask GDB for a back trace. Now, if you do this in actual practice, the output may be quite a bit longer than what I've demonstrated here. I mean, I don't know if anyone here has ever used GDB, but it will generate some pretty big stack traces and a lot of output on the screen. So some of this is lightly formatted and truncated. But this isn't something for us to be scared of, right? This looks a little more complicated, but it's basically the same thing as this, which I'm pretty sure anybody who's programmed in Ruby for more than five minutes has seen. Like this is just a stack trace. This shows us the methods and the files that we've stepped through. So let's try and break down what GDB has shown us and see where this is similar to what we know and what we can glean out of it. First off, because the output is pretty long and because of some of how it displays, it gives us these nice line numbers because some of those lines might wrap. So those are frames that we can go through where we've stepped in our program. And each of them has a function name associated with it. Now, you can think of these a lot like our Ruby methods. These would be methods that you would find in your code that you would be able to identify and locate. And in fact, a lot of the time, if you've got the appropriate debugging symbols available, it will even tell you where in the source code that came from. So here we see something in our Unix system dependencies, but then thread.c on the third one. That's actually a file in MRI itself, and this is sitting on line 43, 42 of that file. So we could look and see exactly where this is executing. And in fact, that line is very pertinent to what we care about as well because we're suspecting some sort of deadlock, right? Something that's stopping our process. Well, MRI methods conventionally start with an RB underscore in the function name, and here we have RB underscore mutex lock. So this is starting to smell even more like a deadlock like we might have expected. So let me say create a mutex object and ask it to lock itself in Ruby. This is the C function that's actually being run by the VM on our behalf. All right, so this is a little bit of a picture of what's going on in this process, but just seeing one thread being somewhere in a lock doesn't necessarily tell us everything we need to know. So let's ask GDB what's going on elsewhere in the process. So if we say info threads, GDB will give us a list of all of the threads that are in the current process and show us the top level of where those are actually executing. So the threads are numbered and those numbers stay consistent across the lifetime of the thread. And like we saw in the back trace, it gives us the function names. And interestingly, here we can see this pthread, condwait. That was actually the top thing in the frame for that particular back trace that we were paused on. Here we see that there are two separate threads which are both at that same line of code waiting on a condition in the pthread library. Okay, so this is giving us even more information. This is smelling more and more like our theory that we've got a deadlock is true. But we'd really like to see how we got to those locations as well. And GDB provides. With the thread apply command, you can select certain threads, and in this case I'm selecting all of the threads in the process, and ask GDB to run a specific command against those. So we can take that back trace that we took originally, and that ran for just the current thread that we were on. And instead, ask GDB to show us all of the back traces for all of the threads. And if we do this, we do in fact see that thread number three has that RBMU text lock right where we had seen it before, where we were paused. And thread one is also paused at that exact same lock. So with this information in hand, we can go to our co-worker who thought that they were so smart putting multithreading into the thing, point out that they may well have introduced a deadlock, and hopefully they're smart enough to fix it. Your mileage may vary depending on who it is. And having done so, we want to get out of GDB. You type exit, and well, exit's not actually defined unfortunately, so we'll quit instead. Okay, so let's pause for a moment before we move on, and just review a little bit of what we've talked about here. So we've seen how you can use GDB and attach to a running process. You can do this to any process on your system as long as you've got the permissions to do it. You might not want to do it to something that's critical that you care about, because it's easy to make things stop and easy to make things do bad things from within GDB, but it's a lot of fun to be able to stop things that you want to inspect and figure out what they're doing. There are a number of commands that allow us to look at the state of our process to understand where our code is executing, get back traces, and see the different things that are going on in the life cycle and stack of our code. And then lastly, having figured out the answer to our question and what we needed to know, we were able to get out of GDB and get back to the rest of our lives. So that is a concrete example of how you can use GDB to debug something that's going on live on a particular process, but I found that GDB is also very useful to use in exploration. And so we're going to look at what I call a unique problem where I saw something happening in Ruby, and I didn't really understand why it was doing what it was doing. And so we could use GDB as the tool to dig into that and figure out what's happening in the VM underneath us. So the particular setup for this is the unique bang method. Now, the unique method takes an array and will return to you the list of unique elements that are in that array. The bang methods in Ruby are mutating methods, typically when there is a mirroring non-bang method available in the standard library, and so it should modify the actual array in place. So in this case, that actual array that we've got there on line one would get modified. Its members would be changed. But as we can see here in pry, it also returns us that list of unique elements. This is kind of what I would expect. It's going to hand me back the list of unique things when I call unique bang. Unfortunately, if you look at it with an array which does not have any duplication, you get nail back. And this seems very mysterious to me. Like when I first encountered it, I was sure that something weird was going on. And so we're going to set out, and we're going to go look at what's going on and just understand what's happening. So to do this, we're going to set up a little script that will duplicate this bug that we think that we found, this thing that we don't understand. When we run this, we see the output of the array that's been unique and gives us the result that we expected, and we see that our other array returns us a nil, which was kind of confusing us. So how do we start this up with gdb? Well, as you might expect, it has some facilities for letting you start an arbitrary process instead of attaching to something already running. You can do that with minus, minus args from the command line. So we give it the same thing that we would want to do to run that Ruby script. There are a variety of other ways and parameters you can use to control the exact execution if you need more fine-grained control. So like before, when gdb starts up, it's going to drop us to a prompt, and the program is paused. Now, this is not a program that has run yet, and so this is paused before anything has happened in our program. And so to get it going, we tell gdb to go and run. Like before, we get a lot of different output. There's a lot of things that it's going to tell you, you know, as threads spin up, as it finds debugging symbols. But in the midst of all of that output, we see the output to standard out, just like we expect from our program there in the middle. We see our array with 123, and we see our nil. So we know that this is actually running the code that we're interested in. It finishes that script, lets us know that that process is finished, and then drops us back to the gdb prompt. So we now have a process by which we can run the same thing. We can run this test case that we want to understand better over and over again from gdb to try to dig into it. But that's not as helpful as we'd like, right? We want to look at what's happening specifically when we're calling this method on an array. So we need to find some way to coerce gdb into putting us in the location that we want. Well, fortunately, the pieces are all out there to help you very easily bridge the gap from the Ruby code that you care about to the C functions that gdb understands for us to be able to hook into. If you go to the Ruby docs for the array class and look at the unique bang method, all of the documentation in Ruby up in the upper right corner, if you hover up there, it has a little thing that says click to toggle source. You don't even have to go clone the repos for Ruby to be able to look at this information. When you click that, a little section expands down below, and it shows you the source code for the particular Ruby method that you're looking at. And so here it is. Here we have that. And don't panic. It's C. Some of the methods you'll find will be implemented in Ruby, things in the standard library. But a lot of the core things in Ruby are implemented in C. And so some of you may not be familiar with that language. Some of you may be a little rusty on it. So let's talk about one of a couple of fundamental pieces that you need to understand to be able to look at things in gdb and get what's going on. And this is a pretty straightforward one that C functions are the key to getting to the location that you want to to debug your problems. So here is an example of one of the simplest possible C functions that you could have. Now, if you kind of squint, this looks a lot like Ruby, but there are a few significant differences. First off, we have to declare a return type. We have to tell C what sort of thing we are going to return. And if you don't return something, you have to tell it void for it to know that there is no return value. We have a name for our functions. So that's very comfortable and familiar and actually really key to us being able to work with this in gdb. Then we also have a list of parameters. And because C is a somewhat typed language, these parameters have a type. So in this case, we have to pass along in as the value. And C has no idea of an implicit return like we have in Ruby where the last line that executes is the value that will come back. So we have to explicitly tell C that we want to return. And here we're just returning 42 because it's the answer. So how does this help us with our quest to figure out what's going on with UnikBang and understand the implementation of it? Well, if we look at that code that showed up on our Ruby docs page here blown up a little bit, we'll see some of those same pieces that we just talked about with any C function. So our return value, which here they do a lot in MRI having the return value beyond the line above the name of the function. So it returns a value and it takes a uppercase value as the type. So value is the name that is used in the C code for a Ruby object. Any Ruby object of any type that gets passed around throughout MRI will be represented by one of these values. So here we have that UnikBang takes an array, it takes a value which we know is the array that we're calling this on. And then it's going to return us back a Ruby object. But the real key here, like what we need to be able to take our next step is this name. So RB underscore array UnikBang is the hook that we need to get GDB to the location where we can peek around and see what's happening in the neighborhood. As mentioned before, RB underscore is the prefix that's used in MRI code to indicate that things are a Ruby method. The ary underscore is the prefix that it follows that with for methods that are related to the array class. And this is fairly conventional. Most types in MRI that you find will have some sort of standard naming convention for the C functions that maps to the Ruby representation of when we execute this. All right, so how do we make use of this name having figured out where the code is that we care about? When we ran our args command to start GDB up with our process, it dropped us in and pauses. And in that pause, we can tell GDB that we want to break whenever the RB array UnikBang method is called. And at that point, if we then call run, then rather than executing the program all the way through and terminating, GDB will stop. Tell us that it hit a break point and give us some of that information about it. Show us a not very useful in this case snippet, which is the single line of code that it has actually paused on. And we'll take a look at how to get a better view around that in a moment. And then it provides us with a prompt. So now we are back in a paused state where we can ask the world what it looks like, what values are, and control where GDB goes next. But before we do that introspection, let's pause. Roll back and review how we've gotten here. So to be able to run a program from scratch ourselves in the context of GDB, we say GDB minus minus args, and then give it the invocation just like we would do with the command line. That will start us in a paused state so we can set break points and take other operations, and then we tell it to run, and it will go do the things that we wanted it to. C functions are one of the fundamental pieces of that language, and they're very similar to what Ruby does with methods. There isn't an object involved, but they look a lot the same, and they're the foundational way that we can kind of navigate around and set break points within GDB. Ruby docs, awesomely, provides us all the information we need to translate between the Ruby methods that we're familiar with and the C implementation that underlies that. And with that information, we can get ourselves to stop anywhere that we want to in the execution of the Ruby VM when we're running our code. All right, so we've started our program, we've gotten to the location we've seen a little bit of the code around where we're at, so let's take a deeper look at what's happening in this unique bang method and what's actually around us as we're executing. The list command, if you have everything wired up correctly in GDB, will show you the source code that's before and after the line that we are paused on in our execution. You do have to do a little bit of setup to inform GDB where the location of a copy of Ruby source code is, because the compiled binaries do not have that code inside of them. But that's pretty simple to do. So here we were on line 41.62, that curly brace right at the beginning of the function, because we asked it to stop there for us. And so that's where we're paused, and we can see a little bit of what's going to come after us that we'll examine in a moment. But before that, you know, there's this array parameter, so that's the array that we're calling this on. And we'd really like to take a look at that. Let's see what we can see about what's getting passed into this function. So GDB has a couple of methods that are around this sort of behavior and have some formatting options and things, but one of them is display. And so you can say display on any local or global variable that's in scope at the point that we're at. And we ask to display the array member, the array variable that's getting passed in. And it gives us this kind of cryptic response. It says array is equal to this very, very large number. Well, don't panic, but that's a memory location, and this is a pointer. So that is the actual location in your computer's memory where the array that's been allocated for you is residing. Now, this is probably not what you're actually wanting when you say display array. You want to see what the actual values are or see something about how Ruby represents that. But to get there, we need to take a slight diversion and talk about the other important piece of C that is good to know, but at least the general shape of to be able to use these tools, and that is the struct. So the struct is the way that C code allows you to group bits of data together to carry around in your program. So here we have a struct. It is named data, so you can think of that a little like a class name. And when we have a struct of type data, it has one integer in it, value. Now, what this means in practice is in C, a struct is literally a description of the layout of values in memory. So if we have this data struct, that means that when we make one of these structs in place at somewhere, like at this particular location, there will be an integer with the value that we've assigned sitting right at that place in memory. Structs will nest and you can combine them together. So if we have a valuable data struct, which its first member is that prior struct that we defined, and then we put a priority after it, what we see in memory is we will see the integer for the value and then the priority from that outer containing struct that layers there. Now, why do we care about this? The reason that we care is because this is how every single Ruby object that you ever deal with in your code is actually represented under the covers. There is a struct called RBasic and RBasic has a flag which indicates what type of Ruby thing this is and a pointer to a class. And so any Ruby object, any of those value things that we saw getting passed around, all that you really know when you get a value is that at that memory location, there will be the flag, which will tell you how to interpret the rest of it, and some data about the class. And then depending on the type of thing that you're looking at, the data that follows that in memory will take a particular shape. So for instance, don't get scared by all of the details, but this is what an array looks like in Ruby. It will have the flags, the class, and those come from that RBasic that it starts out with. But then it has things around the length and the capacity and then the pointers to the values. All of those things follow directly in memory right after those beginning fields that we got. So Ruby has a number of different structs that it deals with. Many of the internal types or the basic types that you deal with in Ruby have explicit structs that it implements, that represent them. So things like strings, hashes, arrays, numbers are all represented by explicitly coded structs in MRI. But all of our Ruby objects are also there. The R object struct is the struct that is every single object of your own custom classes that you instantiate anywhere in your Ruby program. So now we understand a little better when we said display array, what it's actually doing is it's giving us that memory location where we can get a flag and a class. And if we knew how to structure that memory, we could interpret everything that follows from that address to get an actual picture of what Ruby thinks is there. Now, don't get too wigged out, but this is a line of code that you can execute into GDB to tell it to actually peel that memory apart. So GDB has loaded the R array structure so it knows what fields are supposed to be there and it knows how to interpret that and give us some basic display of it. We can see in here that we've got our flags and our class, we've got the array data itself, which looks a little funny, but this is giving us the capability to look literally at what the memory is that these objects represent. And if this was something that we knew a little more about how arrays structured their data, we might be able to dig into the details and understand what's actually there. If this was a custom object or you were writing a native gem, perhaps, these might be values that you would care about and understand. Fortunately, Ruby provides us a simpler way to get at these sorts of values with a function called rb underscore p. So this is a function that's in MRI, it's available there, and it literally will do the sort of output printing that you really actually want, like the things that you get if you just print an object in Ruby itself. You can invoke a C function in GDB by just saying call, so we can call rbp and pass it our array, and it will print something, it will print out a literal representation of that array for us. And here we've hit a little bit of an odd spot. Our script had an array with one, one, two, three in it, but the output of this array where we hit our breakpoint is a string. Well, I don't know exactly what's up with that, but before we dig into this mystery and before we figure out where this weird array value is coming from, let's review what we've done in looking around the neighborhood and looking at the values that we've got. So if you want to see the source code around whatever location you were paused at, you can tell GDB to list, and it has some parameters if you want to list more or less of the information that's there. To look at a local variable, you can call display on it, and if it's just something like a C string or a C number, it will print that out neatly. You might need to take a little more pain. Thanks, Java. Good night. All right. What was I doing? We were looking at an array, so you can display things. There are a lot of formatting options with display that you can dig into to look at the various values that you might find around. We've taken a brief look at C structures, one of the fundamentals of how C programs are built and the way that data gets passed around for every object that you ever interact with in your Ruby program. And we saw the RB print, which gives us some debug printing to be able to have a little bit of our Ruby niceties, even though we're down at the C level. But we kind of got off on a weird path here. Like, we're seeing some values that we weren't expecting. We're seeing this array, this array with one string in it. It would be really nice to know where this is coming from and understand why we have this value. And fortunately, MRI yet again provides us with what we need through the RB backtrace function. So this function, unlike asking GDB for a backtrace, which is the C level backtrace of where we're at, it will give us what Ruby thinks the backtrace is. And this gets us back to territory that's a lot more familiar. And this answers our question pretty quickly. If we look at the files that this is going through, it's Ruby gems. So as of any modern version of Ruby that you will be dealing with, Ruby gems is deeply integrated into Ruby itself. And it assumes that you're going to need some gems and that you're going to do things. And so it does some preparation when you start running your script. And that's what we're hitting. Ruby gems calls unique bang. And that's the breakpoint that we hit was seeing not our script running, but Ruby gems calling unique bang as it started up before it gets anywhere near executing our script. All right. So knowing that, Ruby gems has also provided a way out so we can say minus, minus disabled gems when we run this file. And sure enough, when we do that, we hit the same breakpoint, but the array actually has the values that we expect. It's actually running our script so we can start looking at the implementation and the behavior there. So let's look at how we can move through this function. Just setting breakpoints everywhere is great for a start, but, you know, we need to be able to get around. When we listed, we started out on the first line of this function on this, on this brace. The next two lines that are after it are variable declarations, and so nothing really happens on those lines. And so when we ask GDB to go next, much like you would in any of your Ruby debuggers, it will move us to the next line that has some actual activity going on it. And it's this RB array modified check. If you say next, like in most other debuggers, next will continue execution at the next line in the current function that you're in or return you back out if you've gotten to the end of the function. Step, on the other hand, will drill in if there's anywhere that we can go. So in this case, we were sitting on a function call, we were getting ready to call RB Modify Check. And if we say step, GDB gives us the indication that it's moving into this other function, tells us where it is, and then helpfully gives us the one line that we're actually resting on at this point in time. If we do a listing, we can see that, yes, indeed, our context has completely changed. We're over in this totally different function than where we were before. If we want to get out of that, we can either just next through, or you can say finish, and that will finish the currently executing function and pop you back up a level to where you were before. All right, so now that we've got the basics of motion, the basics of being able to step through this code, let's look at what the unique bang method is actually doing and see if we can get our heads around these return values we weren't exactly expecting. So almost right off the bat, after we get in there, we see this line. So we look at the length of the array, and if it's less than or equal to one, we return q nil. q nil is C Ruby's object for representing nil. That is the nil that you interact with all the time in your Ruby applications. That's the nil that we're all trying to escape with the dot question mark, ampersand, question mark thing, try or whatever that's coming in. Anytime you deal with a nil, that's it right there. But this gives us our first hint that some of our intuition about things was probably wrong. I mean, if the implementation here is this explicit about returning nil in this case, they probably really intended for us to get those nils back that we were kind of surprised to be seeing. All right, so let's keep moving on and see a little bit more of what's around here. If we go down, there's a mode where a block can be given, but if we don't, we call into this function that says, array make a hash out of this, and this creates us a hash based on the values that were in the array. The keys of that hash will be the values, and so this is how unique and unique bang figure out what the unique set of elements are in the array for us to care about. Now, this is actually a really interesting point because to me, I often will call the bang methods for things because I figured they're more efficient in terms of memory. They're not going to allocate new objects because it's going to modify my things in place. But here, in fact, we're allocating an entirely new hash which is then used just to determine whether we've actually changed things or not later on. So unique bang might not be getting you the savings if you're thinking that it's avoiding memory allocations by calling a bang method. Having gotten that hash back, we do the same trick that it was doing earlier of looking at the length of the array, and here it says if the length of the array is the same as the size of the hash, we return nil. And so this is the behavior that we saw up in our Ruby code, that if there are all unique items, if the array is already unique, it's just going to return you a nil. Well, so apparently we didn't find a Ruby bug, but we've had an interesting time being able to dig into it. We'll let our program continue. If you type continue or see, there's a lot of shortcuts as well. Things will go on and execute from there. And in fact, as we go back and we're reviewing at the end of things, apparently if we had actually read the documentation, we might have understood that this method was doing exactly what it was supposed to do. But I feel like the tools are important. Knowing that you can dig into it is worthwhile. So in this section, we've looked at moving around. We've looked at navigating. There's a number of other things that GDB provides, but the documentation for it is really good. So dig into it. Look at your code executing live and see what you can see of the underlayers of your Ruby. To close out with, I want to show you just a few quick snippets of some of the other sillier or crazier things that you can do once you're down at this level. So one part to me is that the output in GDB can occasionally be very confusing. And especially if you had an application that is also writing meaningful things to standard out, those things get interleaved with the debugging output in ways that are kind of difficult to follow. Well, we can actually get around that. And this doesn't have anything to do, in fact, with Ruby. This is just us interacting with C itself. So we can tell C to close file descriptors one and two. So that is standard out and standard error. We can then open those. We can open to a file. And those file descriptors will be reused. And one and two will then be pointed back to this GDB.log file that we have. So any of the output that our program was going to generate, instead of going onto our screen while we're debugging in our terminal, will instead get sent over to a file and get saved. This has been really useful for me in a couple of cases where having that output while I was actually trying to interactively debug was very distracting and confusing. This is probably one of the more complicated examples, probably the most complicated in terms of commands that we will execute here. But it's also one of the most fun. So we can say call and call of C function. Here we're calling the RBP function, which will give us debug printing. And what we're calling that print on is Ruby's own eval method. This eval method takes a string of Ruby. So if there is any piece of Ruby that you want to execute while you're in the course of debugging, you can eval it. And in this case, what I'm doing is I'm asking it to give me a backtrace print of each of the threads. So RB backtrace, which we looked at earlier, is really convenient. But it only shows you the backtrace for the current thread. And there's no facility for telling it to show us all of the threads. So if we want to see the Ruby backtraces of all of our threads, well, we can ask Ruby to go do it itself. Now, be aware, as with many of these things, there is some peril. Depending on your operating system, if there happens to be a garbage collection run happening at the time that your program is being paused, this may not work properly. But, you know, if you're already in the spot that either you're just experimenting or your process is dead already, it may be a fun thing to do. GDB also provides us with a lot of automation facilities and other options for controlling its behavior. You can write a script of GDB commands. So here we attach to a specific process ID, run those backtrace commands that we've looked at earlier and tell it, okay, I'm done, go stop. And we can run this interactively from our command line saying batch and minus x. And this will go and, you know, run a set script for us. In fact, this was so useful that in the Ruby agent, we made a script called nrdbug. And this uses, among other things, it runs a GDB script to extract those thread backtraces. And we've used this in debugging deadlocking cases on users' machines. And if you install this gem, you can run nrdbug and point it to any of your processes and see a whole bunch of different output that it'll provide for you about what's happening in that process. This same sort of scripting also allows you to customize your environment. If you make a.rbinit file in your home directory, you can put commands in there that will get executed whenever GDB starts up. And I can finally rest easy being able to type exit instead of quit to get out of GDB. All right. Let's recap briefly. We started off and took a look at how GDB is a very useful tool, in particular for debugging deadlock situations in places where your Ruby program might be frozen doing something below the Ruby layer. We looked at a unique sort of problem where our understanding of what the Ruby VM was doing for us didn't match up with reality. But we were able to look around, we were able to move through the code and figure out what it was doing for us and realize that finding a Ruby bug isn't always as easy as you think it might be. Lastly, we closed out with some crazy tricks that you can play. And really, the sky's the limit. We've talked about this in the context of Ruby, but most of the applications that you are interacting with, libraries that are on your system at some point, interact with this native layer of code. And GDB can let you hook into that and see what your computer is really doing at a lower level. Well, I hope that you found some cool tidbits in this today. And I hope that maybe you'll go and debug a process somewhere on your computer very soon. Thank you. So the question is whether I found out why UnikBang returns nil. And no, I have not gotten to the bottom of what the rationale of it is, but it is clearly the specified behavior of that method. And it did bite me at one point that it was doing that, as it has probably many other people in the room. The question was, do you need to recompile Ruby with a dash g flag? And yes, that is something that I did gloss over a little bit. There may be a need to compile your Ruby with debugging symbols or at a lower level of optimization. The exact steps for doing that varies depending on exactly how you're installing and compiling your Ruby. So go look that up online. One plug, if you do just want to play with it, the official Docker images that are made for Ruby on Docker Hub actually have this all set up correctly and neatly so you can use those. You could just spin up a little Docker container as a separate environment, and it has all of the things that you need if you're locally compiled. Ruby doesn't have the symbols that you're looking for. The question was whether I've run into situations where re-entrancy has been an issue in debugging, and I don't think that I have. There certainly are sharp edges to using a tool like this, and it strikes me as very possible that you could get effects like that, which would be difficult to work around, but I have not personally run into that. The question was, can you disable the garbage collector or check whether it's running before you do some of those techniques that were hazardous, and the answer is yes. In fact, if you were specifically setting up a test case for doing this sort of debugging, unless what you're wanting to look at is GC-related, it might be worthwhile to pause that. There are Ruby-level methods that you can just say, GC.disable, and it'll turn itself off. That is certainly a good way in debugging that you could handle it. All right, unless there's time for one more, but otherwise, thank you for your attention. Thank you.
 Music  Afternoon everyone.  Thank you for hanging tough until the last session before the closing keynote and coming along.  My name is Jason Clark. I work as an engineer at New Relic.  And that informs a little bit of what we're going to talk about today.  So I'm going to start off in this talk kind of sectioned into a couple of different examples that you may be able to kind of relate to.  So the first one is, has anyone here ever had a process?  I've got some Ruby program that's running and it's stopped.  Like it's still running, it's still there, but it's not doing what it's supposed to do.  It's deadlocked in some fashion, it's hung up somewhere and you don't really know what's going on.  So you look, the process is there, but it's hardly turning any CPU.  There's just no detectable activity.  You look at the log file and the log file's just got nothing.  And imagine that you're in this scenario and just to kind of up the ante a little,  you know, recently at this particular place, maybe your coworkers had been talking about adding some multi-threading support into this service.  You know, it'll make it way faster if we put some threads in there.  And all of a sudden you have this deep and abiding dread because you think you might have a deadlock.  Your Ruby process is stuck and you have no idea why and you don't know where this is happening in the code.  Well, that's the sort of situation where GDB can come to your rescue.  So GDB stands for the GNU debugger.  It is actually a very old tool.  The first version of it was released in 1986, but it is a C-level debugger among other things.  There's lots of extensions to it, but it allows you to dig into the native layer of code below where your Ruby is executing.  Now, we're at RubyConf.  Like, why am I talking about a C debugger?  Why do we care about this?  Well, the majority of us use MRI.  That's reference implementation for Ruby.  Matt says Ruby.  And the truth is that Matt's actually prefers that we call it C-Ruby.  The Ruby VM that you are executing your code on is written in C.  And so GDB as a tool that has visibility to that layer will let us see what's happening underneath.  We can take a peek at what the Ruby VM is actually doing for us to try to solve these sorts of problems.  So this is intended as a gentle introduction.  You're not expected to know any C, but I'm going to step you through how you can use GDB as a tool to understand what's happening in these processes underneath.  So with our stuck process, there's a couple of conventions that we'll follow in this talk and in the output.  So when there's a dollar sign at the beginning of a line, that indicates a command that we're going to issue with our terminal.  And so at our terminal, we're going to say GDB minus P and then give it the process ID of the process that we care about this stuck process running on our system.  Now, GDB is going to throw a fair amount of output.  It's going to tell you it's attaching to things that may spit out stuff about symbols that it's loading.  But then eventually it lands you at this.  So when there is a parentheses GDB in this as well, that is a prompt within GDB itself where you can issue commands.  So GDB has taken the process that we were running.  It's paused its execution and it's now asking us, well, what do you want to do?  Well, okay, the first thing that you might want to know is where is this code that's executing?  I seem to be stopped. Where am I actually stopped at?  So the natural thing to ask for is to ask GDB for a back trace.  Now, if you do this in actual practice, the output may be quite a bit longer than what I've demonstrated here.  I mean, I don't know if anyone here has ever used GDB, but it will generate some pretty big stack traces and a lot of output on the screen.  So some of this is lightly formatted and truncated.  But this isn't something for us to be scared of, right?  This looks a little more complicated, but it's basically the same thing as this,  which I'm pretty sure anybody who's programmed in Ruby for more than five minutes has seen.  Like this is just a stack trace. This shows us the methods and the files that we've stepped through.  So let's try and break down what GDB has shown us and see where this is similar to what we know and what we can glean out of it.  First off, because the output is pretty long and because of some of how it displays,  it gives us these nice line numbers because some of those lines might wrap.  So those are frames that we can go through where we've stepped in our program.  And each of them has a function name associated with it.  Now, you can think of these a lot like our Ruby methods.  These would be methods that you would find in your code that you would be able to identify and locate.  And in fact, a lot of the time, if you've got the appropriate debugging symbols available,  it will even tell you where in the source code that came from.  So here we see something in our Unix system dependencies, but then thread.c on the third one.  That's actually a file in MRI itself, and this is sitting on line 43, 42 of that file.  So we could look and see exactly where this is executing.  And in fact, that line is very pertinent to what we care about as well  because we're suspecting some sort of deadlock, right?  Something that's stopping our process.  Well, MRI methods conventionally start with an RB underscore in the function name,  and here we have RB underscore mutex lock.  So this is starting to smell even more like a deadlock like we might have expected.  So let me say create a mutex object and ask it to lock itself in Ruby.  This is the C function that's actually being run by the VM on our behalf.  All right, so this is a little bit of a picture of what's going on in this process,  but just seeing one thread being somewhere in a lock doesn't necessarily tell us everything we need to know.  So let's ask GDB what's going on elsewhere in the process.  So if we say info threads, GDB will give us a list of all of the threads that are in the current process  and show us the top level of where those are actually executing.  So the threads are numbered and those numbers stay consistent across the lifetime of the thread.  And like we saw in the back trace, it gives us the function names.  And interestingly, here we can see this pthread, condwait.  That was actually the top thing in the frame for that particular back trace that we were paused on.  Here we see that there are two separate threads which are both at that same line of code waiting on a condition in the pthread library.  Okay, so this is giving us even more information.  This is smelling more and more like our theory that we've got a deadlock is true.  But we'd really like to see how we got to those locations as well.  And GDB provides.  With the thread apply command, you can select certain threads,  and in this case I'm selecting all of the threads in the process, and ask GDB to run a specific command against those.  So we can take that back trace that we took originally, and that ran for just the current thread that we were on.  And instead, ask GDB to show us all of the back traces for all of the threads.  And if we do this, we do in fact see that thread number three has that RBMU text lock right where we had seen it before, where we were paused.  And thread one is also paused at that exact same lock.  So with this information in hand, we can go to our co-worker who thought that they were so smart putting multithreading into the thing,  point out that they may well have introduced a deadlock, and hopefully they're smart enough to fix it.  Your mileage may vary depending on who it is.  And having done so, we want to get out of GDB.  You type exit, and well, exit's not actually defined unfortunately, so we'll quit instead.  Okay, so let's pause for a moment before we move on, and just review a little bit of what we've talked about here.  So we've seen how you can use GDB and attach to a running process.  You can do this to any process on your system as long as you've got the permissions to do it.  You might not want to do it to something that's critical that you care about, because it's easy to make things stop and easy to make things do bad things from within GDB,  but it's a lot of fun to be able to stop things that you want to inspect and figure out what they're doing.  There are a number of commands that allow us to look at the state of our process to understand where our code is executing,  get back traces, and see the different things that are going on in the life cycle and stack of our code.  And then lastly, having figured out the answer to our question and what we needed to know,  we were able to get out of GDB and get back to the rest of our lives.  So that is a concrete example of how you can use GDB to debug something that's going on live on a particular process,  but I found that GDB is also very useful to use in exploration.  And so we're going to look at what I call a unique problem where I saw something happening in Ruby,  and I didn't really understand why it was doing what it was doing.  And so we could use GDB as the tool to dig into that and figure out what's happening in the VM underneath us.  So the particular setup for this is the unique bang method.  Now, the unique method takes an array and will return to you the list of unique elements that are in that array.  The bang methods in Ruby are mutating methods, typically when there is a mirroring non-bang method  available in the standard library, and so it should modify the actual array in place.  So in this case, that actual array that we've got there on line one would get modified.  Its members would be changed.  But as we can see here in pry, it also returns us that list of unique elements.  This is kind of what I would expect. It's going to hand me back the list of unique things when I call unique bang.  Unfortunately, if you look at it with an array which does not have any duplication, you get nail back.  And this seems very mysterious to me.  Like when I first encountered it, I was sure that something weird was going on.  And so we're going to set out, and we're going to go look at what's going on and just understand what's happening.  So to do this, we're going to set up a little script that will duplicate this bug that we think that we found,  this thing that we don't understand.  When we run this, we see the output of the array that's been unique and gives us the result that we expected,  and we see that our other array returns us a nil, which was kind of confusing us.  So how do we start this up with gdb?  Well, as you might expect, it has some facilities for letting you start an arbitrary process instead of attaching to something already running.  You can do that with minus, minus args from the command line.  So we give it the same thing that we would want to do to run that Ruby script.  There are a variety of other ways and parameters you can use to control the exact execution if you need more fine-grained control.  So like before, when gdb starts up, it's going to drop us to a prompt, and the program is paused.  Now, this is not a program that has run yet, and so this is paused before anything has happened in our program.  And so to get it going, we tell gdb to go and run.  Like before, we get a lot of different output.  There's a lot of things that it's going to tell you, you know, as threads spin up, as it finds debugging symbols.  But in the midst of all of that output, we see the output to standard out, just like we expect from our program there in the middle.  We see our array with 123, and we see our nil.  So we know that this is actually running the code that we're interested in.  It finishes that script, lets us know that that process is finished, and then drops us back to the gdb prompt.  So we now have a process by which we can run the same thing.  We can run this test case that we want to understand better over and over again from gdb to try to dig into it.  But that's not as helpful as we'd like, right?  We want to look at what's happening specifically when we're calling this method on an array.  So we need to find some way to coerce gdb into putting us in the location that we want.  Well, fortunately, the pieces are all out there to help you very easily bridge the gap from the Ruby code that you care about  to the C functions that gdb understands for us to be able to hook into.  If you go to the Ruby docs for the array class and look at the unique bang method,  all of the documentation in Ruby up in the upper right corner, if you hover up there, it has a little thing that says click to toggle source.  You don't even have to go clone the repos for Ruby to be able to look at this information.  When you click that, a little section expands down below, and it shows you the source code for the particular Ruby method  that you're looking at.  And so here it is.  Here we have that.  And don't panic.  It's C. Some of the methods you'll find will be implemented in Ruby, things in the standard library.  But a lot of the core things in Ruby are implemented in C.  And so some of you may not be familiar with that language.  Some of you may be a little rusty on it.  So let's talk about one of a couple of fundamental pieces that you need to understand to be able to look at things in gdb  and get what's going on.  And this is a pretty straightforward one that C functions are the key to getting to the location that you want to to debug your problems.  So here is an example of one of the simplest possible C functions that you could have.  Now, if you kind of squint, this looks a lot like Ruby, but there are a few significant differences.  First off, we have to declare a return type.  We have to tell C what sort of thing we are going to return.  And if you don't return something, you have to tell it void for it to know that there is no return value.  We have a name for our functions.  So that's very comfortable and familiar and actually really key to us being able to work with this in gdb.  Then we also have a list of parameters.  And because C is a somewhat typed language, these parameters have a type.  So in this case, we have to pass along in as the value.  And C has no idea of an implicit return like we have in Ruby where the last line that executes is the value that will come back.  So we have to explicitly tell C that we want to return.  And here we're just returning 42 because it's the answer.  So how does this help us with our quest to figure out what's going on with UnikBang and understand the implementation of it?  Well, if we look at that code that showed up on our Ruby docs page here blown up a little bit,  we'll see some of those same pieces that we just talked about with any C function.  So our return value, which here they do a lot in MRI having the return value beyond the line above the name of the function.  So it returns a value and it takes a uppercase value as the type.  So value is the name that is used in the C code for a Ruby object.  Any Ruby object of any type that gets passed around throughout MRI will be represented by one of these values.  So here we have that UnikBang takes an array, it takes a value which we know is the array that we're calling this on.  And then it's going to return us back a Ruby object.  But the real key here, like what we need to be able to take our next step is this name.  So RB underscore array UnikBang is the hook that we need to get GDB to the location where we can peek around and see what's happening in the neighborhood.  As mentioned before, RB underscore is the prefix that's used in MRI code to indicate that things are a Ruby method.  The ary underscore is the prefix that it follows that with for methods that are related to the array class.  And this is fairly conventional. Most types in MRI that you find will have some sort of standard naming convention for the C functions that maps to the Ruby representation of when we execute this.  All right, so how do we make use of this name having figured out where the code is that we care about?  When we ran our args command to start GDB up with our process, it dropped us in and pauses.  And in that pause, we can tell GDB that we want to break whenever the RB array UnikBang method is called.  And at that point, if we then call run, then rather than executing the program all the way through and terminating, GDB will stop.  Tell us that it hit a break point and give us some of that information about it.  Show us a not very useful in this case snippet, which is the single line of code that it has actually paused on.  And we'll take a look at how to get a better view around that in a moment.  And then it provides us with a prompt.  So now we are back in a paused state where we can ask the world what it looks like, what values are, and control where GDB goes next.  But before we do that introspection, let's pause.  Roll back and review how we've gotten here.  So to be able to run a program from scratch ourselves in the context of GDB, we say GDB minus minus args, and then give it the invocation just like we would do with the command line.  That will start us in a paused state so we can set break points and take other operations, and then we tell it to run, and it will go do the things that we wanted it to.  C functions are one of the fundamental pieces of that language, and they're very similar to what Ruby does with methods.  There isn't an object involved, but they look a lot the same, and they're the foundational way that we can kind of navigate around and set break points within GDB.  Ruby docs, awesomely, provides us all the information we need to translate between the Ruby methods that we're familiar with and the C implementation that underlies that.  And with that information, we can get ourselves to stop anywhere that we want to in the execution of the Ruby VM when we're running our code.  All right, so we've started our program, we've gotten to the location we've seen a little bit of the code around where we're at, so let's take a deeper look at what's happening in this unique bang method and what's actually around us as we're executing.  The list command, if you have everything wired up correctly in GDB, will show you the source code that's before and after the line that we are paused on in our execution.  You do have to do a little bit of setup to inform GDB where the location of a copy of Ruby source code is, because the compiled binaries do not have that code inside of them.  But that's pretty simple to do.  So here we were on line 41.62, that curly brace right at the beginning of the function, because we asked it to stop there for us.  And so that's where we're paused, and we can see a little bit of what's going to come after us that we'll examine in a moment.  But before that, you know, there's this array parameter, so that's the array that we're calling this on.  And we'd really like to take a look at that.  Let's see what we can see about what's getting passed into this function.  So GDB has a couple of methods that are around this sort of behavior and have some formatting options and things, but one of them is display.  And so you can say display on any local or global variable that's in scope at the point that we're at.  And we ask to display the array member, the array variable that's getting passed in.  And it gives us this kind of cryptic response.  It says array is equal to this very, very large number.  Well, don't panic, but that's a memory location, and this is a pointer.  So that is the actual location in your computer's memory where the array that's been allocated for you is residing.  Now, this is probably not what you're actually wanting when you say display array.  You want to see what the actual values are or see something about how Ruby represents that.  But to get there, we need to take a slight diversion and talk about the other important piece of C that is good to know,  but at least the general shape of to be able to use these tools, and that is the struct.  So the struct is the way that C code allows you to group bits of data together to carry around in your program.  So here we have a struct.  It is named data, so you can think of that a little like a class name.  And when we have a struct of type data, it has one integer in it, value.  Now, what this means in practice is in C, a struct is literally a description of the layout of values in memory.  So if we have this data struct, that means that when we make one of these structs in place at somewhere,  like at this particular location, there will be an integer with the value that we've assigned sitting right at that place in memory.  Structs will nest and you can combine them together. So if we have a valuable data struct,  which its first member is that prior struct that we defined, and then we put a priority after it,  what we see in memory is we will see the integer for the value and then the priority from that outer containing struct that layers there.  Now, why do we care about this?  The reason that we care is because this is how every single Ruby object that you ever deal with in your code is actually represented under the covers.  There is a struct called RBasic and RBasic has a flag which indicates what type of Ruby thing this is and a pointer to a class.  And so any Ruby object, any of those value things that we saw getting passed around,  all that you really know when you get a value is that at that memory location, there will be the flag,  which will tell you how to interpret the rest of it, and some data about the class.  And then depending on the type of thing that you're looking at, the data that follows that in memory will take a particular shape.  So for instance, don't get scared by all of the details, but this is what an array looks like in Ruby.  It will have the flags, the class, and those come from that RBasic that it starts out with.  But then it has things around the length and the capacity and then the pointers to the values.  All of those things follow directly in memory right after those beginning fields that we got.  So Ruby has a number of different structs that it deals with.  Many of the internal types or the basic types that you deal with in Ruby have explicit structs that it implements, that represent them.  So things like strings, hashes, arrays, numbers are all represented by explicitly coded structs in MRI.  But all of our Ruby objects are also there.  The R object struct is the struct that is every single object of your own custom classes that you instantiate anywhere in your Ruby program.  So now we understand a little better when we said display array, what it's actually doing is it's giving us that memory location where we can get a flag and a class.  And if we knew how to structure that memory, we could interpret everything that follows from that address to get an actual picture of what Ruby thinks is there.  Now, don't get too wigged out, but this is a line of code that you can execute into GDB to tell it to actually peel that memory apart.  So GDB has loaded the R array structure so it knows what fields are supposed to be there and it knows how to interpret that and give us some basic display of it.  We can see in here that we've got our flags and our class, we've got the array data itself, which looks a little funny,  but this is giving us the capability to look literally at what the memory is that these objects represent.  And if this was something that we knew a little more about how arrays structured their data, we might be able to dig into the details and understand what's actually there.  If this was a custom object or you were writing a native gem, perhaps, these might be values that you would care about and understand.  Fortunately, Ruby provides us a simpler way to get at these sorts of values with a function called rb underscore p.  So this is a function that's in MRI, it's available there, and it literally will do the sort of output printing that you really actually want,  like the things that you get if you just print an object in Ruby itself.  You can invoke a C function in GDB by just saying call, so we can call rbp and pass it our array,  and it will print something, it will print out a literal representation of that array for us.  And here we've hit a little bit of an odd spot.  Our script had an array with one, one, two, three in it, but the output of this array where we hit our breakpoint is a string.  Well, I don't know exactly what's up with that, but before we dig into this mystery and before we figure out where this weird array value is coming from,  let's review what we've done in looking around the neighborhood and looking at the values that we've got.  So if you want to see the source code around whatever location you were paused at, you can tell GDB to list,  and it has some parameters if you want to list more or less of the information that's there.  To look at a local variable, you can call display on it, and if it's just something like a C string or a C number,  it will print that out neatly. You might need to take a little more pain.  Thanks, Java.  Good night.  All right.  What was I doing?  We were looking at an array, so you can display things.  There are a lot of formatting options with display that you can dig into to look at the various values that you might find around.  We've taken a brief look at C structures, one of the fundamentals of how C programs are built  and the way that data gets passed around for every object that you ever interact with in your Ruby program.  And we saw the RB print, which gives us some debug printing to be able to have a little bit of our Ruby niceties,  even though we're down at the C level.  But we kind of got off on a weird path here.  Like, we're seeing some values that we weren't expecting.  We're seeing this array, this array with one string in it.  It would be really nice to know where this is coming from and understand why we have this value.  And fortunately, MRI yet again provides us with what we need through the RB backtrace function.  So this function, unlike asking GDB for a backtrace, which is the C level backtrace of where we're at,  it will give us what Ruby thinks the backtrace is.  And this gets us back to territory that's a lot more familiar.  And this answers our question pretty quickly.  If we look at the files that this is going through, it's Ruby gems.  So as of any modern version of Ruby that you will be dealing with, Ruby gems is deeply integrated into Ruby itself.  And it assumes that you're going to need some gems and that you're going to do things.  And so it does some preparation when you start running your script.  And that's what we're hitting.  Ruby gems calls unique bang.  And that's the breakpoint that we hit was seeing not our script running,  but Ruby gems calling unique bang as it started up before it gets anywhere near executing our script.  All right.  So knowing that, Ruby gems has also provided a way out so we can say minus, minus disabled gems when we run this file.  And sure enough, when we do that, we hit the same breakpoint,  but the array actually has the values that we expect.  It's actually running our script so we can start looking at the implementation and the behavior there.  So let's look at how we can move through this function.  Just setting breakpoints everywhere is great for a start, but, you know, we need to be able to get around.  When we listed, we started out on the first line of this function on this, on this brace.  The next two lines that are after it are variable declarations,  and so nothing really happens on those lines.  And so when we ask GDB to go next, much like you would in any of your Ruby debuggers,  it will move us to the next line that has some actual activity going on it.  And it's this RB array modified check.  If you say next, like in most other debuggers,  next will continue execution at the next line in the current function that you're in  or return you back out if you've gotten to the end of the function.  Step, on the other hand, will drill in if there's anywhere that we can go.  So in this case, we were sitting on a function call, we were getting ready to call RB Modify Check.  And if we say step, GDB gives us the indication that it's moving into this other function,  tells us where it is, and then helpfully gives us the one line that we're actually resting on at this point in time.  If we do a listing, we can see that, yes, indeed, our context has completely changed.  We're over in this totally different function than where we were before.  If we want to get out of that, we can either just next through, or you can say finish,  and that will finish the currently executing function and pop you back up a level to where you were before.  All right, so now that we've got the basics of motion, the basics of being able to step through this code,  let's look at what the unique bang method is actually doing and see if we can get our heads around these return values  we weren't exactly expecting.  So almost right off the bat, after we get in there, we see this line.  So we look at the length of the array, and if it's less than or equal to one, we return q nil.  q nil is C Ruby's object for representing nil.  That is the nil that you interact with all the time in your Ruby applications.  That's the nil that we're all trying to escape with the dot question mark, ampersand, question mark thing,  try or whatever that's coming in.  Anytime you deal with a nil, that's it right there.  But this gives us our first hint that some of our intuition about things was probably wrong.  I mean, if the implementation here is this explicit about returning nil in this case,  they probably really intended for us to get those nils back that we were kind of surprised to be seeing.  All right, so let's keep moving on and see a little bit more of what's around here.  If we go down, there's a mode where a block can be given, but if we don't, we call into this function that says,  array make a hash out of this, and this creates us a hash based on the values that were in the array.  The keys of that hash will be the values, and so this is how unique and unique bang figure out what the unique set of elements are  in the array for us to care about.  Now, this is actually a really interesting point because to me, I often will call the bang methods for things  because I figured they're more efficient in terms of memory. They're not going to allocate new objects  because it's going to modify my things in place.  But here, in fact, we're allocating an entirely new hash which is then used just to determine whether we've actually changed things or not later on.  So unique bang might not be getting you the savings if you're thinking that it's avoiding memory allocations by calling a bang method.  Having gotten that hash back, we do the same trick that it was doing earlier of looking at the length of the array,  and here it says if the length of the array is the same as the size of the hash, we return nil.  And so this is the behavior that we saw up in our Ruby code, that if there are all unique items, if the array is already unique,  it's just going to return you a nil.  Well, so apparently we didn't find a Ruby bug, but we've had an interesting time being able to dig into it.  We'll let our program continue. If you type continue or see, there's a lot of shortcuts as well.  Things will go on and execute from there.  And in fact, as we go back and we're reviewing at the end of things,  apparently if we had actually read the documentation,  we might have understood that this method was doing exactly what it was supposed to do.  But I feel like the tools are important.  Knowing that you can dig into it is worthwhile.  So in this section, we've looked at moving around.  We've looked at navigating.  There's a number of other things that GDB provides, but the documentation for it is really good.  So dig into it.  Look at your code executing live and see what you can see of the underlayers of your Ruby.  To close out with, I want to show you just a few quick snippets of some of the other sillier or crazier things  that you can do once you're down at this level.  So one part to me is that the output in GDB can occasionally be very confusing.  And especially if you had an application that is also writing meaningful things to standard out,  those things get interleaved with the debugging output in ways that are kind of difficult to follow.  Well, we can actually get around that.  And this doesn't have anything to do, in fact, with Ruby.  This is just us interacting with C itself.  So we can tell C to close file descriptors one and two.  So that is standard out and standard error.  We can then open those.  We can open to a file.  And those file descriptors will be reused.  And one and two will then be pointed back to this GDB.log file that we have.  So any of the output that our program was going to generate,  instead of going onto our screen while we're debugging in our terminal,  will instead get sent over to a file and get saved.  This has been really useful for me in a couple of cases where having that output while I was actually trying to interactively debug  was very distracting and confusing.  This is probably one of the more complicated examples,  probably the most complicated in terms of commands that we will execute here.  But it's also one of the most fun.  So we can say call and call of C function.  Here we're calling the RBP function, which will give us debug printing.  And what we're calling that print on is Ruby's own eval method.  This eval method takes a string of Ruby.  So if there is any piece of Ruby that you want to execute while you're in the course of debugging,  you can eval it.  And in this case, what I'm doing is I'm asking it to give me a backtrace print of each of the threads.  So RB backtrace, which we looked at earlier, is really convenient.  But it only shows you the backtrace for the current thread.  And there's no facility for telling it to show us all of the threads.  So if we want to see the Ruby backtraces of all of our threads,  well, we can ask Ruby to go do it itself.  Now, be aware, as with many of these things, there is some peril.  Depending on your operating system, if there happens to be a garbage collection run happening  at the time that your program is being paused, this may not work properly.  But, you know, if you're already in the spot that either you're just experimenting  or your process is dead already,  it may be a fun thing to do.  GDB also provides us with a lot of automation facilities and other options for controlling its behavior.  You can write a script of GDB commands.  So here we attach to a specific process ID, run those backtrace commands that we've looked at earlier  and tell it, okay, I'm done, go stop.  And we can run this interactively from our command line saying batch and minus x.  And this will go and, you know, run a set script for us.  In fact, this was so useful that in the Ruby agent, we made a script called nrdbug.  And this uses, among other things, it runs a GDB script to extract those thread backtraces.  And we've used this in debugging deadlocking cases on users' machines.  And if you install this gem, you can run nrdbug and point it to any of your processes  and see a whole bunch of different output that it'll provide for you about what's happening in that process.  This same sort of scripting also allows you to customize your environment.  If you make a.rbinit file in your home directory,  you can put commands in there that will get executed whenever GDB starts up.  And I can finally rest easy being able to type exit instead of quit to get out of GDB.  All right. Let's recap briefly.  We started off and took a look at how GDB is a very useful tool,  in particular for debugging deadlock situations in places where your Ruby program might be frozen doing something below the Ruby layer.  We looked at a unique sort of problem where our understanding of what the Ruby VM was doing for us didn't match up with reality.  But we were able to look around, we were able to move through the code and figure out what it was doing for us  and realize that finding a Ruby bug isn't always as easy as you think it might be.  Lastly, we closed out with some crazy tricks that you can play.  And really, the sky's the limit.  We've talked about this in the context of Ruby,  but most of the applications that you are interacting with,  libraries that are on your system at some point, interact with this native layer of code.  And GDB can let you hook into that and see what your computer is really doing at a lower level.  Well, I hope that you found some cool tidbits in this today.  And I hope that maybe you'll go and debug a process somewhere on your computer very soon.  Thank you.  So the question is whether I found out why UnikBang returns nil.  And no, I have not gotten to the bottom of what the rationale of it is,  but it is clearly the specified behavior of that method.  And it did bite me at one point that it was doing that, as it has probably many other people in the room.  The question was, do you need to recompile Ruby with a dash g flag?  And yes, that is something that I did gloss over a little bit.  There may be a need to compile your Ruby with debugging symbols or at a lower level of optimization.  The exact steps for doing that varies depending on exactly how you're installing and compiling your Ruby.  So go look that up online.  One plug, if you do just want to play with it, the official Docker images that are made for Ruby on Docker Hub  actually have this all set up correctly and neatly so you can use those.  You could just spin up a little Docker container as a separate environment,  and it has all of the things that you need if you're locally compiled.  Ruby doesn't have the symbols that you're looking for.  The question was whether I've run into situations where re-entrancy has been an issue in debugging,  and I don't think that I have.  There certainly are sharp edges to using a tool like this,  and it strikes me as very possible that you could get effects like that,  which would be difficult to work around, but I have not personally run into that.  The question was, can you disable the garbage collector or check whether it's running  before you do some of those techniques that were hazardous, and the answer is yes.  In fact, if you were specifically setting up a test case for doing this sort of debugging,  unless what you're wanting to look at is GC-related, it might be worthwhile to pause that.  There are Ruby-level methods that you can just say, GC.disable, and it'll turn itself off.  That is certainly a good way in debugging that you could handle it.  All right, unless there's time for one more, but otherwise, thank you for your attention.  Thank you.
GDB_0_47.png GDB_48_1187.png GDB_1188_2039.png GDB_2040_2159.png GDB_2160_2567.png GDB_2568_2891.png GDB_2892_3107.png GDB_3108_11375.png GDB_11376_15971.png GDB_15972_16511.png GDB_16512_17003.png GDB_17004_20039.png GDB_20040_20795.png GDB_20796_22811.png GDB_22812_23063.png GDB_23064_23279.png GDB_23280_23327.png GDB_23328_26483.png GDB_26484_31031.png GDB_31032_31259.png GDB_31260_32531.png GDB_32532_33227.png GDB_33228_36923.png GDB_36924_40439.png GDB_40440_40991.png GDB_40992_42359.png GDB_42360_44483.png GDB_44484_44915.png GDB_44916_46343.png GDB_46344_52283.png GDB_52284_52583.png GDB_52584_52847.png GDB_52848_53255.png GDB_53256_53567.png GDB_53568_53615.png GDB_53616_53927.png GDB_53928_58499.png GDB_58500_58631.png GDB_58632_64103.png GDB_64104_64331.png GDB_64332_64440.png