devmatch - matching devices to modules
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Losh, M. Warner
FreeBSD ships a large (28MB) kernel by default. One reason is that FreeBSD cannot dynamically load drivers based on their identifying information. All FreeBSD drivers must match this information, and almost all have tables of device ids and other information they use to claim devices. There has been no general way to load these drivers, until devmatch. devmatch uses these tables in kernel modules to match them to devices in the system without drivers. devmatch matches both at boot, and as devices are dynamically added to the system. The kernel is smaller for everybody, while still preserving the convenience of today's larger kernel by loading seldom used drivers as needed. FreeBSD's kernel has been growing since the earliest days. FreeBSD 1.0R had a kernel that would fit on a floppy drive (the kernel + boot loader fit on a 1.2MB floppy). Now it is approaching 28MB on amd64. A large part of this growth has been due to the drivers. A kernel with all the drivers removed is only about 5MB, meaning over 80% of the kernel is drivers, most of them for devices that aren't in any given system. Since FreeBSD 3.2, newbus has published information about devices in the system. Each bus has it's own names for the identifying information the devices have, but FreeBSD uses pnpinfo as a generic term for this information. devd was added in FreeBSD 5.0 to generically do things with this information, including when no currently-loaded driver matched a device some self-enumerating bus found. At the time, it was thought it would be a simple matter to generate devd scripts from the plug and play information in the kernel. PC Card and USB drivers all used the the same format in FreeBSD 4, but all other busses were ad-hoc. In FreeBSD 8, USB moved its plug and play information into its own ELF section and extracted by a script to create devd scripts for autoloading. The experience with USB lead to the next generation design. Starting in FreeBSD 10, kldxref could extract the marked tables embedded in modules, along with metadata describing the table's layour (to better cope with the PCI mess). All the USB and PC Card drivers were marked in FreeBSD 11, as well as a few other drivers with other bus attachments to prove the design. devmatch, to be in FreeBSD 12, looks at the loader.hints file created by kldxref and matches it against unattached devices in the tree. Scripts in /etc/rc.d and devd tie this together. FreeBSD 12 is anticipated to ship with a smaller GENERIC kernel, with the bulk of the drivers loaded automatically through these means
 Okay, it looks like everybody's settled, so we'll go ahead and get started. It's 2.45. My name is Warner Loesch. I work at Netflix, and I'll be talking to you today about Dev Match. This is a program that I wrote to gravel through the kernel modules and figure out which one of them match devices we have in our Dev Tree that don't currently match anything and load the drivers. So we'll talk about all that. So a little bit of trivia to get started. Does anybody know how big the first edition, or at least the first edition we have of PDP 11 Unix kernel was? In terms of object code. No, it was 9K. Yeah, it was just bigger than 9K. That'll fit into a lot of things now. That was back when memory was, I think they had a PO for $100,000 to get the 16K, the 8K word, 16K byte memory card. How big is FreeBSD I386 11.1 generic kernel? 27 megs. That's the closest. It's 24 meg, almost, just under. So how fast has the FreeBSD kernel grown? Year over year as a percentage since the beginning of the project. I hear 5%, anything else? Three. It's about 20% a year. How large was it, the first one? First one was about 500K, 534K. Oh, right, okay. I had a patch, but you see, like, it's from 20%. Right, FreeBSD 1.0. I didn't go back and look at the patch kit images. That's a missing data point. That's a very good point. I'm not even starting with my talk and I'm missing things, but that's why I have you around. So I'm going to talk about why I did this today. And I went back and looked at the kernel size over time to see if the kernels keep growing, spoiler alert they do. And then after I go through that, which I think might be interesting to people, I'll give a background on the different pieces of the system that I had to touch to implement a dev match and then I'll go through the design of dev match and do a walkthrough of the changes I made, at least some of them in brief. And I'll give a little bit of a report about the problems encountered and I'll show you the results of how small I was able to create a kernel that would work. So my main motivation for this work is generic has gotten absolutely huge. This group, you know, it's grown from half a meg to 26 meg in current. And it takes a while for the student to build. I'm impatient. So I want to make it smaller so build times are faster because we're already building, we're building the kernel twice. We build all these modules and then we build the kernel. Why do, you know, we, it's going to work together anyway most of the time for most of the testing I need to do. But why do we have to build everything? As the kernel's gotten bigger, so is load time. If you're trying to net boot something, it's getting much slower and having a smaller generic would help for that. Another thing right now is third party modules. The integration is a little uneven and it would be nice if you install a module on your system if it just worked by default. If it has the plug and play information and you have that hardware, DevMatch will load it. And I've always wanted to do this because I thought it would be a really cool feature to have. Linux has had this feature for a long time. It's implemented in kind of a different way than what I did. But it's just something I've really wanted to do. So the kernel size. Kettle size growth has been exponential for a very long time. There are a lot of reasons for this. There are a lot more drivers now than there used to be. PreBSD has about 1700 drivers in it right now. Yes, some of them are obsolete and we could delete. That's not the topic for this talk. It's topic for another time when I have a baseball bat, perhaps. Just as a precaution. I'm not physically going to harm anybody. And there's a bit of proliferation of technologies. Version 7 Unix didn't really have networking. Yes, there was a packet packing interface in it that you could use. But TCP-IP wasn't really a thing yet. And now we have TCP stacks and SATA stacks and SCSI stacks and IPv6 and IPsec and crypto and the list just goes on and on and on. And this is a partial list that I have up here. I didn't have enough room on my slide to include all the technologies we have little stacks for. Also compiler writers have discovered that if you aggressively inline, the code runs faster. So that's also contributed to the bloat. So if we look at research Unix, it was originally, for those who don't know, done on the PDP-11. The first few versions of Unix were limited to 16K because that's all that the C compiler at the time could produce. In order to break the 16K barrier, they had to improve the C compiler so that it could generate larger images. The size of the kernel that they had was constrained because, like I said earlier, the memory modules were like $100,000 for a 16 or an 8K word module. And you need to have, you can use a little bit of that for the kernel and a little bit, you have to have some left for the user. The kernel can't use all that. As such, the systems were very static. So there wasn't a concept of a generic kernel. So I just picked one that was uniform over the series of data points. You compile the kernel you want and you put it on the hardware because you didn't have a lot of machines and there wasn't a lot of distribution. And the bootstrapping was a little complicated because you had to fast forward to the right tape mark sometimes and load the right kernel to get everything to work. But generally there wasn't a generic kernel. I included data points from the BSD2 series of releases. There were just a few. And system five and system three. So over the 30 years of releases that this spanned, the kernel growth rate was about 15 percent a year. And the early part of this was faster, almost 30 percent a year. I got all my data from the Unix Historical Society, twos.org, which is a good place to go to look for copies of old Unix if you ever need one. And we see there's two phases here. The initial phase of let's do a whole bunch of work where it grew from 8K, 9K, up to about 56K. And then the slower, well, we're in kind of a maintenance mode as it grew over time. And through the different Berkeley and USL releases. There was only one system five release. There is no SVR2 or SVR3 for PDP11. They dropped it by that time. And so I was wondering, well, okay, so that's one set of data points that showed exponential growth. Maybe that's unique. So I looked at the Vax releases because that's another long-lived series that everybody's familiar with. I started with 32V and went through the latest NetBSD release I could find. And the growth rate's about the same, 10 or 15% a year. And the initial part here, 32V was a swapping kernel, not a paging kernel. ThreeBSD, I believe, added swapping or added paging. By the time we get to fourBSD, we have all the network stuff added. So again, we see this pattern of initially a high growth and then kind of a plateau that, in this case, has lasted about the last 30 years. So we get to I386. Maybe threeBSD's different, I thought. I was wrong. FreeBSD has also grown as well. I've not tried to isolate whether all these things are due to additional modules or drivers or new technology or anything. I just took the, or the even compiler effects. I just took the kernel sizes stripped out of the release images that I could find online and came up with this graph. And again, we see initial, the first part we have a rapid growth until we got to about 3.0 and then the growth rate slows down. But we're still seeing an exponential growth rate because this is all the slides have been logarithmic. It's a power of two for each division further up, a doubling in size. So no, FreeBSD's not any different. And if you look at this graph, these are the three different slides I just showed you presented in a different way where all the sizes are normalized and it's a percentage growth or growth factor. The red, or sorry, the blue and the green lines are the PDP11 and the VACs and they're about the same. And you'll see the red data is from FreeBSD and we're growing a little faster even than the historical averages. So that's concerning. But if you look, there must be a lot more commits in FreeBSD, I thought. No, this is the number of commits that we've had going back to the beginning of the project. I have a more complicated version of this slide that I share from time to time on Twitter. The blue line is the trend line and then the red line is the actual numbers. This is a linear scale now, so we have a nice linear progression. It's not we're committing more code or we're doing more changes. It's the fact that the changes are bigger, the drivers are bigger, the technology that the drivers rest on is bigger. This just shows the similar thing. It's not quite as fast as Moore's law in terms of the doubling, but it's still very concerning because there's a lot of indication Moore's law isn't happening anymore. So we need to do something about the growth of the kernel size, otherwise we'll be growing faster than Moore's law and eventually we'll have a problem, particularly since it's exponential growth. So switching gears a little bit. I need to, I went one too many. I need to give a background on the pieces I touched. Obviously I'm touching new bus because that's where the, you know, it's the drivers and that's where all the plug and play data that we have for the project lives. I'm touching modules because I'm loading modules. I leveraged off the module system so I modified KLDX ref and then I added the glue to the system to allow it to automatically happen on boot or when devices were added to the system. So new bus for people that don't know, it's a hierarchical bus and each of the buses supply the plug and play information. Plug and play is kind of the old ISA add in card term for, you know, you just plug it in and it works because you're able to look at the card and enumerate it without having special knowledge of jumpers and address settings and IRQs and all of that. We adapted the term years ago. It's fallen out of fashion in the industry, but since that's what's in the code, that's what my talk will be using. The focus of the work is also on self enumerating buses, buses that know which devices are there. If you are trying to hint devices into the kernel, there's no real way to know whether the device is there or not without running the probe routine and we can't run the probe routine without loading the module and so we have to load all the hinted modules and we have a bunch of hints for things we know aren't there, kind of as fallback. So I just left the hinted buses alone. The bus is where we have to configure that. So this will be self enumerating buses. And generally speaking, the previous model is that we split device initialization into two pieces. We have a probe routine that bids on the device, looks at the plug and play information for the bus and say, oh yeah, I recognize that card. That's mine. I bid however much. Or I recognize that class of card. I'll bid a little bit less and if nobody bids me, I'll take it. And it sounds nice and abstract. But over time, we have a number of different buses and they do different things. Some of them have a nice centralized routine you can call. You pass it a standardized table. You call the routine, you get it back. Your probe is really small. Other buses have decided not to do this or they didn't do this in the original days. PCI is one. And so every PCI driver had to write this on an ad hoc basis. So there's a lot more diversity in those drivers. And the design couldn't just say, well, all buss, all drivers of this bus have this information it had to account for, these wildly varying tables that are in our drivers. So that's an important thing to remember for later. This is just a device tree. The important part here is the PCI bus has a lot of children. The PCI bus populates each of these children with a generic node and then calls the probe routine for each of the PCI drivers to ask if it's yours or not. This is a typical probe routine that is less bad than others for the RAL driver. And the important part here is there's a table and a loop that goes through the table and finds it and returns information. Here's an example of a bad routine that's much harder to fit into the thing. This is from the NVME driver, which is very recent. The first part looks really good. See if it's in my device table. But also it checks the device class and sees if it is something it recognizes. And that's a little bit more problematic because this is two different ways that the driver can attach. So that's another area that the design had to encompass. There's a, then we have a crazy probe. This is a very old driver. It's from the two-whip driver. And it checks the vendor ID. And if it's not deck, it returns. And if it's the sub-vendor isn't this thing, it returns because there were T1 cards that were made using this chip in a different mode. And it had to filter it out. And then if it's a particular chip ID, it sets the name and returns the, hey, this is my card. There's no table here. It's all in raw code. So something different has to be done for this driver. It's not going to be a simple one-line change like my design goal for all the other drivers. If you look at the previous drivers, hey, there's a table here we can tag and maybe do something extra. There's a table here we can tag. Including this, there's no table. So the driver or the modules that we have in the system, every driver is a module and we record a number of metadata. What does this driver attach to? What version is the driver? What does it depend on? That module system has been there for a long time. The boot loader knows how to read this stuff in and deal with it. It's tied into the sysinits to probe when you load a device. It's very well integrated into the system. And all of this metadata, when you do install kernel and you're not cross-compiling, is processed. It looks at all the KOs that were just installed and runs KLDXref to create a linker hits file that is kind of the key to a lot of the stuff I'm doing. This is just an example. This module depends on PCI and the firmware interface and it's a Waveland driver so there's a couple of other things and then it attaches to PCI. KLDXref, like I've said, takes all of the metadata that we put into the KOs and generates the hits file so that we can more quickly load the appropriate driver given a name. Usually this is run at install kernel time unless you're cross-building. That's one of the reasons we also now let you run it at boot. If you're building an ARM64 image on an AMD64 box, you can't create the proper hints file so we just create it at boot now on that platform. This is kind of a simplified data flow for what I just said. You have the driver, you build it, it has the module information going into the KO. We install it into boot kernel and run KLDXref to create the linker hints. It's pretty simple, pretty straightforward. We have an RC system that, and that runs to bring the system up. I'm not going to talk a whole lot about that. There's other places that people are talking about RC.D and other alternatives. The code for this is fairly simple. It'll fit into any system that we have. DevD is a program I wrote years ago that reacts to events in the kernel that happen. One of these events is I discovered a device that I don't have a driver match for. So DevD gets this device from the kernel and does things for it. I hooked into that later to run a script to try to dynamically load a mouse if you plug a mouse, a UMS driver if you plug the mouse in. So I showed you the data flow before and here's another version of it. The things in blue are the areas I changed. I added the PMP module info, which I'll talk about the format of here in a second. I changed KLDXref to parse it and simplify it. I wrote DevMatch, which takes the linker hints and looks at the device tree for the system that is in place and produces a list of modules to load. You can run it in a couple of other modes for debugging or to find which modules that have attached on the system don't have findable plug-and-play information, so you can maybe find some missing modules that wouldn't automatically load but should because if you're using the device, chances are other people are using the device as well. As you recall from the section on new bus, each of the drivers have a table in it. So the design is that we will mark this table in a way that can be interpreted outside of the driver. We'll extend KLDXref to save these markings, these decorations of the driver, and we'll write a program to do the matching, which is called DevMatch, which is the title of the SOC. One of the things we need to do in addition to the stuff I've talked about, there's some extensions to new bus that are needed for when you're loading multiple modules, and I'll explain why that's necessary in a minute. Modules are special in FreeBSD. They have full generic locking rather than the specific locking that is in the kernel, so there's a small penalty for locking, and the atomics are not inlined, so there's a small penalty for atomics. The exact extent to how much slower modules are, I've gotten estimates that vary from I can't measure it to I don't know, it should be something by logic, but I haven't seen any real numbers on that. There's some things we can do that are easy that will fix some of the module penalty, and I'll talk a little bit about those later. And finally, the last step in all of this is I created a minimal kernel years ago that didn't have any drivers in it, and that needs to be tuned a little bit. There are some drivers you can't put in this right now or that you can't have automatically loaded. But we need to have a conversation of ways that we can transition to having a leaner generic. So the plug-and-play information assumes we have a table. 90% of the drivers in the system have a table, probably 95, a few don't. We need to, in order for an external program to go look at this, we need to describe how big each table entry is because drivers have plug-and-play information and maybe some other stuff that says this is the version of the chip to use, use these quirks for this particular board, what have you. We need to be able to accommodate that because those are usually in one table and we don't want to force drivers to split that into two tables. We need to know how many entries there are because there's no real convention. Some say, oh, some pass a count to the routines that match or know how many entries there are. Others just have a sentinel at the end. And then we need to figure out what the format of the entry is so that we can interpret it outside of the kernel. So to do all of this, it leverages the existing module system where we already record this information. I did that to have a smooth transition. The last thing I want is to have big, to require all the drivers to have big changes. We have 1,700 drivers. We're not going to have big changes in all of the drivers. That's just a non-starter. Some of the buses are fortunately have a centralized probe routine, USB, PC card, and have the routine. So we can provide a macro for those which makes converting them very easy. I did all the USB and PC card drivers in a day. It's taken significantly longer than that to push this into the rest of the system. So I don't want to go into this in a huge amount of detail, but each driver table gets marked with a little meta language that has a type and a field name that matches the field name in the plug-and-play info. If you do a dev info minus v on your system, one of the things it prints is PNP info and a whole bunch of key value pairs. And those are the names that match the names here. And I'll get into what some of the cryptic stuff means, but as you might guess, U16 is a two-byte unsigned quantity that we can pull out and z is a string. And the hash sign is just for a comment. You know, this is here. It's a placeholder. You have to skip over it to find the information that's needed. I don't want to talk about all of these fields. You can read the slide. One of the interesting ones is PCI has basically two ways that people do device tables. One is to have a vendor column and a device column and maybe other columns. And the other is to combine the two into a word. The first case is really easy because that's the information that the PCI bus publishes. But the second case is harder because on big Indian machines it's stored one way and on little Indian machines it's stored another. So we have to tag those especially with W32, which has the two different fields so that, depending on the byte order, they, the proper values can be extracted. So for a centralized probe routine, this is all the change that's needed to make the UARC driver work is to add this one line. So host info and there's the table. And you know, this one I chose because it was one of the few drivers that just had one line. I could put this whole thing up on the screen and it would fit. But there are other drivers that have, you know, tables that look like this. This is one of the PCI drivers that has the vendor and device field encoded this way. And there's a whole number of them. So for this we have to say that the, we have to use the more raw macro that tells it, you know, what it is, where it attaches, where the table is, but also how many items are in the table. And there's a part that I didn't notice before that's right here that tells the table size or the entry size. I didn't notice that when I was preparing my slides, I'll have to fix that. So that's a more difficult thing. Now the D driver is the bent fork if you're not getting that. To fix that, we basically have to rewrite it. All these lines should have pluses in front of it. But basically you have a table now, you go through and find the right entry to do the matching and then you would write this entry here. So that's the changes for that. Fortunately there are not very many drivers like that in the tree. So one of the things that we need to do with Nubus is we need to introduce a concept of freezing and thawing the probing. If you've got multiple modules that match a particular device, let's say you have a mouse, a USB mouse, there's two different drivers in the system that could attach to that mouse. One is U-HID, which is a generic human interaction device driver that attaches to a whole lot of things and understands different protocols and presents one interface. And the other is UMS, which is specialized for mice. And based on the plug and play information, you can figure out which one will attach. But you don't know the priority. You know that both of these drivers could attach, but you don't know which one would have the priority. Now you might think, oh, well, we'll just encode the priority in that table you're generating, add another argument to your macro. Well, for this example, that would work. But if you go and look at the RE and RL drivers or the SK and DC drivers, you'll find there's a number of cases where the probe routine matches the same plug and play information, returns the same level, but has to go look at something secret or special on the card to know, oh, is this one for me or not? It isn't published by the PCI bus. And in that case, there's no way we could, that DevMatch could know and have enough information. It has to load both those drivers in hope for the best. But with the sysinids that are in the driver, we load one driver, the sysinids run, we load the other driver, the sysinids run. In the case where the tie happens, if you load the U-HID driver first, it wins. The U-Mouse driver comes up, oh, there's no unattached devices. I'm not going to do anything. It doesn't try to rebit or do anything like that. So to get around this problem, we need to say, hey, I know I'm going to be bloating a bunch of drivers. Don't do any probe and attaching for a little bit. Any probe and attaching that comes in put in a list to defer for later. And so we freeze plug-and-play, we load the drivers that we need to, or the modules that we need to, and then we thaw plug-and-play. And then all the drivers attached, they have a chance to bid. We handle both of the cases and the system can go on. One of the areas I didn't try to do is when we load multiple drivers, I don't try to figure out which one attached and then unload the other ones. That's a fairly rare case and an optimization that I don't think is worth making, but perhaps others do. So one of the big things I had to do to make this all work was modify KLDXREF to parse this new plug-and-play information. And one of the things I had hoped to accomplish was we've got all, you know, we had a very complicated table of the language that we had to understand, but I wanted to simplify that so that we had a simpler table that we could more easily match things with. So the simplified language I came up with is much simpler. Everything's an int, so we upcast to an int. There's a couple of types where ff means ignore, and so all the ffs get widened out to minus one so that we can ignore. And we just have a few types so that we could put this code that interprets this in the bootloader where we have a resource-constrained environment in addition to dev-match. I've not done the bootloader yet, but I designed it so that we potentially could in the future. So dev-match, as I've said already, parses the Winkr hits, uses live dev info to get the information from the kernel, matches up the unattached devices and prints out a list in its default mode. And there's a couple other things you can do. You can dump the Winkr hits file to debug it. You can look at all the devices in the system to see what matches to make sure that things are matching up like you think. And also look for devices in the system that have an attachment but don't have a plug-and-play market. One of the interesting things about this is you can take the Winkr hits from generic and put it on a system that has no modules and then run dev-match and point it at this file and it will tell you which modules you need to then copy over to that system. It's completely, once it hits the Winkr hits file, it's completely decoupled from all of the loadable modules, which is useful if you're trying to bring up a new system or you have a nano-BSD situation where you're trying to keep things really small and you can't put all of the drivers on to run a test to see which ones it would load. There's another mode where rather than parse the whole kernel tree every time a new device shows up, you can just hand it the no-match string that comes from the kernel and that's what the mode it operates in when we run from dev-d. We have a simple dev-match RC script that freezes the bus, loads the cache, runs dev-match to find the chaos to load, and then thaws the bus and everything works from there. And then the dev-match script leverages off of dev-d. When there's a no-match event, it just runs the RC script. Now this might not be so good, might not be very future-proof. This wouldn't work in an open RC environment, so I'll have to change that to using service. I noticed that right as I put the slide up, I should have changed it and committed it while I was preparing these slides, but it's an imperfect world that we live in. I had no idea it would be so talked about at this conference. So the next thing we need to do is look at a minimal kernel. None of this is any good with the current generic because everything's in generic. Well, if we look at a minimal kernel, what kind of minimal kernel do we need to make this work? Well, you need root in order to load the devices. So you can't load the devices for root with this mechanism. We could enhance the boot loader so that it would load all storage devices, but sometimes you can know that and sometimes you can't. In PCI-Land, there are all storage drivers, our particular class, so you could load that. Although there might be a few false positives with RAID controllers that didn't quite follow the PCI standard. Maybe they're not relevant to what we're doing, but they're out in the wild because every weird thing in the PCI-Land that can go wrong has gone wrong on at least one RAID controller. Scott's not laughing. He's like, yep. And I had to deal with every single one. So we could put that into the boot loader and it's probably useful to put in the boot loader because we can find the common cases. We can find NVME. We can find AHCI. We can find MPT, MPR, MPS, those drivers. We can maybe even load SDHCI. But then you go, well, what if I have a USB root? Then you can't do that through the boot loader or you need to tell the boot loader you have to load USB because USB isn't a storage device. It's a USB device. And do we speculate if we load it because it might possibly have storage on it or do we not because the loading things from the loader is a little slower than loading things in user land. And we'd rather wait till we have user land to load it. I don't have a good answer for that. It's one of the reasons I've not tried to implement this in the boot loader yet. Another class of devices. So let's say I load MPT. I also need to load CAM. Otherwise, there'll be nothing for MPT to attach to. Fortunately, most of the time the dependencies are listed. I didn't audit all the dependencies and I'll bet there's at least one that's missing. USB doesn't depend on that. We'd have to look at the USB bus and see if we find a UMass and go from there. UMass depends on it, though. So the other thing that you can't load at all from the boot loader is console drivers. Console drivers have to be compiled into the kernel right now. This is non-negotiable. There's no way around this. The console is initialized before the kernel goes off and looks to find the metadata that the loader left that tells it where all of the modules are to link in. So if you can load the driver, it'll act like a driver. It loads the UART driver, for example. It'll act like a UART driver. It just will never, ever be the console because it's not there at the time it needs to be there. So let's just note that Winkerhints is also read by the boot loader. I've talked about this slide. I'll move on. There's a Google Summer of Code project this year. I have a very good student, Lonk on Kenny Reddy, who is adding markings to all the PCI drivers that we have in the system. There's about 380 of them in the system right now. I will acknowledge that some of them are obsolete and should be removed. Again, that's a conversation for a different venue. He's going through the ones that are in the tree, and if we remove them before he gets to them, he'll not do them, and if not, he'll just keep cranking through. Probably 300 of them are entirely table-driven. So all that you need to do is figure out the right little syntax to write and write it and check it out and make sure it matches. There may be 50 more that are close, like the NVMe driver that I showed you, where there's some additional quirk or twist that you need to deal with. One common example is, like, for the NCR driver. It checks to see if it's a NCR vendor ID, and then it checks this big table of device IDs. So the NCR vendor ID is not in the table. So you would need to modify the table so that it's in there so that Dev Match can match with it. I might, this is a common of case, I might extend the language a little bit to be able to put something that says, assume that this field in the plug-and-play information is this particular value when you're looking for devices. And then there's 30 others that are like the two-whip driver. They're just crazy. Some of them are very ad hoc. They're no tables at all, a bunch of case statements. You know, take this number and this number and mask them together, and if it's one of these three numbers, then attach the device. We probably can turn all of that into table-driven. Some of the masking stuff we'll need to look at, maybe extending this for as well. I had no idea. I thought I knew all the drivers in the tree really well, and I had no idea this stuff existed before my summer of code student said, well, what about this driver? It's like, I've never heard of that driver. Wow, that's crazy, you're right. That is just insane. Why would they do that? So clearly, several of these were written a long time ago and are likely candidates, so I'm having him do those last because maybe never on some of them. So there are a couple of problems. The first one was lingering in the system for a long time. USB creates this huge string that's like 200 characters long for plug-and-play information, but the interface to get information out of the kernel was fixed strings of 128 bytes. So it didn't fit, it truncated, and this was the truncated string. It didn't match any of the drivers' plug-and-play information, so keyboards weren't loaded, mice weren't loaded. I had no clue until problem reports started rolling in that this was going on. It took a little while to figure this out. I changed the kernel library interface so that it exports this as just, you have a 4K buffer and you use the strings however you use the strings. You don't try to reserve space for each of the individual fields, so it's a lot like the string table and the old ADOT out executable files. But I was lucky. Mike Smith designed things such that the live DevInfo interface had pointers to this internal structure. So I could then, well, I can malloc the stuff and have pointers and there's no change, no user-visible change. So I didn't have to bump the library version number. So one of the problems that I came across earlier on is the IoAT driver. There's a DMA assist engine that's in newer Intel CPUs that lets you do DMA and there's like six of them or eight of them on at least the systems I had. And so we would try to load that driver eight times. The first time it would load and everything would probe and then the second time it would load and say it's already loaded. And the third time, you know, and so I would get a lot of noise at saying this is already loaded. So the solution was to sort and unify the list of modules to load. But the UMouse U-HIT issue that I've been referring to had to partially back that out. Hans put in a clue that knew what more specific and less specific was for USB and sorted things based on that in the DevMatch. And I'm looking forward to removing that code when we get the freeze and thaw into the system because I did not like it and I did not react well to it. Let's just say. So this is the slide for the U-HIT UMS problem. We need to have freeze and thaw to make it work. There are a lot of legacy drivers. We have a GSOC student working on the PCI ones. They're the most relevant for the project because our machines have PCI and PCIe. We have about 40 of those done. One is, three have been committed. The other 40 will get committed once the talk is done. I've been focused on my talk and not pushing his stuff in. He's getting a little antsy and I need to throw him a bone particularly since next week is the evaluation period. It's like, are you happy with my work? You haven't pushed it in but you haven't said anything? So no, we're happy. It's just busy. So the FDT drivers for ARM64 and other ARMS, we have 384 FDT drivers and maybe six of them are marked. If you want to help, find me afterwards. I'll drop everything and help you on the spot because we don't have the bandit. I don't have the bandwidth to do this. I thought I might. Most of these are table driven. A few aren't. So it's not as bad a mess as PCI was but it's not something that I can just sit down and go into brain dead mode and crank through in a day. So I talked about the module penalty that we encountered. I'm still looking for anybody that has reliable information on how doing something in the kernel versus doing something as a module affects things. So if you have that, I'd be interested to know. Two of the easy ones are the, we should go ahead and inline the locking and maybe look at why we have the different cases for the driver. I believe it's because the old UP versus MP kernels used to be a thing but now if we always build MP kernels then maybe we don't need modules that can deal with UP. So we need to look at that. We can inline the atomics and that will get rid of a lot of the penalty. The PIC penalty we put up with anyway for shared libraries so maybe it's not such a big deal to move to this. But it is one of the things that as we roll this out and slim down generic and have more modules loaded, we'll have to be mindful of and keep an eye out on the mailing list to make sure that there are no regressions. Does anybody know the answer? Yes, Jonathan. Okay, so for the record Jonathan says the long dead hand of the past has not un-gripped them yet and we should just, we should, no, no, no, we should throw the dead hand away, we should just honor it and make them be inlined, right? No. Okay. Okay, so as part of the, as part of the continual tuning we need to make them be whatever is appropriate and not force them to be one or the other and have the kernel and the module treated the same and if it gets inlined because it's better we'll do it the same both places. Is there going to be the solution for that? That's probably one of the reasons why people are saying that it's not a big problem. Okay, so the old ATA PCI driver, I had totally forgotten that this existed and I thought it was gone from the kernel but my student asked me about it. The PCI bus publishes instead of class, subclass and revision ID as three separate fields, it puts them all into one field and just calls it class. And the ATA driver matches the first two, the class and subclass and then uses the rev ID to figure out a bunch of things. It's not a, oh this is revision one or revision two or revision three of the spec, it's well a bit field. And it's this funky thing if this bit is set, it's that funky thing if that bit's set and there's this extra register over here if this bit is set and all the bits are valid and so we would need like 256 entries in order to cover them all. So we probably need to do some kind of masking. If we do masking for this, we can do masking for the other crazy PCI drivers that are in the tree that have a mask component to them. But you know, it's how do you match up the mask to the thing that's masked that might be a little bit tricky. So I thought we haven't done that yet. That's, we're putting this towards the end as well. So these are the things that aren't done yet. I'm not going to go over it in a whole lot of detail because I've talked about all of these points. This is where we're at, so the parts of the talk that aren't these are done. And then these are in process right now. So I bet you're wondering, well, did it matter? What did minimal turn out to be? Can anybody guess how big minimal is if you? I did three cases. Strip out all the non-root drivers, strip out all the non-root drivers and just need the root drivers that are popular. So just one or, you know, HCI, NVME, and maybe one or two others, MPT. And then what if you also strip out auxiliary fire systems and kernel options that could also be loaded? What do people think the, where approximately how big do you think the kernel is if you do those things? That's the bad. You're right. It's not 1.2 megabytes. I would love that to be the answer. Well, it's close. If you do the most extreme, most reduced version, it is like six, it's two-thirds, it's two-thirds savings, one-third the size. If you do just the first one where you leave all the possible root devices, including all the old crazy raid cards and stuff, it's about half. And if you have a medium thing, it's in between. So this should give a range of where we think minimal will wind up. It's going to be somewhere in this range. And in this slide, this is the same slide I showed earlier, except with Goddier colors, and it just goes back to FreeBSD 6, 7, and so forth. So the size of the kernel drops from there back to, it was when we, about FreeBSD 7, which is, if you look down here, about 10 years ago. I don't know. I don't know why there's a dip in 11.1. I don't know if something got deleted or we had a new compiler that was better. I don't know. I had made a note to find, try to figure that out, and I don't know. We didn't do that in, that's in current, but that's not in stable. It's only in LLD, so. So that's the answer. I have that in a chart, you know, about half for the moderate options, and the aggressive options is, you know, two-thirds. You know, the, today's kernel was 26 gig, AMD64, with fewer drivers is 28. I don't know how that happens, but it's a thing. So that's the talk that I have. Are there any questions from people? Emanuel? I can do that later, and I will make that my number one priority probably after going to the bathroom, so. Brooks? Is there a reason why KLA, LDX, or actually is it across to? No. It produces ints, and ints are the same across everything. It's just not. Indian friendly right now. So there's not a marking on the file that says this is big Indian or little Indian, and that's probably the biggest barrier. We'd have to change the format of it to make it so. Or we might be able to guess that from looking at the tag, the first few tags to see what order they're in, because the tags are stored as ints. But yeah, that was on the list of things I know I need to do, but it was below the line before the conference. That's all, well, if we solve that problem, we can make it a cross tool. And I mean, if worst case, we can pass it the Indianness on the command line, because we know that. We used to do that for the password files. We might still do that. We'll solve the cross building that way. Yes? Have you looked at other systems that, whether they're doing similar things or is this the previous differentiator now? Linux does this by putting all this data into a special elf section. Windows and OSX have it in a separate text file or registry that describes what to load. And both of them, I thought about doing that here as well, having this post-processing step just do that. But I was worried about multiple matches. And in OSX and Windows, that is the probe routine. That driver is assigned based on that information period. There's no secondary probe in the kernel. And I thought that would be too big of a change to FreeBSD's driver model. And I didn't want to introduce that. So that's the summary of the other systems. If I'd had more time for my talk, I would have added that. But I thought, you know, that's a good question. I almost had a section in my talk on that. But I was worried about running way too long as it was. And not everybody's enjoyed my talk. So I worried about the length as well. So are there other questions? That looks like no. Thank you very much. I'll take it. Thank you. Thank you.
 Okay, it looks like everybody's settled, so we'll go ahead and get started.  It's 2.45.  My name is Warner Loesch.  I work at Netflix, and I'll be talking to you today about Dev Match.  This is a program that I wrote to gravel through the kernel modules and figure out which one  of them match devices we have in our Dev Tree that don't currently match anything and load  the drivers.  So we'll talk about all that.  So a little bit of trivia to get started.  Does anybody know how big the first edition, or at least the first edition we have of PDP  11 Unix kernel was?  In terms of object code.  No, it was 9K.  Yeah, it was just bigger than 9K.  That'll fit into a lot of things now.  That was back when memory was, I think they had a PO for $100,000 to get the 16K, the  8K word, 16K byte memory card.  How big is FreeBSD I386 11.1 generic kernel?  27 megs.  That's the closest.  It's 24 meg, almost, just under.  So how fast has the FreeBSD kernel grown?  Year over year as a percentage since the beginning of the project.  I hear 5%, anything else?  Three.  It's about 20% a year.  How large was it, the first one?  First one was about 500K, 534K.  Oh, right, okay.  I had a patch, but you see, like, it's from 20%.  Right, FreeBSD 1.0.  I didn't go back and look at the patch kit images.  That's a missing data point.  That's a very good point.  I'm not even starting with my talk and I'm missing things, but that's why I have you  around.  So I'm going to talk about why I did this today.  And I went back and looked at the kernel size over time to see if the kernels keep growing,  spoiler alert they do.  And then after I go through that, which I think might be interesting to people, I'll  give a background on the different pieces of the system that I had to touch to implement  a dev match and then I'll go through the design of dev match and do a walkthrough of the changes  I made, at least some of them in brief.  And I'll give a little bit of a report about the problems encountered and I'll show you  the results of how small I was able to create a kernel that would work.  So my main motivation for this work is generic has gotten absolutely huge.  This group, you know, it's grown from half a meg to 26 meg in current.  And it takes a while for the student to build.  I'm impatient.  So I want to make it smaller so build times are faster because we're already building,  we're building the kernel twice.  We build all these modules and then we build the kernel.  Why do, you know, we, it's going to work together anyway most of the time for most of the testing  I need to do.  But why do we have to build everything?  As the kernel's gotten bigger, so is load time.  If you're trying to net boot something, it's getting much slower and having a smaller generic  would help for that.  Another thing right now is third party modules.  The integration is a little uneven and it would be nice if you install a module on your  system if it just worked by default.  If it has the plug and play information and you have that hardware, DevMatch will load  it.  And I've always wanted to do this because I thought it would be a really cool feature  to have.  Linux has had this feature for a long time.  It's implemented in kind of a different way than what I did.  But it's just something I've really wanted to do.  So the kernel size.  Kettle size growth has been exponential for a very long time.  There are a lot of reasons for this.  There are a lot more drivers now than there used to be.  PreBSD has about 1700 drivers in it right now.  Yes, some of them are obsolete and we could delete.  That's not the topic for this talk.  It's topic for another time when I have a baseball bat, perhaps.  Just as a precaution.  I'm not physically going to harm anybody.  And there's a bit of proliferation of technologies.  Version 7 Unix didn't really have networking.  Yes, there was a packet packing interface in it that you could use.  But TCP-IP wasn't really a thing yet.  And now we have TCP stacks and SATA stacks and SCSI stacks and IPv6 and IPsec and crypto  and the list just goes on and on and on.  And this is a partial list that I have up here.  I didn't have enough room on my slide to include all the technologies we have little  stacks for.  Also compiler writers have discovered that if you aggressively inline, the code runs  faster.  So that's also contributed to the bloat.  So if we look at research Unix, it was originally, for those who don't know, done on the PDP-11.  The first few versions of Unix were limited to 16K because that's all that the C compiler  at the time could produce.  In order to break the 16K barrier, they had to improve the C compiler so that it could  generate larger images.  The size of the kernel that they had was constrained because, like I said earlier, the memory modules  were like $100,000 for a 16 or an 8K word module.  And you need to have, you can use a little bit of that for the kernel and a little bit,  you have to have some left for the user.  The kernel can't use all that.  As such, the systems were very static.  So there wasn't a concept of a generic kernel.  So I just picked one that was uniform over the series of data points.  You compile the kernel you want and you put it on the hardware because you didn't have  a lot of machines and there wasn't a lot of distribution.  And the bootstrapping was a little complicated because you had to fast forward to the right  tape mark sometimes and load the right kernel to get everything to work.  But generally there wasn't a generic kernel.  I included data points from the BSD2 series of releases.  There were just a few.  And system five and system three.  So over the 30 years of releases that this spanned, the kernel growth rate was about 15  percent a year.  And the early part of this was faster, almost 30 percent a year.  I got all my data from the Unix Historical Society, twos.org, which is a good place to  go to look for copies of old Unix if you ever need one.  And we see there's two phases here.  The initial phase of let's do a whole bunch of work where it grew from 8K, 9K, up to about  56K.  And then the slower, well, we're in kind of a maintenance mode as it grew over time.  And through the different Berkeley and USL releases.  There was only one system five release.  There is no SVR2 or SVR3 for PDP11.  They dropped it by that time.  And so I was wondering, well, okay, so that's one set of data points that showed exponential  growth.  Maybe that's unique.  So I looked at the Vax releases because that's another long-lived series that everybody's  familiar with.  I started with 32V and went through the latest NetBSD release I could find.  And the growth rate's about the same, 10 or 15% a year.  And the initial part here, 32V was a swapping kernel, not a paging kernel.  ThreeBSD, I believe, added swapping or added paging.  By the time we get to fourBSD, we have all the network stuff added.  So again, we see this pattern of initially a high growth and then kind of a plateau that,  in this case, has lasted about the last 30 years.  So we get to I386.  Maybe threeBSD's different, I thought.  I was wrong.  FreeBSD has also grown as well.  I've not tried to isolate whether all these things are due to additional modules or drivers  or new technology or anything.  I just took the, or the even compiler effects.  I just took the kernel sizes stripped out of the release images that I could find online  and came up with this graph.  And again, we see initial, the first part we have a rapid growth until we got to about  3.0 and then the growth rate slows down.  But we're still seeing an exponential growth rate because this is all the slides have been  logarithmic.  It's a power of two for each division further up, a doubling in size.  So no, FreeBSD's not any different.  And if you look at this graph, these are the three different slides I just showed you presented  in a different way where all the sizes are normalized and it's a percentage growth or  growth factor.  The red, or sorry, the blue and the green lines are the PDP11 and the VACs and they're  about the same.  And you'll see the red data is from FreeBSD and we're growing a little faster even than  the historical averages.  So that's concerning.  But if you look, there must be a lot more commits in FreeBSD, I thought.  No, this is the number of commits that we've had going back to the beginning of the project.  I have a more complicated version of this slide that I share from time to time on Twitter.  The blue line is the trend line and then the red line is the actual numbers.  This is a linear scale now, so we have a nice linear progression.  It's not we're committing more code or we're doing more changes.  It's the fact that the changes are bigger, the drivers are bigger, the technology that  the drivers rest on is bigger.  This just shows the similar thing.  It's not quite as fast as Moore's law in terms of the doubling, but it's still very concerning  because there's a lot of indication Moore's law isn't happening anymore.  So we need to do something about the growth of the kernel size, otherwise we'll be growing  faster than Moore's law and eventually we'll have a problem, particularly since it's exponential  growth.  So switching gears a little bit.  I need to, I went one too many.  I need to give a background on the pieces I touched.  Obviously I'm touching new bus because that's where the, you know, it's the drivers and  that's where all the plug and play data that we have for the project lives.  I'm touching modules because I'm loading modules.  I leveraged off the module system so I modified KLDX ref and then I added the glue to the  system to allow it to automatically happen on boot or when devices were added to the  system.  So new bus for people that don't know, it's a hierarchical bus and each of the buses  supply the plug and play information.  Plug and play is kind of the old ISA add in card term for, you know, you just plug it  in and it works because you're able to look at the card and enumerate it without having  special knowledge of jumpers and address settings and IRQs and all of that.  We adapted the term years ago.  It's fallen out of fashion in the industry, but since that's what's in the code, that's  what my talk will be using.  The focus of the work is also on self enumerating buses, buses that know which devices are  there.  If you are trying to hint devices into the kernel, there's no real way to know whether  the device is there or not without running the probe routine and we can't run the probe  routine without loading the module and so we have to load all the hinted modules and  we have a bunch of hints for things we know aren't there, kind of as fallback.  So I just left the hinted buses alone.  The bus is where we have to configure that.  So this will be self enumerating buses.  And generally speaking, the previous model is that we split device initialization into  two pieces.  We have a probe routine that bids on the device, looks at the plug and play information for  the bus and say, oh yeah, I recognize that card.  That's mine.  I bid however much.  Or I recognize that class of card.  I'll bid a little bit less and if nobody bids me, I'll take it.  And it sounds nice and abstract.  But over time, we have a number of different buses and they do different things.  Some of them have a nice centralized routine you can call.  You pass it a standardized table.  You call the routine, you get it back.  Your probe is really small.  Other buses have decided not to do this or they didn't do this in the original days.  PCI is one.  And so every PCI driver had to write this on an ad hoc basis.  So there's a lot more diversity in those drivers.  And the design couldn't just say, well, all buss, all drivers of this bus have this information  it had to account for, these wildly varying tables that are in our drivers.  So that's an important thing to remember for later.  This is just a device tree.  The important part here is the PCI bus has a lot of children.  The PCI bus populates each of these children with a generic node and then calls the probe  routine for each of the PCI drivers to ask if it's yours or not.  This is a typical probe routine that is less bad than others for the RAL driver.  And the important part here is there's a table and a loop that goes through the table and  finds it and returns information.  Here's an example of a bad routine that's much harder to fit into the thing.  This is from the NVME driver, which is very recent.  The first part looks really good.  See if it's in my device table.  But also it checks the device class and sees if it is something it recognizes.  And that's a little bit more problematic because this is two different ways that the driver  can attach.  So that's another area that the design had to encompass.  There's a, then we have a crazy probe.  This is a very old driver.  It's from the two-whip driver.  And it checks the vendor ID.  And if it's not deck, it returns.  And if it's the sub-vendor isn't this thing, it returns because there were T1 cards that  were made using this chip in a different mode.  And it had to filter it out.  And then if it's a particular chip ID, it sets the name and returns the, hey, this is  my card.  There's no table here.  It's all in raw code.  So something different has to be done for this driver.  It's not going to be a simple one-line change like my design goal for all the other drivers.  If you look at the previous drivers, hey, there's a table here we can tag and maybe do something  extra.  There's a table here we can tag.  Including this, there's no table.  So the driver or the modules that we have in the system, every driver is a module and  we record a number of metadata.  What does this driver attach to?  What version is the driver?  What does it depend on?  That module system has been there for a long time.  The boot loader knows how to read this stuff in and deal with it.  It's tied into the sysinits to probe when you load a device.  It's very well integrated into the system.  And all of this metadata, when you do install kernel and you're not cross-compiling, is  processed.  It looks at all the KOs that were just installed and runs KLDXref to create a linker hits file  that is kind of the key to a lot of the stuff I'm doing.  This is just an example.  This module depends on PCI and the firmware interface and it's a Waveland driver so there's  a couple of other things and then it attaches to PCI.  KLDXref, like I've said, takes all of the metadata that we put into the KOs and generates  the hits file so that we can more quickly load the appropriate driver given a name.  Usually this is run at install kernel time unless you're cross-building.  That's one of the reasons we also now let you run it at boot.  If you're building an ARM64 image on an AMD64 box, you can't create the proper hints file  so we just create it at boot now on that platform.  This is kind of a simplified data flow for what I just said.  You have the driver, you build it, it has the module information going into the KO.  We install it into boot kernel and run KLDXref to create the linker hints.  It's pretty simple, pretty straightforward.  We have an RC system that, and that runs to bring the system up.  I'm not going to talk a whole lot about that.  There's other places that people are talking about RC.D and other alternatives.  The code for this is fairly simple.  It'll fit into any system that we have.  DevD is a program I wrote years ago that reacts to events in the kernel that happen.  One of these events is I discovered a device that I don't have a driver match for.  So DevD gets this device from the kernel and does things for it.  I hooked into that later to run a script to try to dynamically load a mouse if you plug  a mouse, a UMS driver if you plug the mouse in.  So I showed you the data flow before and here's another version of it.  The things in blue are the areas I changed.  I added the PMP module info, which I'll talk about the format of here in a second.  I changed KLDXref to parse it and simplify it.  I wrote DevMatch, which takes the linker hints and looks at the device tree for the system  that is in place and produces a list of modules to load.  You can run it in a couple of other modes for debugging or to find which modules that  have attached on the system don't have findable plug-and-play information, so you can maybe  find some missing modules that wouldn't automatically load but should because if you're using the  device, chances are other people are using the device as well.  As you recall from the section on new bus, each of the drivers have a table in it.  So the design is that we will mark this table in a way that can be interpreted outside of  the driver.  We'll extend KLDXref to save these markings, these decorations of the driver, and we'll  write a program to do the matching, which is called DevMatch, which is the title of  the SOC.  One of the things we need to do in addition to the stuff I've talked about, there's some  extensions to new bus that are needed for when you're loading multiple modules, and I'll  explain why that's necessary in a minute.  Modules are special in FreeBSD.  They have full generic locking rather than the specific locking that is in the kernel,  so there's a small penalty for locking, and the atomics are not inlined, so there's a  small penalty for atomics.  The exact extent to how much slower modules are, I've gotten estimates that vary from  I can't measure it to I don't know, it should be something by logic, but I haven't seen  any real numbers on that.  There's some things we can do that are easy that will fix some of the module penalty,  and I'll talk a little bit about those later.  And finally, the last step in all of this is I created a minimal kernel years ago that  didn't have any drivers in it, and that needs to be tuned a little bit.  There are some drivers you can't put in this right now or that you can't have automatically  loaded.  But we need to have a conversation of ways that we can transition to having a leaner  generic.  So the plug-and-play information assumes we have a table.  90% of the drivers in the system have a table, probably 95, a few don't.  We need to, in order for an external program to go look at this, we need to describe how  big each table entry is because drivers have plug-and-play information and maybe some other  stuff that says this is the version of the chip to use, use these quirks for this particular  board, what have you.  We need to be able to accommodate that because those are usually in one table and we don't  want to force drivers to split that into two tables.  We need to know how many entries there are because there's no real convention.  Some say, oh, some pass a count to the routines that match or know how many entries there  are.  Others just have a sentinel at the end.  And then we need to figure out what the format of the entry is so that we can interpret it  outside of the kernel.  So to do all of this, it leverages the existing module system where we already record this  information.  I did that to have a smooth transition.  The last thing I want is to have big, to require all the drivers to have big changes.  We have 1,700 drivers.  We're not going to have big changes in all of the drivers.  That's just a non-starter.  Some of the buses are fortunately have a centralized probe routine, USB, PC card, and  have the routine.  So we can provide a macro for those which makes converting them very easy.  I did all the USB and PC card drivers in a day.  It's taken significantly longer than that to push this into the rest of the system.  So I don't want to go into this in a huge amount of detail, but each driver table gets  marked with a little meta language that has a type and a field name that matches the field  name in the plug-and-play info.  If you do a dev info minus v on your system, one of the things it prints is PNP info and  a whole bunch of key value pairs.  And those are the names that match the names here.  And I'll get into what some of the cryptic stuff means, but as you might guess, U16 is  a two-byte unsigned quantity that we can pull out and z is a string.  And the hash sign is just for a comment.  You know, this is here.  It's a placeholder.  You have to skip over it to find the information that's needed.  I don't want to talk about all of these fields.  You can read the slide.  One of the interesting ones is PCI has basically two ways that people do device tables.  One is to have a vendor column and a device column and maybe other columns.  And the other is to combine the two into a word.  The first case is really easy because that's the information that the PCI bus publishes.  But the second case is harder because on big Indian machines it's stored one way and on  little Indian machines it's stored another.  So we have to tag those especially with W32, which has the two different fields so that,  depending on the byte order, they, the proper values can be extracted.  So for a centralized probe routine, this is all the change that's needed to make the  UARC driver work is to add this one line.  So host info and there's the table.  And you know, this one I chose because it was one of the few drivers that just had one  line.  I could put this whole thing up on the screen and it would fit.  But there are other drivers that have, you know, tables that look like this.  This is one of the PCI drivers that has the vendor and device field encoded this way.  And there's a whole number of them.  So for this we have to say that the, we have to use the more raw macro that tells it, you  know, what it is, where it attaches, where the table is, but also how many items are  in the table.  And there's a part that I didn't notice before that's right here that tells the table size  or the entry size.  I didn't notice that when I was preparing my slides, I'll have to fix that.  So that's a more difficult thing.  Now the D driver is the bent fork if you're not getting that.  To fix that, we basically have to rewrite it.  All these lines should have pluses in front of it.  But basically you have a table now, you go through and find the right entry to do the  matching and then you would write this entry here.  So that's the changes for that.  Fortunately there are not very many drivers like that in the tree.  So one of the things that we need to do with Nubus is we need to introduce a concept of  freezing and thawing the probing.  If you've got multiple modules that match a particular device, let's say you have a  mouse, a USB mouse, there's two different drivers in the system that could attach to  that mouse.  One is U-HID, which is a generic human interaction device driver that attaches to a whole lot  of things and understands different protocols and presents one interface.  And the other is UMS, which is specialized for mice.  And based on the plug and play information, you can figure out which one will attach.  But you don't know the priority.  You know that both of these drivers could attach, but you don't know which one would  have the priority.  Now you might think, oh, well, we'll just encode the priority in that table you're generating,  add another argument to your macro.  Well, for this example, that would work.  But if you go and look at the RE and RL drivers or the SK and DC drivers, you'll find there's  a number of cases where the probe routine matches the same plug and play information, returns  the same level, but has to go look at something secret or special on the card to know, oh,  is this one for me or not?  It isn't published by the PCI bus.  And in that case, there's no way we could, that DevMatch could know and have enough information.  It has to load both those drivers in hope for the best.  But with the sysinids that are in the driver, we load one driver, the sysinids run, we load  the other driver, the sysinids run.  In the case where the tie happens, if you load the U-HID driver first, it wins.  The U-Mouse driver comes up, oh, there's no unattached devices.  I'm not going to do anything.  It doesn't try to rebit or do anything like that.  So to get around this problem, we need to say, hey, I know I'm going to be bloating  a bunch of drivers.  Don't do any probe and attaching for a little bit.  Any probe and attaching that comes in put in a list to defer for later.  And so we freeze plug-and-play, we load the drivers that we need to, or the modules that  we need to, and then we thaw plug-and-play.  And then all the drivers attached, they have a chance to bid.  We handle both of the cases and the system can go on.  One of the areas I didn't try to do is when we load multiple drivers, I don't try to figure  out which one attached and then unload the other ones.  That's a fairly rare case and an optimization that I don't think is worth making, but perhaps  others do.  So one of the big things I had to do to make this all work was modify KLDXREF to parse this  new plug-and-play information.  And one of the things I had hoped to accomplish was we've got all, you know, we had a very  complicated table of the language that we had to understand, but I wanted to simplify  that so that we had a simpler table that we could more easily match things with.  So the simplified language I came up with is much simpler.  Everything's an int, so we upcast to an int.  There's a couple of types where ff means ignore, and so all the ffs get widened out to minus  one so that we can ignore.  And we just have a few types so that we could put this code that interprets this in the bootloader  where we have a resource-constrained environment in addition to dev-match.  I've not done the bootloader yet, but I designed it so that we potentially could in the future.  So dev-match, as I've said already, parses the Winkr hits, uses live dev info to get  the information from the kernel, matches up the unattached devices and prints out a list  in its default mode.  And there's a couple other things you can do.  You can dump the Winkr hits file to debug it.  You can look at all the devices in the system to see what matches to make sure that things  are matching up like you think.  And also look for devices in the system that have an attachment but don't have a plug-and-play  market.  One of the interesting things about this is you can take the Winkr hits from generic and  put it on a system that has no modules and then run dev-match and point it at this file  and it will tell you which modules you need to then copy over to that system.  It's completely, once it hits the Winkr hits file, it's completely decoupled from all of  the loadable modules, which is useful if you're trying to bring up a new system or you have  a nano-BSD situation where you're trying to keep things really small and you can't put  all of the drivers on to run a test to see which ones it would load.  There's another mode where rather than parse the whole kernel tree every time a new device  shows up, you can just hand it the no-match string that comes from the kernel and that's  what the mode it operates in when we run from dev-d.  We have a simple dev-match RC script that freezes the bus, loads the cache, runs dev-match  to find the chaos to load, and then thaws the bus and everything works from there.  And then the dev-match script leverages off of dev-d.  When there's a no-match event, it just runs the RC script.  Now this might not be so good, might not be very future-proof.  This wouldn't work in an open RC environment, so I'll have to change that to using service.  I noticed that right as I put the slide up, I should have changed it and committed it  while I was preparing these slides, but it's an imperfect world that we live in.  I had no idea it would be so talked about at this conference.  So the next thing we need to do is look at a minimal kernel.  None of this is any good with the current generic because everything's in generic.  Well, if we look at a minimal kernel, what kind of minimal kernel do we need to make  this work?  Well, you need root in order to load the devices.  So you can't load the devices for root with this mechanism.  We could enhance the boot loader so that it would load all storage devices, but sometimes  you can know that and sometimes you can't.  In PCI-Land, there are all storage drivers, our particular class, so you could load that.  Although there might be a few false positives with RAID controllers that didn't quite follow  the PCI standard.  Maybe they're not relevant to what we're doing, but they're out in the wild because  every weird thing in the PCI-Land that can go wrong has gone wrong on at least one RAID  controller.  Scott's not laughing.  He's like, yep.  And I had to deal with every single one.  So we could put that into the boot loader and it's probably useful to put in the boot  loader because we can find the common cases.  We can find NVME.  We can find AHCI.  We can find MPT, MPR, MPS, those drivers.  We can maybe even load SDHCI.  But then you go, well, what if I have a USB root?  Then you can't do that through the boot loader or you need to tell the boot loader you have  to load USB because USB isn't a storage device.  It's a USB device.  And do we speculate if we load it because it might possibly have storage on it or do  we not because the loading things from the loader is a little slower than loading things  in user land.  And we'd rather wait till we have user land to load it.  I don't have a good answer for that.  It's one of the reasons I've not tried to implement this in the boot loader yet.  Another class of devices.  So let's say I load MPT.  I also need to load CAM.  Otherwise, there'll be nothing for MPT to attach to.  Fortunately, most of the time the dependencies are listed.  I didn't audit all the dependencies and I'll bet there's at least one that's missing.  USB doesn't depend on that.  We'd have to look at the USB bus and see if we find a UMass and go from there.  UMass depends on it, though.  So the other thing that you can't load at all from the boot loader is console drivers.  Console drivers have to be compiled into the kernel right now.  This is non-negotiable.  There's no way around this.  The console is initialized before the kernel goes off and looks to find the metadata that  the loader left that tells it where all of the modules are to link in.  So if you can load the driver, it'll act like a driver.  It loads the UART driver, for example.  It'll act like a UART driver.  It just will never, ever be the console because it's not there at the time it needs to be  there.  So let's just note that Winkerhints is also read by the boot loader.  I've talked about this slide.  I'll move on.  There's a Google Summer of Code project this year.  I have a very good student, Lonk on Kenny Reddy, who is adding markings to all the PCI  drivers that we have in the system.  There's about 380 of them in the system right now.  I will acknowledge that some of them are obsolete and should be removed.  Again, that's a conversation for a different venue.  He's going through the ones that are in the tree, and if we remove them before he gets  to them, he'll not do them, and if not, he'll just keep cranking through.  Probably 300 of them are entirely table-driven.  So all that you need to do is figure out the right little syntax to write and write it  and check it out and make sure it matches.  There may be 50 more that are close, like the NVMe driver that I showed you, where there's  some additional quirk or twist that you need to deal with.  One common example is, like, for the NCR driver.  It checks to see if it's a NCR vendor ID, and then it checks this big table of device IDs.  So the NCR vendor ID is not in the table.  So you would need to modify the table so that it's in there so that Dev Match can match  with it.  I might, this is a common of case, I might extend the language a little bit to be able  to put something that says, assume that this field in the plug-and-play information is  this particular value when you're looking for devices.  And then there's 30 others that are like the two-whip driver.  They're just crazy.  Some of them are very ad hoc.  They're no tables at all, a bunch of case statements.  You know, take this number and this number and mask them together, and if it's one of  these three numbers, then attach the device.  We probably can turn all of that into table-driven.  Some of the masking stuff we'll need to look at, maybe extending this for as well.  I had no idea.  I thought I knew all the drivers in the tree really well, and I had no idea this stuff  existed before my summer of code student said, well, what about this driver?  It's like, I've never heard of that driver.  Wow, that's crazy, you're right.  That is just insane.  Why would they do that?  So clearly, several of these were written a long time ago and are likely candidates,  so I'm having him do those last because maybe never on some of them.  So there are a couple of problems.  The first one was lingering in the system for a long time.  USB creates this huge string that's like 200 characters long for plug-and-play information,  but the interface to get information out of the kernel was fixed strings of 128 bytes.  So it didn't fit, it truncated, and this was the truncated string.  It didn't match any of the drivers' plug-and-play information, so keyboards weren't loaded,  mice weren't loaded.  I had no clue until problem reports started rolling in that this was going on.  It took a little while to figure this out.  I changed the kernel library interface so that it exports this as just, you have a  4K buffer and you use the strings however you use the strings.  You don't try to reserve space for each of the individual fields, so it's a lot like  the string table and the old ADOT out executable files.  But I was lucky.  Mike Smith designed things such that the live DevInfo interface had pointers to this internal  structure.  So I could then, well, I can malloc the stuff and have pointers and there's no change, no  user-visible change.  So I didn't have to bump the library version number.  So one of the problems that I came across earlier on is the IoAT driver.  There's a DMA assist engine that's in newer Intel CPUs that lets you do DMA and there's  like six of them or eight of them on at least the systems I had.  And so we would try to load that driver eight times.  The first time it would load and everything would probe and then the second time it would  load and say it's already loaded.  And the third time, you know, and so I would get a lot of noise at saying this is already  loaded.  So the solution was to sort and unify the list of modules to load.  But the UMouse U-HIT issue that I've been referring to had to partially back that out.  Hans put in a clue that knew what more specific and less specific was for USB and sorted things  based on that in the DevMatch.  And I'm looking forward to removing that code when we get the freeze and thaw into the system  because I did not like it and I did not react well to it.  Let's just say.  So this is the slide for the U-HIT UMS problem.  We need to have freeze and thaw to make it work.  There are a lot of legacy drivers.  We have a GSOC student working on the PCI ones.  They're the most relevant for the project because our machines have PCI and PCIe.  We have about 40 of those done.  One is, three have been committed.  The other 40 will get committed once the talk is done.  I've been focused on my talk and not pushing his stuff in.  He's getting a little antsy and I need to throw him a bone particularly since next week  is the evaluation period.  It's like, are you happy with my work?  You haven't pushed it in but you haven't said anything?  So no, we're happy.  It's just busy.  So the FDT drivers for ARM64 and other ARMS, we have 384 FDT drivers and maybe six of them  are marked.  If you want to help, find me afterwards.  I'll drop everything and help you on the spot because we don't have the bandit.  I don't have the bandwidth to do this.  I thought I might.  Most of these are table driven.  A few aren't.  So it's not as bad a mess as PCI was but it's not something that I can just sit down and  go into brain dead mode and crank through in a day.  So I talked about the module penalty that we encountered.  I'm still looking for anybody that has reliable information on how doing something in the  kernel versus doing something as a module affects things.  So if you have that, I'd be interested to know.  Two of the easy ones are the, we should go ahead and inline the locking and maybe look  at why we have the different cases for the driver.  I believe it's because the old UP versus MP kernels used to be a thing but now if we always  build MP kernels then maybe we don't need modules that can deal with UP.  So we need to look at that.  We can inline the atomics and that will get rid of a lot of the penalty.  The PIC penalty we put up with anyway for shared libraries so maybe it's not such a  big deal to move to this.  But it is one of the things that as we roll this out and slim down generic and have more  modules loaded, we'll have to be mindful of and keep an eye out on the mailing list to  make sure that there are no regressions.  Does anybody know the answer?  Yes, Jonathan.  Okay, so for the record Jonathan says the long dead hand of the past has not un-gripped  them yet and we should just, we should, no, no, no, we should throw the dead hand away,  we should just honor it and make them be inlined, right?  No.  Okay.  Okay, so as part of the, as part of the continual tuning we need to make them be whatever is  appropriate and not force them to be one or the other and have the kernel and the module  treated the same and if it gets inlined because it's better we'll do it the same both places.  Is there going to be the solution for that?  That's probably one of the reasons why people are saying that it's not a big problem.  Okay, so the old ATA PCI driver, I had totally forgotten that this existed and I thought  it was gone from the kernel but my student asked me about it.  The PCI bus publishes instead of class, subclass and revision ID as three separate fields,  it puts them all into one field and just calls it class.  And the ATA driver matches the first two, the class and subclass and then uses the rev  ID to figure out a bunch of things.  It's not a, oh this is revision one or revision two or revision three of the spec, it's well  a bit field.  And it's this funky thing if this bit is set, it's that funky thing if that bit's  set and there's this extra register over here if this bit is set and all the bits are valid  and so we would need like 256 entries in order to cover them all.  So we probably need to do some kind of masking.  If we do masking for this, we can do masking for the other crazy PCI drivers that are in  the tree that have a mask component to them.  But you know, it's how do you match up the mask to the thing that's masked that might  be a little bit tricky.  So I thought we haven't done that yet.  That's, we're putting this towards the end as well.  So these are the things that aren't done yet.  I'm not going to go over it in a whole lot of detail because I've talked about all of  these points.  This is where we're at, so the parts of the talk that aren't these are done.  And then these are in process right now.  So I bet you're wondering, well, did it matter?  What did minimal turn out to be?  Can anybody guess how big minimal is if you?  I did three cases.  Strip out all the non-root drivers, strip out all the non-root drivers and just need  the root drivers that are popular.  So just one or, you know, HCI, NVME, and maybe one or two others, MPT.  And then what if you also strip out auxiliary fire systems and kernel options that could  also be loaded?  What do people think the, where approximately how big do you think the kernel is if you  do those things?  That's the bad.  You're right.  It's not 1.2 megabytes.  I would love that to be the answer.  Well, it's close.  If you do the most extreme, most reduced version, it is like six, it's two-thirds, it's two-thirds  savings, one-third the size.  If you do just the first one where you leave all the possible root devices, including all  the old crazy raid cards and stuff, it's about half.  And if you have a medium thing, it's in between.  So this should give a range of where we think minimal will wind up.  It's going to be somewhere in this range.  And in this slide, this is the same slide I showed earlier, except with Goddier colors,  and it just goes back to FreeBSD 6, 7, and so forth.  So the size of the kernel drops from there back to, it was when we, about FreeBSD 7,  which is, if you look down here, about 10 years ago.  I don't know.  I don't know why there's a dip in 11.1.  I don't know if something got deleted or we had a new compiler that was better.  I don't know.  I had made a note to find, try to figure that out, and I don't know.  We didn't do that in, that's in current, but that's not in stable.  It's only in LLD, so.  So that's the answer.  I have that in a chart, you know, about half for the moderate options, and the aggressive  options is, you know, two-thirds.  You know, the, today's kernel was 26 gig, AMD64, with fewer drivers is 28.  I don't know how that happens, but it's a thing.  So that's the talk that I have.  Are there any questions from people?  Emanuel?  I can do that later, and I will make that my number one priority probably after going  to the bathroom, so.  Brooks?  Is there a reason why KLA, LDX, or actually is it across to?  No.  It produces ints, and ints are the same across everything.  It's just not.  Indian friendly right now.  So there's not a marking on the file that says this is big Indian or little Indian, and  that's probably the biggest barrier.  We'd have to change the format of it to make it so.  Or we might be able to guess that from looking at the tag, the first few tags to see what  order they're in, because the tags are stored as ints.  But yeah, that was on the list of things I know I need to do, but it was below the line  before the conference.  That's all, well, if we solve that problem, we can make it a cross tool.  And I mean, if worst case, we can pass it the Indianness on the command line, because  we know that.  We used to do that for the password files.  We might still do that.  We'll solve the cross building that way.  Yes?  Have you looked at other systems that, whether they're doing similar things or is this the  previous differentiator now?  Linux does this by putting all this data into a special elf section.  Windows and OSX have it in a separate text file or registry that describes what to load.  And both of them, I thought about doing that here as well, having this post-processing  step just do that.  But I was worried about multiple matches.  And in OSX and Windows, that is the probe routine.  That driver is assigned based on that information period.  There's no secondary probe in the kernel.  And I thought that would be too big of a change to FreeBSD's driver model.  And I didn't want to introduce that.  So that's the summary of the other systems.  If I'd had more time for my talk, I would have added that.  But I thought, you know, that's a good question.  I almost had a section in my talk on that.  But I was worried about running way too long as it was.  And not everybody's enjoyed my talk.  So I worried about the length as well.  So are there other questions?  That looks like no.  Thank you very much.  I'll take it.  Thank you.  Thank you.
Warner_Losh__devmatch_matching_devices_to_modules_0_107.png Warner_Losh__devmatch_matching_devices_to_modules_108_251.png Warner_Losh__devmatch_matching_devices_to_modules_252_971.png Warner_Losh__devmatch_matching_devices_to_modules_972_3947.png Warner_Losh__devmatch_matching_devices_to_modules_3948_4007.png Warner_Losh__devmatch_matching_devices_to_modules_4008_4139.png Warner_Losh__devmatch_matching_devices_to_modules_4140_5171.png Warner_Losh__devmatch_matching_devices_to_modules_5172_5219.png Warner_Losh__devmatch_matching_devices_to_modules_5220_7283.png Warner_Losh__devmatch_matching_devices_to_modules_7284_7331.png Warner_Losh__devmatch_matching_devices_to_modules_7332_9347.png Warner_Losh__devmatch_matching_devices_to_modules_9348_12227.png Warner_Losh__devmatch_matching_devices_to_modules_12228_13055.png Warner_Losh__devmatch_matching_devices_to_modules_13056_13355.png Warner_Losh__devmatch_matching_devices_to_modules_13356_13739.png Warner_Losh__devmatch_matching_devices_to_modules_13740_14423.png Warner_Losh__devmatch_matching_devices_to_modules_14424_15311.png Warner_Losh__devmatch_matching_devices_to_modules_15312_16055.png Warner_Losh__devmatch_matching_devices_to_modules_16056_16799.png Warner_Losh__devmatch_matching_devices_to_modules_16800_17735.png Warner_Losh__devmatch_matching_devices_to_modules_17736_18431.png Warner_Losh__devmatch_matching_devices_to_modules_18432_18635.png Warner_Losh__devmatch_matching_devices_to_modules_18636_18695.png Warner_Losh__devmatch_matching_devices_to_modules_18696_18827.png Warner_Losh__devmatch_matching_devices_to_modules_18828_19391.png Warner_Losh__devmatch_matching_devices_to_modules_19392_23111.png Warner_Losh__devmatch_matching_devices_to_modules_23112_23627.png Warner_Losh__devmatch_matching_devices_to_modules_23628_24107.png Warner_Losh__devmatch_matching_devices_to_modules_24108_24899.png Warner_Losh__devmatch_matching_devices_to_modules_24900_25919.png Warner_Losh__devmatch_matching_devices_to_modules_25920_26003.png Warner_Losh__devmatch_matching_devices_to_modules_26004_26075.png Warner_Losh__devmatch_matching_devices_to_modules_26076_26159.png Warner_Losh__devmatch_matching_devices_to_modules_26160_27515.png Warner_Losh__devmatch_matching_devices_to_modules_27516_27995.png Warner_Losh__devmatch_matching_devices_to_modules_27996_28895.png Warner_Losh__devmatch_matching_devices_to_modules_28896_29315.png Warner_Losh__devmatch_matching_devices_to_modules_29316_29687.png Warner_Losh__devmatch_matching_devices_to_modules_29688_29771.png Warner_Losh__devmatch_matching_devices_to_modules_29772_29987.png Warner_Losh__devmatch_matching_devices_to_modules_29988_30299.png Warner_Losh__devmatch_matching_devices_to_modules_30300_30383.png Warner_Losh__devmatch_matching_devices_to_modules_30384_30431.png Warner_Losh__devmatch_matching_devices_to_modules_30432_30923.png Warner_Losh__devmatch_matching_devices_to_modules_30924_32303.png Warner_Losh__devmatch_matching_devices_to_modules_32304_35219.png Warner_Losh__devmatch_matching_devices_to_modules_35220_37859.png Warner_Losh__devmatch_matching_devices_to_modules_37860_39203.png Warner_Losh__devmatch_matching_devices_to_modules_39204_40691.png Warner_Losh__devmatch_matching_devices_to_modules_40692_41327.png Warner_Losh__devmatch_matching_devices_to_modules_41328_42443.png Warner_Losh__devmatch_matching_devices_to_modules_42444_42695.png Warner_Losh__devmatch_matching_devices_to_modules_42696_43175.png Warner_Losh__devmatch_matching_devices_to_modules_43176_47507.png Warner_Losh__devmatch_matching_devices_to_modules_47508_48419.png Warner_Losh__devmatch_matching_devices_to_modules_48420_49247.png Warner_Losh__devmatch_matching_devices_to_modules_49248_51755.png Warner_Losh__devmatch_matching_devices_to_modules_51756_53099.png Warner_Losh__devmatch_matching_devices_to_modules_53100_58223.png Warner_Losh__devmatch_matching_devices_to_modules_58224_58583.png Warner_Losh__devmatch_matching_devices_to_modules_58584_62639.png Warner_Losh__devmatch_matching_devices_to_modules_62640_62687.png Warner_Losh__devmatch_matching_devices_to_modules_62688_65003.png Warner_Losh__devmatch_matching_devices_to_modules_65004_66827.png Warner_Losh__devmatch_matching_devices_to_modules_66828_67211.png Warner_Losh__devmatch_matching_devices_to_modules_67212_68231.png Warner_Losh__devmatch_matching_devices_to_modules_68232_69215.png Warner_Losh__devmatch_matching_devices_to_modules_69216_73295.png Warner_Losh__devmatch_matching_devices_to_modules_73296_75359.png Warner_Losh__devmatch_matching_devices_to_modules_75360_75971.png Warner_Losh__devmatch_matching_devices_to_modules_75972_77147.png Warner_Losh__devmatch_matching_devices_to_modules_77148_77267.png Warner_Losh__devmatch_matching_devices_to_modules_77268_79415.png Warner_Losh__devmatch_matching_devices_to_modules_79416_79991.png Warner_Losh__devmatch_matching_devices_to_modules_79992_80051.png Warner_Losh__devmatch_matching_devices_to_modules_80052_81011.png Warner_Losh__devmatch_matching_devices_to_modules_81012_81551.png Warner_Losh__devmatch_matching_devices_to_modules_81552_81827.png Warner_Losh__devmatch_matching_devices_to_modules_81828_83027.png Warner_Losh__devmatch_matching_devices_to_modules_83028_84887.png Warner_Losh__devmatch_matching_devices_to_modules_84888_85091.png Warner_Losh__devmatch_matching_devices_to_modules_85092_85101.png