NDA: NVMe CAM attachment
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Losh, M. Warner
A walk through of CAM, its data flows, code flows and how to write drivers for it will be presented. CAM has been in the FreeBSD system for 17 years, yet it remains one of the more difficult subsystems to approach. It started out in a parallel SCSI world on systems with 16MB doing maybe 20 I/Os per second (iops) and a few disks. It has evolve to cope with systems with dozens of solid state drives that push tens of thousands of iops poised to move to hundreds of thousand. Towards that end, the author has written a CAM front end to the NVMe driver. This talk will walk through the new nvme front end. To understand the work, however, you need to understand CAM and how it fits together. The talk will focus on providing a tutorial for CAM. Starting with a brief overview of the FreeBSD I/O stack, the tutorial will focus on how CAM's connections to the system. I/O will be traced trough CAM, from its "periph" drivers that connect to the GEOM system to consume I/O from the upper layers, down through the transport (XPT) layer to the host interface modules (SIMs) which send the I/O requests to the drives. In addition to the simple data flows through CAM, various auxiliary details of CAM will be discussed. From a walk through of how CAM enumerates devices and how it divides those roles, to how the SIM drivers are created by the "new bus" drivers and how they interact with the BUS DMA system, these little discussed auxiliary details will be presented. Details of the new NVMe CAM front end will also be presented along side the more general details of CAM. Accompanying the talk will be a paper documenting the connections within CAM in more detail than the one hour format of the talk will allow. The author hopes to also complete man pages for all the CAM APIs before BSDcan, which this paper will cross reference.
 Okay, it looks like everybody's here that's going to be here. So I'll go ahead and get started. My name's Warner Loesch. I work for Netflix and I spent the last several months exploring the mysteries of CAM. When I submitted the talk, I had this rather boring title, NDA, the NVME CAM attachment. And if you want to grab the slides, they're available right now. If for some reason you want to follow along at home. But the real title of this talk is, How I Learn to Stop Worrying and Love CAM. Because it kind of reminded me of a wild ride that the folks in the bomber took when everybody went crazy in that Dr. Strangelove movie. So like I said, I work for Netflix. Most people here probably have heard of Netflix. Has anybody ever watched Netflix online in here? Okay, everybody. Yeah, how many on the previous laptop? Can you raise your hand, George? We know who to call to get the right string in so that the browser works. Yeah, yeah. I'm sorry, what? Yeah. Anyway, Netflix runs its own CDN. I believe a few other people in the room that don't work for Netflix also run CDNs. Is that right? Do I have anybody from? No, I guess not. Okay. Anyway, one of the things that we do at Netflix is operate at a fairly large scale as well. Is it 75 million yet, Scott? The number of subscribers worldwide? Something like that. We don't have 100,000 yet. There would be a party for that. 100 million, sorry, 75 million. So we need to anticipate the future. So when we're streaming in the evenings, according to Sandvine, about one-third of the internet traffic, I think 39% in their last report, is Netflix-related videos. So we're streaming terabits of second from all of our machines, all our thousands of machines around the world. This is a picture of some of our storage boxes that we have just racks and racks and racks of. So when we started the CDN, we had a bunch of storage boxes with hard disks with some flash to boot off of. Then we thought, well, flash is fast. We need to build some off-load devices, some flash storage devices. Anyway, we built some flash storage devices based on SSDs. And those worked out for a while, and then we got faster machines and needed a faster interconnect. So along with the rest of the industry, we're transitioning away from SSE drives to NVMe. And these happen to be Intel drives. We may or may not be using Intel drives in our product, but it was a picture that I had. They come in two form factors, either the hard disk look-alike form factor, which has the PCI lanes in a special cable to it, or an actual PCI board. Maybe I'll just do this. So what's motivating the move to NVMe? A lot of people probably have heard about this, yet the reasons for going with NVMe have not been well explained, perhaps. So when you have spinning disks connected via SAS or SATA, you have a very high latency. So it doesn't matter how much protocol overhead you have. The long barge clearly shows that you have like 20 milliseconds of latency and then microseconds of protocol and software overhead. But as devices got faster and faster and you wanted to increase IOPS and increase bandwidth, the software time and the protocol time started to matter a lot. So Intel went out and designed NVMe Express to try to overcome some of these limitations. And this graph shows the second generation card that has like a 60 microsecond latency. Some of the latest Gen3 cards actually get down to 10 or 15 microseconds of latency. So any software overhead, which used to be just in the noise, can be a significant part of the latency. Previous D can get full bandwidth from the NVMe cards that we've been using, thanks primarily to Drew's efforts at optimizing our whole network stack. We get good latency with FreeBSD, but we still need some optimization. We have about 30 microseconds. We could optimize out maybe another five or 10 if we look at the stack that we have on FreeBSD a little more closely. So one of the big questions I always get asked when I tell people I've done this, hey, I wrote a CAM front end for NVMe. People are like, why? There already is a disc front end for NVMe, the Jim Harris rip. And it seems to work fairly well. Well, this is true. But when we were evaluating NVMe, it became obvious to me fairly quickly that it would be very difficult to take advantage of the IOS scheduler that I had written for CAM to adapt it for the NVD driver. One thing to know about Netflix is we like to buy the lowest cost drives that are performance. And sometimes that low cost, you get kind of what you pay for. We can put more drives in the system that gives more redundancy. So if we buy drives that happen to fail a little bit, if one or two drives fail, we still have enough drives in the system to drive traffic and to have it be a useful system. But usually what happens is we'll get the drives in, everything looks good, we'll deploy it, and then we'll find some scenario in our workload where the drives are quirky. It can write at full speed only for a while. And once you write 10 or 20 gigabytes, the drives no longer behave at full speed. If you give it too many writes at the same time, it can have problems due to the problems with the firmware of the drive. So the IOS scheduler that I wrote and talked about last year smooths out all the performance quirks. And we figured we would need to do this if we were deploying NVMe. So this talk will be a little bit about how I did it, but primarily it will be about CAM, all the pieces of CAM, how CAM fits into the system, a little bit of the bigger part of the IOS stack so you could understand what CAM's role is in that, as well as understand the code while we're, if you were to go look at it. So we'll start with the FreeBSD IOS stack. This is a picture that I took and adapted from Kirk's and George's book on the design and implementation of FreeBSD. I simplified it a little bit and added the upper and lower designations and added the different pieces of CAM as well as a couple of the other drivers that are in the system that don't use CAM but do similar things. So this should be familiar to most people. The upper part of the stack, which was the top part of the picture, primarily focuses on the VM system. It's the buffer cache of the system. This allows you to touch pages and eventually they wind up on disk. But the actions in the upper part of the system are loosely coupled to the lower part of the file system stack. That means that if I make a page dirty at the upper layers, maybe it'll get flushed to disk now, maybe it'll be flushed to disk later, maybe I can change it a few more times, but there's not a direct, I touch the page and it goes to disk. At the next layer down is Gion, which handles partitioning and stuff. You can go look at one of Paul Hennings talks about that if you want to learn all the details of Gion. I'm just going to touch on it lightly here because CAM fits in underneath Gion. And CAM handles queuing and scheduling of IO handles, freezing the queues when there are errors so that error recovery can happen. And then IO resumed. It enforces different rules about what you can send to the drives, what the hardware in the system can support at any given time as well. If there are limitations, if you have a lot of disks on a particular, particularly an older disk controller, SCSI controller, you might, the disks might be able to do a bunch of IO, but the controller can only have so many outstanding IO's at once. So CAM handles all of that behind the scenes transparently to the user. Last year I talked about the CAM IO scheduler. I wanted to include pointers to that this year in my talk so that people can go find it, maybe watch the YouTube of it as well. If they wanted to read about the IO scheduler in more detail so that they can understand, so you guys can understand why we wanted to use it. Basically it collects real-time statistics that we find useful and helpful in managing our systems as well as shaping traffic to the drives, to keep the drives from saturating and going into overloaded, into a mode where they're overloaded and don't perform well. Okay so when I was preparing this talk I realized that there are a lot of things that are going on in the system with CAM that depend on the upper layers but aren't actually part of CAM. So I thought I'd touch on that just briefly. The design implementation book covers the buffer cache and the upper layers fairly well. So I thought I'd just start there. You have a file system or maybe a swapper or pager that's paging things in and out, reading around, writing things to the disk. It calls a bunch of stuff that eventually winds up as a B-read or a B-write which calls the strategy routine for the buff-obj and that winds up calling into Gion. So the red part here, the blue part is the buffer cache and then the red part is Gion where it converts the struct buffs to BIOS and then sends it down through all the layers and eventually winds up at the Gion disk layer at the bottom with a BIOS that's going to go to the disk exactly ready for the driver to deal with it. Drivers don't have to deal with partitioning, they don't have to deal with striping or anything. By the time the driver gets the command it's ready to go. So get this block, get this length and go. And that's the layer that I'll be talking about for the rest of the talk, the CAM layer. Another way to look at CAM is that we have the request coming in the top and they're dealt with by the Perif drivers. A Perif driver, I'll talk about this in more detail, all of these things in more detail, but briefly, a Perif driver is a driver that takes IO requests and turns them into some kind of protocol. It will take a block read and turn it into a SCSI read 10 command or an ATA command or a tape command or it will take an IOCTL with a protocol block already formed and just pass it through. It passes it off to the transport layer. One of the things that took me a while to understand with CAM a long time ago when it was first committed was I saw all these XPT things and it's like XPT, what the heck is that? And that's just transport. And the transport is responsible for taking these protocol blocks and giving them to somebody that can actually talk to the hardware. And that's something that's called a SIM. It's a system interface module. The name is the same for ATA or SCSI or whatever. I listed a few of the SIMs. I thought about listing them all, but we have like 30 or so in the tree right now and that would make the slide ugly. But the SIM takes the request from the transport layer, sets up DMA to send the request blocks to read data, write data, whatever. It goes into the hardware as you see at the bottom of the screen. And then the hardware does its thing, the drive does its thing, and eventually an interrupt happens and then the process reverses. The SIM notifies the Perif that the IO is done. The Perif calls bio done. It goes up the stack, calls buff done or one of the variations of it and the IO completes. So, I was talking about the different layers of CAM. I was talking about the Perif talking to the XBT talking to the SIM. And all that's done through a generalized message passing system called CCDs, command control blocks. These are the messages that flow back and forth between the various parts of CAM. They're present as one giant union of all possible messages so that the action routines that I'll talk about later have a nice simple prototype and don't have to cast void pointers. It helps with type safety a little. When you create a CCD and hand it off to be executed, some of the commands will execute right away. It connects to ask the transport layer about details about the transport. And the transport layer goes, oh, I know those. It fills in a little structure and completes the CCD. Others obviously are queued. I want to read a block off of the disk. Well, you hand that off. It goes down to the SIM. The SIM can't complete it right away, but it starts it, marks it as started and returns. And then when the controller finishes the IO, it will complete the CCD. We know about the users of the CCDs know that the CCD is completed because there's a callback routine that gets called. In some cases, you set up the CCD, you call done on the CCD, and that done routine actually does the thing that is requested. It's almost like you're just completing it and as a side effect, you get the information you want. That's one of the things when you go off and read the CAM code that can be a little confusing. It's like, this bit of code set up the CCD and then called done. How does that work? Well, if you see the callback routine that it does and go look at that, all the work gets done in the done routine. And once the CCD is completed, we have a number of status that's basically two parts. One part tells whether it's successful or not and the other part tells details of how it was successful or unsuccessful. So I also talked about the different layers, the messages that passed between them, but I didn't talk about how, what glues them together. CAM paths are what glue them together. They describe different nodes in the tree and all the physical objects in the tree may have one or more paths. For example, in the system I'm running, I have the pass through driver and the DA driver to talk to my two SSDs. And as you see, both of them at the bottom here in the CAM control devilist. As you see, both of them have both DA object or DA driver attached to them and a pass through driver attached to them. You can talk to the physical devices in a number of different ways. And CAM's paths help deal with that. We rough count the topology so as more instances, more drivers attached to a particular instance, we get, we're able to keep the resources around if other drivers go away. It used to be the case before all the bugs in the SCSI or the USB removal code got fixed. Yay! That you pull a USB drive out and USB would do its thing perfectly or not, but usually perfectly after Hans has worked on it. But you'd get a rough count error so you would not be able to detach it and you'd have to reboot. And that's what this is a result of. The rough counts on the topology, they were bugs so that when things went away, rough counts weren't being decremented so you couldn't unload drivers. Another piece for CAM that you need to know about is the async notifications. And the async notifications are just a way to broadcast information or broadcast that something has happened to different parts of CAM. And this is a very integral part of many things in CAM. It's used to enumerate the drives. It's used to attach device drivers to those drives. And it's used by the SIM to tell attached drivers that something interesting has happened. Some devices need special handling after a SCSI bus reset, for example. And when the SIM detects a SCSI bus reset happening, it's supposed to send a bus reset asynchronous message. One of the things that's interesting about CAM is relative to new bus is the model of finding devices is a bit inverted. The SIM, which you'd think would know about the devices, doesn't tell the rest of the system the devices are there. The rest of the system tells the SIM, hey, go scan for our devices. And the transport layer finds the devices, gets all the information for them, and then sends, hey, I found a device to all the drivers that are listening and they go do their thing. I'll get into that a little bit more detail. And new bus, in contrast, new bus would say, I'm in an attached routine. I know what's going on. I know what my children are. Let me tell new bus about the children and then the children go off and probe and attach. So it's a little different mindset. Both of these systems were created contemporaneously. And so each chose a slightly different way to model the devices. Earlier when I was talking about what CAM did for the system, one of the things I said was that it handled queuing. Modern disk drives have a number of slots that you can put IOs into. With SATA, they're tagged to tag queuing. You have 32 slots. You put the IO in, CAM's dev queue mechanism ensures that you never send more than 32 IOs down to the SIM for the drive. So you can't overflow it. The SIM has to deal with it if you accidentally do or for some reason through error-pandling, it winds up with more than it can handle. But for the most part, we try to pace it so that we always have just the right number of requests in the drive. And this can be dynamically resizable. At Netflix, even though our spinning disks have 32 slots, we finally get better performance if we only use the first 16. So at runtime, we set it to 16. Other people use this to say, well, I've monitored the drive somehow. I know it can handle eight right now. And then later, I've monitored the drive some more, and it can handle 12. And dynamically grow and shrink the number of commands that they allow. In addition, the dev queue can be frozen for error recovery. One of the interesting parts, unfortunately, that I won't get into too deeply today, is in CAM, the dev queue freezes and dies for various parts of the system. So when you want to submit IOs, but you know that the underlying layers aren't ready, you can freeze the queue, submit IOs, and then when you know the device is ready, you unfreeze it. This can happen during startup, but it can also happen during error recovery. The typical error recovery for most disk drives is cancel all the IOs that are present, do something special for that one IO that failed, and then resubmit the IOs. Well, the dev queue mechanism lets you freeze the dev queue so you have all the devices there. So new commands will happen, and then you can go deal with the error recovery, and once that's done, you can unfreeze it, and all the normal mechanisms for queuing and stuff apply, and you go from there. Yes? When you have the queue paused, it's only a problem if IO needs to complete. So if you have a drive that's failing, sorry, let me repeat the question. The question was, are there error types where you have to worry about the queue filling up? And there are some errors, like if the hard drive is spinning or has a bad failing, I mean, has a bad sector on it, and you have to keep retrying that one sector, or maybe it's a group of sectors. The dev queue will likely fill up with other requests that people want to do while you're doing that error recovery. Eventually, you need to terminate the error recovery, or the system becomes unresponsive, because like most preemptive kernels, previous D really wants to be able to do IO to do anything interesting. So now that I've talked about the various bits that glue the Perif, the XBT, and the SIM together, let me delve a little bit more deeply into those devices. Chem perforals, like I said, implement some protocol. They take block IO requests. They take block IO requests and convert them to Schesri read requests, or say to read requests, or something. But in addition to that, they participate in device enumeration. And I'll get into exactly how they do that over the next few slides. So once the Perif driver has converted the IO request to the protocol blocks, the Schesri protocol, NVMe protocol, whatever, it hands it off to the SIM via the XBT. I said that earlier. It's a very basic mechanism, so it bears repeating. Once the SIM signals completion, it notifies up the stack that the IO is done. The transport layer passes the messages back and forth between the Perif and the SIM, but it also enumerates the drives, or the devices that are on a given SIM. It will answer some of the CCBs that are asking about information about the transport, how fast it is, and so forth. It will answer those as well. I'll get into the probe device that the transport creates to enumerate devices in the system here in a little bit. SIM drivers, first of all, the S and SIM does not scan for Schesri. For a long time, I thought it did, but I went back and looked at the original CAM standards, and they said system interface module from the beginning. I thought that that was a later thinking under the feet when ATA was added, but that was there from the start. So it accepts the protocol blocks from the peripheral driver, writes them to the adapter, and answers different questions about the drive that it has figured out. So how does the SIM get created? One of the mysteries to me when I was first starting out with this was, how do we get one of these going in the system? That attaches to the new bus device enumeration. In your attach routine, typically, for new bus or in a routine that gets called in a deferred manner, you create a dev queue with CAM-SIM-Q-ALOC. You create the SIM, and you pass it a couple of interesting things. You pass it the name, you pass it the dev queue you just created, and you also pass it an action routine for receiving CCBs that it will deal with, and you give it a polling routine. The polling routine generally is only used when we're writing crash dumps because we don't have interrupts running, and we need some way to pull the SIM to see if the CCBs are done yet. After you create the SIM, you need to register each bus that the SIM supports. These days, it's usually a one-for-one mapping. In the past, it's been many to one. You do that with XBTBus register, and it goes off, and it registers the bus and the global list that CAM maintains. That allows you to participate in device enumeration. You then create a CAM path for the device enumeration with XBT Create Path. One of the things that I had to go looking for when I was preparing this talk was where does the XBT get created. That's something that's handled automatically in bus register. It associates the transport to the bus by asking the SIM what kind of bus it is. It sends a path in core ACCB. It finds out what type of bus it is. It sets up the function pointers to do that. In CAM right now, there's no per transport data. So it's just a function pointer table that will map for all instances of a particular transport type. That's handled automatically behind the scenes. You don't need to do anything special. One of the areas that I had a lot of trouble understanding when I was working on the NDA driver was how do the peripherals instantiate. I thought I would spend a little bit of time in this talk going through that. When the path is created by the SIM driver that I mentioned a couple slides back, it sends out a path registered notification. The XBT drivers that have registered for this, the probe devices for the XBT drivers get a notification and they kick off the whole process. The early part of boot when interrupts aren't enabled, the path registered get queued up and get released later once interrupts are running and the devices can be discovered. These path registered notifications get turned into XBT bus scan CCBs that go to the transport layer which kicks off everything. The transport layer runs a bit of a state machine to query the device. I'll get to the state machine and what it is in a little bit. Basically the transport knows how to ask all the devices on the system, all the common information. The SCSI transport knows how to do the inquiry command and maybe some auxiliary stuff to make sure the device is there and fill in the internal data structures that the peripherals will want to look at to see whether or not they want to claim this device. Similarly for the ATA transport layer, it knows how to send, identify commands. NVME knows how to get this information as well. The probe devices receive this information. Let me back up. Once the XBT probe device has finished its state machine, it will send an asynchronous message, AC found device. All the device perf drivers on the system that are listening for things will then take a look at the information that the transport has found and decide, yes, I can deal with this or no, I can't. In the example I gave earlier where I had an SSD attached to the pass device and the DA device, both of those drivers said, yes, I can deal with this. The pass through device created an instance of itself and the DA device created an instance of itself so they could handle the stuff that they handle. The reason I have the pass through device enabled in my system is the drives in that system generally are from vendors that wish that send us firmware updates as we find problems and the pass through device is what CAM control uses to update the device. Otherwise I just read and write blocks off of it with the DA device. Why is the probe a complicated state machine? Primary reason is, first, we can't block the, when you make a CCB request, it can't block waiting to complete. The probe routine has to go, oh, I know that there's a device there. I need to send an inquiry command in the case of SCSI. Well, since the inquiry command, it has to return right away. So it updates its state to a state that indicates it sent the inquiry command. When it completes, it knows, oh, I was in this state where I sent an inquiry command, I'll look at the inquiry command and based on the details of that, it will go to the next state. The state diagrams for the probe routines are a little bit complicated, so I'll go through a simple one and show you what at least my attempt at drawing SCSI's resulted in. So that basically leads us to the second reason that the state machines are complicated is because the protocols are complicated. If you're doing domain validation, you have to do particular things in a particular order to make sure that you can get all the commands through. If you want to support tag queuing, you have to do particular things in a particular order. And the state machines deal with this. It gets particularly complicated when you have enclosures and power management and stuff, so you have to spin drives up or spin them down or turn enclosures on and off and do stuff that Ken could probably tell you a lot about. But I looked at that and thought that would make a bad example and went to the next one to find a good example for a state machine. So the one good example I found was the one I just wrote. And part of the reason for this is that the NVMe Sim that I wrote, the NVMe driver has callbacks for when it detects devices and so forth. So it does all the state machine stuff behind the scenes and then sends a notification, hey, I found it. So I was able to leverage that and have just a couple of states. We start off in the invalid state. The first thing we do when we get a bus scan is we go to the identify state. We look at the data that's present and do all the stuff to notify that devices are found and wind up in the done state. If at any time during this a reset comes in, we go to the reset state, which just then immediately goes back to identify. We could probably skip the reset state, but when I was writing this I didn't realize that the reset state and the identify state are really the same state. So I tried to draw the SCSI state machine. It wasn't completely successful. This is after... I was about to add a disclaimer that I'm sure I'm missing at least one. But do you know which one that is so I can add it? But this shows the whole glory of it. There's no good tool to do this. But that talks about TUR as test unit ready. It goes into inquiry. We see if it has LUNs or not and report all the LUNs and VPDs and get the serial number and do domain verification. And eventually after you run the sausage mill backwards you've figured out that you've got a device and you get to the done state. But I invite you to go read the code if you'd like to A, fact check me, or B, understand what's going on. And there's something similar for ATA. There are more states and I couldn't fit that on the slide. So I'm not picking favorites here. I just chose this one as an example. So as I said earlier, when the Peruv driver is attaching, it gets the device found notification that is sent to everything. It has an async notifiter. If you're looking at the NDA driver, it's NDA async. It finds the device that it gets and goes, oh, and asks the transport, oh, what kind of device is this? And it looks at some details and said, yes, I can deal with that. So then it creates a CAM peripheral with CAM-PRIF-ALIC and it passes a number of routines into that. It passes a register routine, a start routine, and some more that I'll get to in a minute. So the registry routine is like an attached routine in new bus. It allocates the SASC, initializes the IOS scheduler. There's a whole bunch of stuff. I've listed them here and I want a little bit of time at the end for questions. The interesting bits are that it registers an async event, async callback for interesting events that might happen on that bus. And then it calls XPT schedule to get things started. Nothing happens in CAM without a trip through XPT schedule. XPT schedule is what is the mechanism that the DevQ that I talked about earlier uses to make sure that transaction slots are happening. And I believe I have a slide on it a little bit later. Anyway, here's a list of the required routines that you need to create a block device. They're pretty standard. Most of these come from the disk interface, which most people are familiar with. The interesting one, there are two here that are not part of the disk interface, though, that I'm going to call out. One is start. And that's called by XPT schedule when there's room for work. And the other is done. And you could have a different done routine for each request type. Most of the drivers in the tree just have one. And that gets called when the CCB has completed. So XPT schedule checks to see if there's room in the DevQ. If there is room, it allocates memory for the CCB and calls the peripheral start routine. The peripheral start routine goes, oh, I have a CCB. I can do work. Let me check the, you know, any work queues that I have. And in a couple of minutes, I'll get to where those work queues actually come from. I have a nice picture here that I will talk to. The other thing that XPT schedule does is if there's any shared resources that are limited. So if I've got a SIM that can do five requests at a time and I've got 10 drives attached to it, even though the drives can obviously do at least one IO at a time, it can only ever schedule five IOs in that case because that's the limit of the SIM driver. A lot of these limitations are for older hardware, maybe less relevant today. I know Justin has advocated rewriting a lot of the scheduling to simplify it, to simplify memory allocation. And that discussion is an interesting discussion, but beyond the scope of this talk. So when you're doing IO, you create the CCB and call XPT action, and that pushes it down to the XPT or to the SIM. And the SIM is done with a CCB. It gets an interrupt, it sees that the IO is completed, it usually looks up in a mapping table that says this cookie that I get an interrupt corresponds to the CCB so I can complete it. That calls XPT done. XPT done also calls XPT schedule. And that's, you can wind up with really weird stack traces, particularly if you have bugs in your code, I'm told. So before the callback, the done callback finishes, your start callback happens. And that's so that we can overlap the IO in the SIM a little bit better. The other thing it can do, if there's an error, it can freeze the queue and re-queue IO so that they can be retried. The strategy routine is like most normal strategy routines. It gets BIOS from the upper layers and queues them using either the CAMIO scheduler or Bio queue disk sort. And then it calls XPT schedule to maybe go do that IO. So in CAM, everything gets queued once and then it will get de-queued and executed. That's part of the inefficiency that we'll need to address if we want to handle high transaction rates for NVME. It's one of the reasons I think Justin's idea of rewriting this layer is a useful one because we can generally know if we can call the start routine immediately and bypass the strategy perhaps. Drew. What is the granularity of the processes like maxViz and maxViz? It's somewhere in between those two. Drew asked, what's the granularity of these messages? Is it maxViz? Is it a single disk block? It's a single transaction. So usually, like for UFS, it might be 16K or 32K, if it's one of our systems doing a big read ahead, it could be as large as maxViz. So that's the size. Does that answer your question? Yeah, with 30 microseconds of latency, you wind up, if I'm doing the math right in my head with 30,000 IOs a second before you run into problems. But if you're doing 30,000 IOs a second, you're going to consume an entire core just to do that and do the CAM processing. In our systems, we're doing like 128K or 256K transactions. So the transaction rate isn't a big concern for us. Inefficiency is not a huge deal, although I know you'll want me to get rid of it because every little bit counts. This is the primary reason why Justin thinks that you don't need to call XBT scheduled to schedule the IO, which queues and dequeues, you can do it directly from the strategy routine. So the start routine, you have a CCB, you know you have a slot in the SIM to do IO, so you fill it in and call XBT action and then call XBT schedule at the end to see if you have another slot because sometimes multiple slots open up, multiple completions happen before you can get back to the system, sometimes queues freeze and when they thaw you can get a bunch of stuff going on. This is also the place if you have complicated interaction between different types of IO. In SATA IO, for example, you can send a trim down, but it doesn't participate in NCQ. So either you send a trim down, you can only send one trim down, or you send your IO request down that are NCQ and the start routine enforces that restriction. Yes, there is a newer NCQ version of trim, but a lot of drives don't support that yet, so we still need to have code to deal with that. And then the done routine would be NDA done if you are looking at the source code and it's called by the SIM as part of the XBT done processing. It also calls XBT schedule to see if there is a slot available and like I said you can get some interesting inversions because it calls it before the done routine, sorry, wrong point. The done routine sometimes also has to call XBT schedule because of the IO restrictions. If I am doing a trim in SATA and it completes, I have to call XBT schedule because I want to unleash any read and write IOs that have come in in the interim and calling XBT schedule will kick it off. Everything gets clocked through XBT schedule. If you are riding a Perif driver and it's hanging, it never hurts except for performance sometimes to call XBT schedule. There were several bugs that I had when I was riding in DA. I guess I do know something about bugs. There were several bugs that I had in the DA where I wasn't calling XBT schedule. The IO would happen, I would see that was happening and then it would never complete and another one would never start. That was because I had left out calls to XBT schedule. Here is a picture of everything that I have been talking about where an IO comes in at the top and the middle of this into NDA strategy. It cues the bio to the disk queue and calls NDA schedule. NDA schedule is a wrapper around XBT schedule. If there are slots in the dev queue, XBT schedule calls NDA start which then calls bio queue first to get the bio off of the queue. The bio to CCB translation happens here as we pass it down and then it goes all the way down into hardware and comes back up and we get back to our done routine which calls NDA schedule. Then to make more things happen or it calls bio done to complete the IO up the storage stack. When you are writing a SIM, there are three routines that you need to write. SIM action is there to handle the CCBs. I have talked about that. Again there are the two types of CCBs, either the ones you queued to the device. It will take a while to happen or things you can immediately tell the XBT or the PRIF about the device. SIM pull is called when you have got a CCB outstanding in the dump routines. It is only used in dump although it could potentially be used elsewhere. Right now we only use it when we are dumping the system. When the system panics and we are taking a dump, we disable interrupts. I put that in for my nine-year-old son. When the system is writing out a crash dump, interrupts are disabled and we need some way to find that the IO is completed so we can write the next set of blocks. The SEMI RQ is what completes everything up the stack and calls XBT done. The key points, I hope people take away from this. These were some of the harder things that I had to learn. Although the first one was easy to learn once I realized. XBT means transport. SIM scans the bus for devices either when it starts or part of a path registered. The XBT probes the device by creating a special probe device or probes the information about the device, creating a special probe device. These probe devices go through a state machine to get all the information and then tell the PRIF drivers with a AC device found routine. There are many PRIF drivers that claim a single physical instance of hardware, which is new bus can't do that. Can can. And CCBs drive everything. If you don't know about CCBs, you're going to have a really rough time understanding the CAM code. And so I put this picture in to contrast with the picture at the beginning that had in the middle NVD and NVMe. This is what the IO stack looks like with the NDA peripheral where we've got all the – dang it, my slide's wrong. This NDA XBT should be NVMe XBT. But the layering is correct. So this is the different layering when you enable the NDA driver. I thought it might be interesting to, again, have another picture to see what's going on with this. And with that, I'd like to take any questions people have. Drew? Yes. I'm glad you asked that. The person you're sitting next to, Scott Long, maybe you know each other. He's been working with me, mostly him, though, on multi-Q extensions for CAM so that instead of having a SIMQ or a DevQ, you can have one per CPU or one per CPU complex or whatever makes sense. And in that case, then all of the IO wouldn't funnel through one data structure, which we all know from high-core count experience is bad for performance. Each one could get their own and they wouldn't contend on any locks to manipulate that data structure. So that's a very good thing. And hopefully next year, Scott will present his talk on CAM multi-Q after doing the work and getting some phenomenal results. Okay, so when I put disk sort up, that was an example of how you could schedule the IO. The disk sort does the elevator algorithm, which you really need to do for spinning rust, but for flash, no. You just enqueue it as the first one or the last one and then take it off. Right, and the DA, ADA and NDA drivers, the NDA driver doesn't do any disk sort at all. The other two, if it's got a spinning rate of zero, which means it's an SSD, it doesn't do any disk sort either. So George. I started looking at some of the CAMs and I just pointed out that it's almost all the working CAMs done at the boundary of those. Is that even going to be quite decreased without having to add statically defined points? Yes. Okay, sorry, my mistake. George was asking if all of the, he was looking at the CAM routines and noticed that a lot of the work seems to be done at the boundary points. And the answer is, and wanted to know if we could exploit that to make it easier for instrumenting it with Dtrace. And the answer is yes. CAM has, I didn't talk about it today, but CAM has a tracing routine that you can go through and see the CCBs going down and coming back up. And a lot of the code that's CAM, debug, blah, blah, blah, would be excellent candidates for Dtrace. Most of it would be in CAMXPT.C, although you would probably need a couple in each of the individual transports, the ATAXPT and the NVMEXPT as well to make that happen. But it would not be a ton of work to make that happen. Right now, so let me repeat the question. Alexander wanted to know what the overhead of the NVD versus NDA. Right now, NVD does things unmapped and the NDA driver still has a bug where it thinks it can't do unmapped IO. So with NVD, we see maybe 85 gigabytes a second, sorry, gigabytes a second in our 100G machines and with NDA, what were the best numbers? 65, 75? Right, right. So because of contention in the VM layer, we would get large latency spikes. So the control plane would slow down traffic to our system, so it was hard to see what the best peak you could get. But right now, there's a little bit of a performance difference because it doesn't deal unmapped IO. And the 80 gigabits is off of four drives, four lanes, PCIe4, so it's pretty close to line rate. Yes, Hans? I had a question about the state machine when it was right in this system. So it's a banking column and callback driven state machine? Okay, so yes. So the question was, is the state machine asynchronous, we callback driven? And the answer is yes. And I have you thought about making it synchronous like that, once that's our enumeration, do you simplify it? You don't leak reference time? I don't know if anybody's thought of doing that. One of the things with CAM is all of this happens in parallel anyway. We can start this device and this device and this device and send all those requests down to the SIM. And as they complete, it goes to the next one, so we don't unlike Nubus, which has one synchronous thread to enumerate everything and you have to wait for the next thing to come in. CAM's enumeration model allows all of that to proceed in parallel across even multiple SIMs and drives and buses and everything that can happen in parallel so that you don't have to wait for that. Yes, Justin, the other Justin. Hello, wieder met, and I'll see you guys in a bit. innocent, more than innocent, when errors are typically all over the public that have been hidden, that's innocent. But I thought, you know, that this there can't be those about to increase the community to the other side. So, to briefly repeat the question, when HCI completes a CCB, it marks it as innocent? It marks it as an error of the CCB, so there's an error of CCB that says it marks all the others as innocent. Oh, right, okay, so when the HCI is completing the IO, if it gets an error, it marks all the others as async, because when an error does occur with SATA, automatically all the other requests are considered to have not happened, and will return errors back, and so that reflects that. I think I'll have to talk with you afterwards for the details, or you can talk with Alexander, who wrote that code originally. He's sitting two people down from you to resolve that. But I think that's a little bit more detail than I can go into up here. And so, last question. Yes. Okay, so the question was, can you print the CAM tree similar to the JOM methods that we have? And there's CAM control dev list, which will list the devices, but I don't think we have anything that will print that in tree form. Ken or Scott or Justin, do we? Just to repeat your answer, since you're not mic'd, Ken said no, there's not. And he's doubtful that there would be additional useful information apart from the linear list, because CAM tends to be bus, here's the things on the bus, and there's not much more of a tree beyond that. And you would need to dig down into the specific details of Fiber Channel or SCSI to find more complicated topologies to report back, or to look at enclosures and that kind of thing. Okay, I think we're done with time, so if you have any other questions, come see me. Thank you very much.
 Okay, it looks like everybody's here that's going to be here.  So I'll go ahead and get started.  My name's Warner Loesch.  I work for Netflix and I spent the last several months exploring the mysteries of CAM.  When I submitted the talk, I had this rather boring title, NDA, the NVME CAM attachment.  And if you want to grab the slides, they're available right now.  If for some reason you want to follow along at home.  But the real title of this talk is, How I Learn to Stop Worrying and Love CAM.  Because it kind of reminded me of a wild ride that the folks in the bomber took when everybody  went crazy in that Dr. Strangelove movie.  So like I said, I work for Netflix.  Most people here probably have heard of Netflix.  Has anybody ever watched Netflix online in here?  Okay, everybody.  Yeah, how many on the previous laptop?  Can you raise your hand, George?  We know who to call to get the right string in so that the browser works.  Yeah, yeah.  I'm sorry, what?  Yeah.  Anyway, Netflix runs its own CDN.  I believe a few other people in the room that don't work for Netflix also run CDNs.  Is that right?  Do I have anybody from?  No, I guess not.  Okay.  Anyway, one of the things that we do at Netflix is operate at a fairly large scale as well.  Is it 75 million yet, Scott?  The number of subscribers worldwide?  Something like that.  We don't have 100,000 yet.  There would be a party for that.  100 million, sorry, 75 million.  So we need to anticipate the future.  So when we're streaming in the evenings, according to Sandvine, about one-third of the internet  traffic, I think 39% in their last report, is Netflix-related videos.  So we're streaming terabits of second from all of our machines, all our thousands of  machines around the world.  This is a picture of some of our storage boxes that we have just racks and racks and racks  of.  So when we started the CDN, we had a bunch of storage boxes with hard disks with some  flash to boot off of.  Then we thought, well, flash is fast.  We need to build some off-load devices, some flash storage devices.  Anyway, we built some flash storage devices based on SSDs.  And those worked out for a while, and then we got faster machines and needed a faster  interconnect.  So along with the rest of the industry, we're transitioning away from SSE drives to NVMe.  And these happen to be Intel drives.  We may or may not be using Intel drives in our product, but it was a picture that I had.  They come in two form factors, either the hard disk look-alike form factor, which has  the PCI lanes in a special cable to it, or an actual PCI board.  Maybe I'll just do this.  So what's motivating the move to NVMe?  A lot of people probably have heard about this, yet the reasons for going with NVMe  have not been well explained, perhaps.  So when you have spinning disks connected via SAS or SATA, you have a very high latency.  So it doesn't matter how much protocol overhead you have.  The long barge clearly shows that you have like 20 milliseconds of latency and then microseconds  of protocol and software overhead.  But as devices got faster and faster and you wanted to increase IOPS and increase bandwidth,  the software time and the protocol time started to matter a lot.  So Intel went out and designed NVMe Express to try to overcome some of these limitations.  And this graph shows the second generation card that has like a 60 microsecond latency.  Some of the latest Gen3 cards actually get down to 10 or 15 microseconds of latency.  So any software overhead, which used to be just in the noise, can be a significant part  of the latency.  Previous D can get full bandwidth from the NVMe cards that we've been using, thanks primarily  to Drew's efforts at optimizing our whole network stack.  We get good latency with FreeBSD, but we still need some optimization.  We have about 30 microseconds.  We could optimize out maybe another five or 10 if we look at the stack that we have on  FreeBSD a little more closely.  So one of the big questions I always get asked when I tell people I've done this, hey, I  wrote a CAM front end for NVMe.  People are like, why?  There already is a disc front end for NVMe, the Jim Harris rip.  And it seems to work fairly well.  Well, this is true.  But when we were evaluating NVMe, it became obvious to me fairly quickly that it would  be very difficult to take advantage of the IOS scheduler that I had written for CAM to  adapt it for the NVD driver.  One thing to know about Netflix is we like to buy the lowest cost drives that are performance.  And sometimes that low cost, you get kind of what you pay for.  We can put more drives in the system that gives more redundancy.  So if we buy drives that happen to fail a little bit, if one or two drives fail, we  still have enough drives in the system to drive traffic and to have it be a useful system.  But usually what happens is we'll get the drives in, everything looks good, we'll deploy  it, and then we'll find some scenario in our workload where the drives are quirky.  It can write at full speed only for a while.  And once you write 10 or 20 gigabytes, the drives no longer behave at full speed.  If you give it too many writes at the same time, it can have problems due to the problems  with the firmware of the drive.  So the IOS scheduler that I wrote and talked about last year smooths out all the performance  quirks.  And we figured we would need to do this if we were deploying NVMe.  So this talk will be a little bit about how I did it, but primarily it will be about CAM,  all the pieces of CAM, how CAM fits into the system, a little bit of the bigger part of  the IOS stack so you could understand what CAM's role is in that, as well as understand  the code while we're, if you were to go look at it.  So we'll start with the FreeBSD IOS stack.  This is a picture that I took and adapted from Kirk's and George's book on the design  and implementation of FreeBSD.  I simplified it a little bit and added the upper and lower designations and added the  different pieces of CAM as well as a couple of the other drivers that are in the system  that don't use CAM but do similar things.  So this should be familiar to most people.  The upper part of the stack, which was the top part of the picture, primarily focuses  on the VM system.  It's the buffer cache of the system.  This allows you to touch pages and eventually they wind up on disk.  But the actions in the upper part of the system are loosely coupled to the lower part of the  file system stack.  That means that if I make a page dirty at the upper layers, maybe it'll get flushed  to disk now, maybe it'll be flushed to disk later, maybe I can change it a few more times,  but there's not a direct, I touch the page and it goes to disk.  At the next layer down is Gion, which handles partitioning and stuff.  You can go look at one of Paul Hennings talks about that if you want to learn all the details  of Gion.  I'm just going to touch on it lightly here because CAM fits in underneath Gion.  And CAM handles queuing and scheduling of IO handles, freezing the queues when there  are errors so that error recovery can happen.  And then IO resumed.  It enforces different rules about what you can send to the drives, what the hardware  in the system can support at any given time as well.  If there are limitations, if you have a lot of disks on a particular, particularly an  older disk controller, SCSI controller, you might, the disks might be able to do a bunch  of IO, but the controller can only have so many outstanding IO's at once.  So CAM handles all of that behind the scenes transparently to the user.  Last year I talked about the CAM IO scheduler.  I wanted to include pointers to that this year in my talk so that people can go find  it, maybe watch the YouTube of it as well.  If they wanted to read about the IO scheduler in more detail so that they can understand,  so you guys can understand why we wanted to use it.  Basically it collects real-time statistics that we find useful and helpful in managing  our systems as well as shaping traffic to the drives, to keep the drives from saturating  and going into overloaded, into a mode where they're overloaded and don't perform well.  Okay so when I was preparing this talk I realized that there are a lot of things that are going  on in the system with CAM that depend on the upper layers but aren't actually part of CAM.  So I thought I'd touch on that just briefly.  The design implementation book covers the buffer cache and the upper layers fairly well.  So I thought I'd just start there.  You have a file system or maybe a swapper or pager that's paging things in and out, reading  around, writing things to the disk.  It calls a bunch of stuff that eventually winds up as a B-read or a B-write which calls  the strategy routine for the buff-obj and that winds up calling into Gion.  So the red part here, the blue part is the buffer cache and then the red part is Gion  where it converts the struct buffs to BIOS and then sends it down through all the layers  and eventually winds up at the Gion disk layer at the bottom with a BIOS that's going to  go to the disk exactly ready for the driver to deal with it.  Drivers don't have to deal with partitioning, they don't have to deal with striping or anything.  By the time the driver gets the command it's ready to go.  So get this block, get this length and go.  And that's the layer that I'll be talking about for the rest of the talk, the CAM layer.  Another way to look at CAM is that we have the request coming in the top and they're  dealt with by the Perif drivers.  A Perif driver, I'll talk about this in more detail, all of these things in more detail,  but briefly, a Perif driver is a driver that takes IO requests and turns them into some  kind of protocol.  It will take a block read and turn it into a SCSI read 10 command or an ATA command or  a tape command or it will take an IOCTL with a protocol block already formed and just pass  it through.  It passes it off to the transport layer.  One of the things that took me a while to understand with CAM a long time ago when it  was first committed was I saw all these XPT things and it's like XPT, what the heck is  that?  And that's just transport.  And the transport is responsible for taking these protocol blocks and giving them to somebody  that can actually talk to the hardware.  And that's something that's called a SIM.  It's a system interface module.  The name is the same for ATA or SCSI or whatever.  I listed a few of the SIMs.  I thought about listing them all, but we have like 30 or so in the tree right now and that  would make the slide ugly.  But the SIM takes the request from the transport layer, sets up DMA to send the request blocks  to read data, write data, whatever.  It goes into the hardware as you see at the bottom of the screen.  And then the hardware does its thing, the drive does its thing, and eventually an interrupt  happens and then the process reverses.  The SIM notifies the Perif that the IO is done.  The Perif calls bio done.  It goes up the stack, calls buff done or one of the variations of it and the IO completes.  So, I was talking about the different layers of CAM.  I was talking about the Perif talking to the XBT talking to the SIM.  And all that's done through a generalized message passing system called CCDs, command  control blocks.  These are the messages that flow back and forth between the various parts of CAM.  They're present as one giant union of all possible messages so that the action routines  that I'll talk about later have a nice simple prototype and don't have to cast void pointers.  It helps with type safety a little.  When you create a CCD and hand it off to be executed, some of the commands will execute  right away.  It connects to ask the transport layer about details about the transport.  And the transport layer goes, oh, I know those.  It fills in a little structure and completes the CCD.  Others obviously are queued.  I want to read a block off of the disk.  Well, you hand that off.  It goes down to the SIM.  The SIM can't complete it right away, but it starts it, marks it as started and returns.  And then when the controller finishes the IO, it will complete the CCD.  We know about the users of the CCDs know that the CCD is completed because there's a callback  routine that gets called.  In some cases, you set up the CCD, you call done on the CCD, and that done routine actually  does the thing that is requested.  It's almost like you're just completing it and as a side effect, you get the information  you want.  That's one of the things when you go off and read the CAM code that can be a little confusing.  It's like, this bit of code set up the CCD and then called done.  How does that work?  Well, if you see the callback routine that it does and go look at that, all the work  gets done in the done routine.  And once the CCD is completed, we have a number of status that's basically two parts.  One part tells whether it's successful or not and the other part tells details of how  it was successful or unsuccessful.  So I also talked about the different layers, the messages that passed between them, but  I didn't talk about how, what glues them together.  CAM paths are what glue them together.  They describe different nodes in the tree and all the physical objects in the tree may  have one or more paths.  For example, in the system I'm running, I have the pass through driver and the DA driver  to talk to my two SSDs.  And as you see, both of them at the bottom here in the CAM control devilist.  As you see, both of them have both DA object or DA driver attached to them and a pass through  driver attached to them.  You can talk to the physical devices in a number of different ways.  And CAM's paths help deal with that.  We rough count the topology so as more instances, more drivers attached to a particular instance,  we get, we're able to keep the resources around if other drivers go away.  It used to be the case before all the bugs in the SCSI or the USB removal code got fixed.  Yay!  That you pull a USB drive out and USB would do its thing perfectly or not, but usually  perfectly after Hans has worked on it.  But you'd get a rough count error so you would not be able to detach it and you'd have to  reboot.  And that's what this is a result of.  The rough counts on the topology, they were bugs so that when things went away, rough  counts weren't being decremented so you couldn't unload drivers.  Another piece for CAM that you need to know about is the async notifications.  And the async notifications are just a way to broadcast information or broadcast that  something has happened to different parts of CAM.  And this is a very integral part of many things in CAM.  It's used to enumerate the drives.  It's used to attach device drivers to those drives.  And it's used by the SIM to tell attached drivers that something interesting has happened.  Some devices need special handling after a SCSI bus reset, for example.  And when the SIM detects a SCSI bus reset happening, it's supposed to send a bus reset  asynchronous message.  One of the things that's interesting about CAM is relative to new bus is the model of  finding devices is a bit inverted.  The SIM, which you'd think would know about the devices, doesn't tell the rest of the  system the devices are there.  The rest of the system tells the SIM, hey, go scan for our devices.  And the transport layer finds the devices, gets all the information for them, and then  sends, hey, I found a device to all the drivers that are listening and they go do their thing.  I'll get into that a little bit more detail.  And new bus, in contrast, new bus would say, I'm in an attached routine.  I know what's going on.  I know what my children are.  Let me tell new bus about the children and then the children go off and probe and attach.  So it's a little different mindset.  Both of these systems were created contemporaneously.  And so each chose a slightly different way to model the devices.  Earlier when I was talking about what CAM did for the system, one of the things I said  was that it handled queuing.  Modern disk drives have a number of slots that you can put IOs into.  With SATA, they're tagged to tag queuing.  You have 32 slots.  You put the IO in, CAM's dev queue mechanism ensures that you never send more than 32 IOs  down to the SIM for the drive.  So you can't overflow it.  The SIM has to deal with it if you accidentally do or for some reason through error-pandling,  it winds up with more than it can handle.  But for the most part, we try to pace it so that we always have just the right number  of requests in the drive.  And this can be dynamically resizable.  At Netflix, even though our spinning disks have 32 slots, we finally get better performance  if we only use the first 16.  So at runtime, we set it to 16.  Other people use this to say, well, I've monitored the drive somehow.  I know it can handle eight right now.  And then later, I've monitored the drive some more, and it can handle 12.  And dynamically grow and shrink the number of commands that they allow.  In addition, the dev queue can be frozen for error recovery.  One of the interesting parts, unfortunately, that I won't get into too deeply today, is  in CAM, the dev queue freezes and dies for various parts of the system.  So when you want to submit IOs, but you know that the underlying layers aren't ready, you  can freeze the queue, submit IOs, and then when you know the device is ready, you unfreeze  it.  This can happen during startup, but it can also happen during error recovery.  The typical error recovery for most disk drives is cancel all the IOs that are present, do  something special for that one IO that failed, and then resubmit the IOs.  Well, the dev queue mechanism lets you freeze the dev queue so you have all the devices there.  So new commands will happen, and then you can go deal with the error recovery, and once  that's done, you can unfreeze it, and all the normal mechanisms for queuing and stuff  apply, and you go from there.  Yes?  When you have the queue paused, it's only a problem if IO needs to complete.  So if you have a drive that's failing, sorry, let me repeat the question.  The question was, are there error types where you have to worry about the queue filling  up?  And there are some errors, like if the hard drive is spinning or has a bad failing, I  mean, has a bad sector on it, and you have to keep retrying that one sector, or maybe  it's a group of sectors.  The dev queue will likely fill up with other requests that people want to do while you're  doing that error recovery.  Eventually, you need to terminate the error recovery, or the system becomes unresponsive,  because like most preemptive kernels, previous D really wants to be able to do IO to do anything  interesting.  So now that I've talked about the various bits that glue the Perif, the XBT, and the  SIM together, let me delve a little bit more deeply into those devices.  Chem perforals, like I said, implement some protocol.  They take block IO requests.  They take block IO requests and convert them to Schesri read requests, or say to read requests,  or something.  But in addition to that, they participate in device enumeration.  And I'll get into exactly how they do that over the next few slides.  So once the Perif driver has converted the IO request to the protocol blocks, the Schesri  protocol, NVMe protocol, whatever, it hands it off to the SIM via the XBT.  I said that earlier.  It's a very basic mechanism, so it bears repeating.  Once the SIM signals completion, it notifies up the stack that the IO is done.  The transport layer passes the messages back and forth between the Perif and the SIM, but  it also enumerates the drives, or the devices that are on a given SIM.  It will answer some of the CCBs that are asking about information about the transport, how  fast it is, and so forth.  It will answer those as well.  I'll get into the probe device that the transport creates to enumerate devices in the system  here in a little bit.  SIM drivers, first of all, the S and SIM does not scan for Schesri.  For a long time, I thought it did, but I went back and looked at the original CAM standards,  and they said system interface module from the beginning.  I thought that that was a later thinking under the feet when ATA was added, but that was  there from the start.  So it accepts the protocol blocks from the peripheral driver, writes them to the adapter,  and answers different questions about the drive that it has figured out.  So how does the SIM get created?  One of the mysteries to me when I was first starting out with this was, how do we get  one of these going in the system?  That attaches to the new bus device enumeration.  In your attach routine, typically, for new bus or in a routine that gets called in a deferred  manner, you create a dev queue with CAM-SIM-Q-ALOC.  You create the SIM, and you pass it a couple of interesting things.  You pass it the name, you pass it the dev queue you just created, and you also pass it an  action routine for receiving CCBs that it will deal with, and you give it a polling routine.  The polling routine generally is only used when we're writing crash dumps because we  don't have interrupts running, and we need some way to pull the SIM to see if the CCBs  are done yet.  After you create the SIM, you need to register each bus that the SIM supports.  These days, it's usually a one-for-one mapping.  In the past, it's been many to one.  You do that with XBTBus register, and it goes off, and it registers the bus and the global  list that CAM maintains.  That allows you to participate in device enumeration.  You then create a CAM path for the device enumeration with XBT Create Path.  One of the things that I had to go looking for when I was preparing this talk was where  does the XBT get created.  That's something that's handled automatically in bus register.  It associates the transport to the bus by asking the SIM what kind of bus it is.  It sends a path in core ACCB.  It finds out what type of bus it is.  It sets up the function pointers to do that.  In CAM right now, there's no per transport data.  So it's just a function pointer table that will map for all instances of a particular  transport type.  That's handled automatically behind the scenes.  You don't need to do anything special.  One of the areas that I had a lot of trouble understanding when I was working on the NDA  driver was how do the peripherals instantiate.  I thought I would spend a little bit of time in this talk going through that.  When the path is created by the SIM driver that I mentioned a couple slides back, it  sends out a path registered notification.  The XBT drivers that have registered for this, the probe devices for the XBT drivers get  a notification and they kick off the whole process.  The early part of boot when interrupts aren't enabled, the path registered get queued up  and get released later once interrupts are running and the devices can be discovered.  These path registered notifications get turned into XBT bus scan CCBs that go to the transport  layer which kicks off everything.  The transport layer runs a bit of a state machine to query the device.  I'll get to the state machine and what it is in a little bit.  Basically the transport knows how to ask all the devices on the system, all the common  information.  The SCSI transport knows how to do the inquiry command and maybe some auxiliary stuff to make  sure the device is there and fill in the internal data structures that the peripherals  will want to look at to see whether or not they want to claim this device.  Similarly for the ATA transport layer, it knows how to send, identify commands.  NVME knows how to get this information as well.  The probe devices receive this information.  Let me back up.  Once the XBT probe device has finished its state machine, it will send an asynchronous  message, AC found device.  All the device perf drivers on the system that are listening for things will then take  a look at the information that the transport has found and decide, yes, I can deal with  this or no, I can't.  In the example I gave earlier where I had an SSD attached to the pass device and the  DA device, both of those drivers said, yes, I can deal with this.  The pass through device created an instance of itself and the DA device created an instance  of itself so they could handle the stuff that they handle.  The reason I have the pass through device enabled in my system is the drives in that  system generally are from vendors that wish that send us firmware updates as we find problems  and the pass through device is what CAM control uses to update the device.  Otherwise I just read and write blocks off of it with the DA device.  Why is the probe a complicated state machine?  Primary reason is, first, we can't block the, when you make a CCB request, it can't block  waiting to complete.  The probe routine has to go, oh, I know that there's a device there.  I need to send an inquiry command in the case of SCSI.  Well, since the inquiry command, it has to return right away.  So it updates its state to a state that indicates it sent the inquiry command.  When it completes, it knows, oh, I was in this state where I sent an inquiry command,  I'll look at the inquiry command and based on the details of that, it will go to the  next state.  The state diagrams for the probe routines are a little bit complicated, so I'll go through  a simple one and show you what at least my attempt at drawing SCSI's resulted in.  So that basically leads us to the second reason that the state machines are complicated is  because the protocols are complicated.  If you're doing domain validation, you have to do particular things in a particular order  to make sure that you can get all the commands through.  If you want to support tag queuing, you have to do particular things in a particular order.  And the state machines deal with this.  It gets particularly complicated when you have enclosures and power management and stuff,  so you have to spin drives up or spin them down or turn enclosures on and off and do  stuff that Ken could probably tell you a lot about.  But I looked at that and thought that would make a bad example and went to the next one  to find a good example for a state machine.  So the one good example I found was the one I just wrote.  And part of the reason for this is that the NVMe Sim that I wrote, the NVMe driver has  callbacks for when it detects devices and so forth.  So it does all the state machine stuff behind the scenes and then sends a notification,  hey, I found it.  So I was able to leverage that and have just a couple of states.  We start off in the invalid state.  The first thing we do when we get a bus scan is we go to the identify state.  We look at the data that's present and do all the stuff to notify that devices are found  and wind up in the done state.  If at any time during this a reset comes in, we go to the reset state, which just then  immediately goes back to identify.  We could probably skip the reset state, but when I was writing this I didn't realize that  the reset state and the identify state are really the same state.  So I tried to draw the SCSI state machine.  It wasn't completely successful.  This is after...  I was about to add a disclaimer that I'm sure I'm missing at least one.  But do you know which one that is so I can add it?  But this shows the whole glory of it.  There's no good tool to do this.  But that talks about TUR as test unit ready.  It goes into inquiry.  We see if it has LUNs or not and report all the LUNs and VPDs and get the serial number  and do domain verification.  And eventually after you run the sausage mill backwards you've figured out that you've got  a device and you get to the done state.  But I invite you to go read the code if you'd like to A, fact check me, or B, understand  what's going on.  And there's something similar for ATA.  There are more states and I couldn't fit that on the slide.  So I'm not picking favorites here.  I just chose this one as an example.  So as I said earlier, when the Peruv driver is attaching, it gets the device found notification  that is sent to everything.  It has an async notifiter.  If you're looking at the NDA driver, it's NDA async.  It finds the device that it gets and goes, oh, and asks the transport, oh, what kind  of device is this?  And it looks at some details and said, yes, I can deal with that.  So then it creates a CAM peripheral with CAM-PRIF-ALIC and it passes a number of routines into that.  It passes a register routine, a start routine, and some more that I'll get to in a minute.  So the registry routine is like an attached routine in new bus.  It allocates the SASC, initializes the IOS scheduler.  There's a whole bunch of stuff.  I've listed them here and I want a little bit of time at the end for questions.  The interesting bits are that it registers an async event, async callback for interesting  events that might happen on that bus.  And then it calls XPT schedule to get things started.  Nothing happens in CAM without a trip through XPT schedule.  XPT schedule is what is the mechanism that the DevQ that I talked about earlier uses  to make sure that transaction slots are happening.  And I believe I have a slide on it a little bit later.  Anyway, here's a list of the required routines that you need to create a block device.  They're pretty standard.  Most of these come from the disk interface, which most people are familiar with.  The interesting one, there are two here that are not part of the disk interface, though,  that I'm going to call out.  One is start.  And that's called by XPT schedule when there's room for work.  And the other is done.  And you could have a different done routine for each request type.  Most of the drivers in the tree just have one.  And that gets called when the CCB has completed.  So XPT schedule checks to see if there's room in the DevQ.  If there is room, it allocates memory for the CCB and calls the peripheral start routine.  The peripheral start routine goes, oh, I have a CCB.  I can do work.  Let me check the, you know, any work queues that I have.  And in a couple of minutes, I'll get to where those work queues actually come from.  I have a nice picture here that I will talk to.  The other thing that XPT schedule does is if there's any shared resources that are limited.  So if I've got a SIM that can do five requests at a time and I've got 10 drives attached  to it, even though the drives can obviously do at least one IO at a time, it can only  ever schedule five IOs in that case because that's the limit of the SIM driver.  A lot of these limitations are for older hardware, maybe less relevant today.  I know Justin has advocated rewriting a lot of the scheduling to simplify it, to simplify  memory allocation.  And that discussion is an interesting discussion, but beyond the scope of this talk.  So when you're doing IO, you create the CCB and call XPT action, and that pushes it down  to the XPT or to the SIM.  And the SIM is done with a CCB.  It gets an interrupt, it sees that the IO is completed, it usually looks up in a mapping  table that says this cookie that I get an interrupt corresponds to the CCB so I can  complete it.  That calls XPT done.  XPT done also calls XPT schedule.  And that's, you can wind up with really weird stack traces, particularly if you have bugs  in your code, I'm told.  So before the callback, the done callback finishes, your start callback happens.  And that's so that we can overlap the IO in the SIM a little bit better.  The other thing it can do, if there's an error, it can freeze the queue and re-queue IO so  that they can be retried.  The strategy routine is like most normal strategy routines.  It gets BIOS from the upper layers and queues them using either the CAMIO scheduler or Bio  queue disk sort.  And then it calls XPT schedule to maybe go do that IO.  So in CAM, everything gets queued once and then it will get de-queued and executed.  That's part of the inefficiency that we'll need to address if we want to handle high  transaction rates for NVME.  It's one of the reasons I think Justin's idea of rewriting this layer is a useful one because  we can generally know if we can call the start routine immediately and bypass the strategy  perhaps.  Drew.  What is the granularity of the processes like maxViz and maxViz?  It's somewhere in between those two.  Drew asked, what's the granularity of these messages?  Is it maxViz?  Is it a single disk block?  It's a single transaction.  So usually, like for UFS, it might be 16K or 32K, if it's one of our systems doing a  big read ahead, it could be as large as maxViz.  So that's the size.  Does that answer your question?  Yeah, with 30 microseconds of latency, you wind up, if I'm doing the math right in my  head with 30,000 IOs a second before you run into problems.  But if you're doing 30,000 IOs a second, you're going to consume an entire core just to do  that and do the CAM processing.  In our systems, we're doing like 128K or 256K transactions.  So the transaction rate isn't a big concern for us.  Inefficiency is not a huge deal, although I know you'll want me to get rid of it because  every little bit counts.  This is the primary reason why Justin thinks that you don't need to call XBT scheduled  to schedule the IO, which queues and dequeues, you can do it directly from the strategy routine.  So the start routine, you have a CCB, you know you have a slot in the SIM to do IO,  so you fill it in and call XBT action and then call XBT schedule at the end to see if  you have another slot because sometimes multiple slots open up, multiple completions happen  before you can get back to the system, sometimes queues freeze and when they thaw you can get  a bunch of stuff going on.  This is also the place if you have complicated interaction between different types of IO.  In SATA IO, for example, you can send a trim down, but it doesn't participate in NCQ.  So either you send a trim down, you can only send one trim down, or you send your IO request  down that are NCQ and the start routine enforces that restriction.  Yes, there is a newer NCQ version of trim, but a lot of drives don't support that yet,  so we still need to have code to deal with that.  And then the done routine would be NDA done if you are looking at the source code and  it's called by the SIM as part of the XBT done processing.  It also calls XBT schedule to see if there is a slot available and like I said you can  get some interesting inversions because it calls it before the done routine, sorry, wrong  point.  The done routine sometimes also has to call XBT schedule because of the IO restrictions.  If I am doing a trim in SATA and it completes, I have to call XBT schedule because I want  to unleash any read and write IOs that have come in in the interim and calling XBT schedule  will kick it off.  Everything gets clocked through XBT schedule.  If you are riding a Perif driver and it's hanging, it never hurts except for performance  sometimes to call XBT schedule.  There were several bugs that I had when I was riding in DA.  I guess I do know something about bugs.  There were several bugs that I had in the DA where I wasn't calling XBT schedule.  The IO would happen, I would see that was happening and then it would never complete  and another one would never start.  That was because I had left out calls to XBT schedule.  Here is a picture of everything that I have been talking about where an IO comes in at  the top and the middle of this into NDA strategy.  It cues the bio to the disk queue and calls NDA schedule.  NDA schedule is a wrapper around XBT schedule.  If there are slots in the dev queue, XBT schedule calls NDA start which then calls bio queue  first to get the bio off of the queue.  The bio to CCB translation happens here as we pass it down and then it goes all the way  down into hardware and comes back up and we get back to our done routine which calls NDA  schedule.  Then to make more things happen or it calls bio done to complete the IO up the storage  stack.  When you are writing a SIM, there are three routines that you need to write.  SIM action is there to handle the CCBs.  I have talked about that.  Again there are the two types of CCBs, either the ones you queued to the device.  It will take a while to happen or things you can immediately tell the XBT or the PRIF about  the device.  SIM pull is called when you have got a CCB outstanding in the dump routines.  It is only used in dump although it could potentially be used elsewhere.  Right now we only use it when we are dumping the system.  When the system panics and we are taking a dump, we disable interrupts.  I put that in for my nine-year-old son.  When the system is writing out a crash dump, interrupts are disabled and we need some way  to find that the IO is completed so we can write the next set of blocks.  The SEMI RQ is what completes everything up the stack and calls XBT done.  The key points, I hope people take away from this.  These were some of the harder things that I had to learn.  Although the first one was easy to learn once I realized.  XBT means transport.  SIM scans the bus for devices either when it starts or part of a path registered.  The XBT probes the device by creating a special probe device or probes the information about  the device, creating a special probe device.  These probe devices go through a state machine to get all the information and then tell the  PRIF drivers with a AC device found routine.  There are many PRIF drivers that claim a single physical instance of hardware, which is new  bus can't do that.  Can can.  And CCBs drive everything.  If you don't know about CCBs, you're going to have a really rough time understanding  the CAM code.  And so I put this picture in to contrast with the picture at the beginning that had in the  middle NVD and NVMe.  This is what the IO stack looks like with the NDA peripheral where we've got all the  – dang it, my slide's wrong.  This NDA XBT should be NVMe XBT.  But the layering is correct.  So this is the different layering when you enable the NDA driver.  I thought it might be interesting to, again, have another picture to see what's going  on with this.  And with that, I'd like to take any questions people have.  Drew?  Yes.  I'm glad you asked that.  The person you're sitting next to, Scott Long, maybe you know each other.  He's been working with me, mostly him, though, on multi-Q extensions for CAM so that instead  of having a SIMQ or a DevQ, you can have one per CPU or one per CPU complex or whatever  makes sense.  And in that case, then all of the IO wouldn't funnel through one data structure, which we  all know from high-core count experience is bad for performance.  Each one could get their own and they wouldn't contend on any locks to manipulate that data  structure.  So that's a very good thing.  And hopefully next year, Scott will present his talk on CAM multi-Q after doing the work  and getting some phenomenal results.  Okay, so when I put disk sort up, that was an example of how you could schedule the IO.  The disk sort does the elevator algorithm, which you really need to do for spinning rust,  but for flash, no.  You just enqueue it as the first one or the last one and then take it off.  Right, and the DA, ADA and NDA drivers, the NDA driver doesn't do any disk sort at all.  The other two, if it's got a spinning rate of zero, which means it's an SSD, it doesn't  do any disk sort either.  So George.  I started looking at some of the CAMs and I just pointed out that it's almost all the  working CAMs done at the boundary of those.  Is that even going to be quite decreased without having to add statically defined points?  Yes.  Okay, sorry, my mistake.  George was asking if all of the, he was looking at the CAM routines and noticed that a lot  of the work seems to be done at the boundary points.  And the answer is, and wanted to know if we could exploit that to make it easier for instrumenting  it with Dtrace.  And the answer is yes.  CAM has, I didn't talk about it today, but CAM has a tracing routine that you can go  through and see the CCBs going down and coming back up.  And a lot of the code that's CAM, debug, blah, blah, blah, would be excellent candidates  for Dtrace.  Most of it would be in CAMXPT.C, although you would probably need a couple in each of  the individual transports, the ATAXPT and the NVMEXPT as well to make that happen.  But it would not be a ton of work to make that happen.  Right now, so let me repeat the question.  Alexander wanted to know what the overhead of the NVD versus NDA.  Right now, NVD does things unmapped and the NDA driver still has a bug where it thinks  it can't do unmapped IO.  So with NVD, we see maybe 85 gigabytes a second, sorry, gigabytes a second in our 100G machines  and with NDA, what were the best numbers?  65, 75?  Right, right.  So because of contention in the VM layer, we would get large latency spikes.  So the control plane would slow down traffic to our system, so it was hard to see what  the best peak you could get.  But right now, there's a little bit of a performance difference because it doesn't  deal unmapped IO.  And the 80 gigabits is off of four drives, four lanes, PCIe4, so it's pretty close to  line rate.  Yes, Hans?  I had a question about the state machine when it was right in this system.  So it's a banking column and callback driven state machine?  Okay, so yes.  So the question was, is the state machine asynchronous, we callback driven?  And the answer is yes.  And I have you thought about making it synchronous like that, once that's our enumeration, do  you simplify it?  You don't leak reference time?  I don't know if anybody's thought of doing that.  One of the things with CAM is all of this happens in parallel anyway.  We can start this device and this device and this device and send all those requests down  to the SIM.  And as they complete, it goes to the next one, so we don't unlike Nubus, which has one  synchronous thread to enumerate everything and you have to wait for the next thing to  come in.  CAM's enumeration model allows all of that to proceed in parallel across even multiple  SIMs and drives and buses and everything that can happen in parallel so that you don't have  to wait for that.  Yes, Justin, the other Justin.  Hello, wieder met, and I'll see you guys in a bit.  innocent, more than innocent, when errors are typically all over the public that have been hidden,  that's innocent. But I thought, you know, that this there can't be those about to increase the community to the other side.  So, to briefly repeat the question, when HCI completes a CCB, it marks it as innocent?  It marks it as an error of the CCB, so there's an error of CCB that says it marks all the others as innocent.  Oh, right, okay, so when the HCI is completing the IO, if it gets an error, it marks all the others as async,  because when an error does occur with SATA, automatically all the other requests are considered to have not happened,  and will return errors back, and so that reflects that. I think I'll have to talk with you afterwards for the details,  or you can talk with Alexander, who wrote that code originally. He's sitting two people down from you to resolve that.  But I think that's a little bit more detail than I can go into up here.  And so, last question. Yes.  Okay, so the question was, can you print the CAM tree similar to the JOM methods that we have?  And there's CAM control dev list, which will list the devices, but I don't think we have anything that will print that in tree form.  Ken or Scott or Justin, do we?  Just to repeat your answer, since you're not mic'd, Ken said no, there's not.  And he's doubtful that there would be additional useful information apart from the linear list, because CAM tends to be bus,  here's the things on the bus, and there's not much more of a tree beyond that.  And you would need to dig down into the specific details of Fiber Channel or SCSI to find more complicated topologies to report back,  or to look at enclosures and that kind of thing.  Okay, I think we're done with time, so if you have any other questions, come see me. Thank you very much.
A_Walkthrough_of_CAM_eUSNea3I-hE_0_47.png A_Walkthrough_of_CAM_eUSNea3I-hE_48_95.png A_Walkthrough_of_CAM_eUSNea3I-hE_96_143.png A_Walkthrough_of_CAM_eUSNea3I-hE_144_1007.png A_Walkthrough_of_CAM_eUSNea3I-hE_1008_1439.png A_Walkthrough_of_CAM_eUSNea3I-hE_1440_3767.png A_Walkthrough_of_CAM_eUSNea3I-hE_3768_4571.png A_Walkthrough_of_CAM_eUSNea3I-hE_4572_6263.png A_Walkthrough_of_CAM_eUSNea3I-hE_6264_9155.png A_Walkthrough_of_CAM_eUSNea3I-hE_9156_11771.png A_Walkthrough_of_CAM_eUSNea3I-hE_11772_12551.png A_Walkthrough_of_CAM_eUSNea3I-hE_12552_12611.png A_Walkthrough_of_CAM_eUSNea3I-hE_12612_13655.png A_Walkthrough_of_CAM_eUSNea3I-hE_13656_16271.png A_Walkthrough_of_CAM_eUSNea3I-hE_16272_17447.png A_Walkthrough_of_CAM_eUSNea3I-hE_17448_20003.png A_Walkthrough_of_CAM_eUSNea3I-hE_20004_23195.png A_Walkthrough_of_CAM_eUSNea3I-hE_23196_26855.png A_Walkthrough_of_CAM_eUSNea3I-hE_26856_27251.png A_Walkthrough_of_CAM_eUSNea3I-hE_27252_27299.png A_Walkthrough_of_CAM_eUSNea3I-hE_27300_28307.png A_Walkthrough_of_CAM_eUSNea3I-hE_28308_29171.png A_Walkthrough_of_CAM_eUSNea3I-hE_29172_29519.png A_Walkthrough_of_CAM_eUSNea3I-hE_29520_29627.png A_Walkthrough_of_CAM_eUSNea3I-hE_29628_29987.png A_Walkthrough_of_CAM_eUSNea3I-hE_29988_33215.png A_Walkthrough_of_CAM_eUSNea3I-hE_33216_36827.png A_Walkthrough_of_CAM_eUSNea3I-hE_36828_40487.png A_Walkthrough_of_CAM_eUSNea3I-hE_40488_41543.png A_Walkthrough_of_CAM_eUSNea3I-hE_41544_42647.png A_Walkthrough_of_CAM_eUSNea3I-hE_42648_45371.png A_Walkthrough_of_CAM_eUSNea3I-hE_45372_46823.png A_Walkthrough_of_CAM_eUSNea3I-hE_46824_52559.png A_Walkthrough_of_CAM_eUSNea3I-hE_52560_55223.png A_Walkthrough_of_CAM_eUSNea3I-hE_55224_57155.png A_Walkthrough_of_CAM_eUSNea3I-hE_57156_59087.png A_Walkthrough_of_CAM_eUSNea3I-hE_59088_61643.png A_Walkthrough_of_CAM_eUSNea3I-hE_61644_62819.png A_Walkthrough_of_CAM_eUSNea3I-hE_62820_64871.png A_Walkthrough_of_CAM_eUSNea3I-hE_64872_65207.png A_Walkthrough_of_CAM_eUSNea3I-hE_65208_66695.png A_Walkthrough_of_CAM_eUSNea3I-hE_66696_70643.png A_Walkthrough_of_CAM_eUSNea3I-hE_70644_72395.png A_Walkthrough_of_CAM_eUSNea3I-hE_72396_74735.png A_Walkthrough_of_CAM_eUSNea3I-hE_74736_76367.png A_Walkthrough_of_CAM_eUSNea3I-hE_76368_76559.png A_Walkthrough_of_CAM_eUSNea3I-hE_76560_77195.png A_Walkthrough_of_CAM_eUSNea3I-hE_77196_78251.png A_Walkthrough_of_CAM_eUSNea3I-hE_78252_78431.png A_Walkthrough_of_CAM_eUSNea3I-hE_78432_80111.png A_Walkthrough_of_CAM_eUSNea3I-hE_80112_81227.png A_Walkthrough_of_CAM_eUSNea3I-hE_81228_88727.png A_Walkthrough_of_CAM_eUSNea3I-hE_88728_94283.png A_Walkthrough_of_CAM_eUSNea3I-hE_94284_94559.png A_Walkthrough_of_CAM_eUSNea3I-hE_94560_94607.png A_Walkthrough_of_CAM_eUSNea3I-hE_94608_94667.png A_Walkthrough_of_CAM_eUSNea3I-hE_94668_94775.png A_Walkthrough_of_CAM_eUSNea3I-hE_94776_94931.png A_Walkthrough_of_CAM_eUSNea3I-hE_94932_94979.png A_Walkthrough_of_CAM_eUSNea3I-hE_94980_96786.png