BSD/ARM Kernel Internals
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Thomas, Arun
In this talk, I'll discuss how BSD kernels interface with the ARM processor. I will cover the kernel internals of the FreeBSD and NetBSD ARM ports, focusing on ARMv7 primarily. I will discuss how booting, memory management, exceptions, and interrupts work using plenty of BSD code. This talk is meant to be a quick start guide for BSD hackers who aren't familiar with the ARM architecture.
 Turn the light on. Okay, we're good. We're good? Yeah. Yeah, yeah. Here we go. Sorry about that. So, is it okay if I get six minutes over? Should I try to finish at 5.30 or is it okay if I go to 5.36? Okay. Cool. Thanks. Thanks. You got both mics on? Yeah, I haven't turned this one on yet. They're both on? Yeah, I just turned it on. It's on now. Okay, we'll just go grab some so we can start recording. Okay, sure. Okay, cool. Is it on? I don't know. Okay. Okay. Hi, okay. So, we've resolved the recording stuff, so we're good to go. So, my name is Arun Thomas and I'm going to talk about BSD ARM kernel internals. This is my second BSD conference. I gave a talk at EuroBSDCon 2011 a few years ago and so it's good to be back at another BSD conference. They're always a good time. So, let's talk about a BSD ARM. So, I'm going to start off with a little bit of a quick demo. So, this is a previous BSD current running I guess a few weeks ago when I built it and a little bit of debug code. And so, you can see it booting. There's a mix of debug code and witnesses on, so it's going to be a little bit slower. So, you'll see that there was, and I have some screenshots later that'll kind of, you can actually see what's going on. But there were a couple versions of Uboot that ran. Uboot loader is running now and then it's going to load the kernel. So, I'll take a couple seconds. Okay, here we go. Yes, I want to start now. So, you've got all this kernel code. It's booting up, it's doing some device stuff. And then, shortly it'll boot up the rest of the way. So, this is kind of, I'm going to kind of go through the flow exactly what's going on. So, to get to this point there was a lot of like machine-dependent stuff that needed to happen. And so, here we go. You're in user land and stuff's happening around the file systems and pretty soon you get to root prompt. So, I'm running it off of Beaglebone Black. This is the board here. It's pretty cheap. It's a nice platform. It's supported by all the BSTs that have ARM ports. And so, yeah, it'll keep going. And so, this is the black. I have the white tube. So, we're looking at an SSH session from your laptop. No, no, this is just, I plugged in the serial. There's a serial cable going on. So, Adafruit has these handy little cables you can just plug in. So, just plug in root. Password, you get the free BST prompt. So, this uses UBoot. So, free BST, I think everyone uses UBoot on ARM boards basically. So, that's that. So, that's the basic process and then I'm going to kind of go into like what happens when you're booting up. I'm going to focus on the machine-dependent parts, the machine-independent parts that I don't really talk about at all, but we'll look at some of the code and see what happens. So, all right. So, these are a couple of those screenshots so you can kind of see what happens. So, there's a couple versions of UBoot running. That's one. That's another one. And then, the free BST UBoot loader runs. And then, there's something here about a DTB. It's a device tree blob. I'll talk about what that is. And then, you get the copyright notice. Before that happens, there's a whole bunch of machine-dependent stuff that needs to happen before you get to this. And I'll talk about all of that stuff that happens. So, you can tell that it's free BST, 11 current. And it's got a weird revision because I'm using git and it's dirty, so because I had some local hacks. Postname is imaginatively beasty. And it's the Beaglebone kernel right there. And I'm using crochet. It's just who I'll talk about a little bit later to build this image. And it's kind of cool for BSTs built with Clang. And this is running in Cortex-A8. I'll talk a little bit about the other processors that ARM has, all the Cortex processors. Here are some of the features. And then, it talks a little bit about caches. And then, this is the system on-ship, the SOC, the TI AM-3358. And we'll talk a little bit about SOCs and all that stuff, so if you're not familiar with ARM. And then, there's some more stuff about device tree. And this is basically how the UART gets mapped. So, this is the memory range. This is the IRQ. And we'll go into all of this. So, the goal of this talk is to get you hacking BST on ARM. So, are there any people here who haven't done any hacking on any ARM hacking or have done little hacking? Raise your hands. Excellent. So, my goal is to get all of you, like, at least interested in hacking ARM. So, that's my goal for this talk. So, we'll start off with the ARM 101. So, we'll talk a little bit about the basics of the ARM architecture, what the assembly looks like, what the system level stuff happens, looks like, various boards that you could buy, the SOCs, all that stuff. So, just like a quick bootstrap to the ARM architecture. Then, we'll go through some of the kernel code. We'll look at NetBSD and FreeBSD. I just picked those because those are the ones I've hacked. But, after watching some of the OpenBSD MIPS talks, I'm hoping to get OpenBSD running on ARM. Maybe in an updated version of this talk, I'll add some OpenBSD stuff. So, we'll focus on the machine-dependent stuff. I won't look at the machine-independent stuff, as I mentioned earlier. So, we'll look at how the kernel boots up and does some exception handling and sets up the initial page table and all of that stuff. It'll still be kind of high-level, but you can at least, I'll give you at least kind of like the files you want to look at. It's sort of high-level, so you can dig into more detail later. Then, there's a short section on tips for hacking on BSDRM and debugging and more resources, so you can dig in further. I won't be able to get into everything, but there's a lot of good material out there that you can look into. So, there we go. Okay, so the ARM architecture. Hugely popular and embedded systems, you probably own several, each of you. It's also moving into general-purpose computing. So, you've got laptops, Sops netbooks. Some of the Chromebooks have ARM chips in them. They're pretty nice, actually. There's servers. HP was making ARM servers at one point. It's also moving into high-performance computing. GPGPU and Vida have some pretty cool SOCs coming out. And the main reason for this is power efficiency. So, it's a big push. So, ARM has an interesting business model. It doesn't manufacture chips. It basically licensed the architecture and designs to various silicon vendors like TI and Samsung and all that stuff. So, they basically just come up with the architecture and some of the processor designs, someone else will package it up into SOC and fabricate it. Okay, so ARM architecture. So, ARM stands for Advanced Risk Machine. It used to be a corn risk machine. And since it's a risk machine, it's reduced instructions like computer, fewer instructions, simpler instructions. Because it's risk, it's load-store architecture. So, if you want to operate on memory, you have to load it from memory into register, operate it on there, and then store it back out to memory, unlike, say, x86, which is not very risky. So, it's big Indian or little Indian. Little Indian is more common. So, the current versions, there's several versions of the ISA. The current version is ARMv7 and ARMv8. So, ARMv7 is 32-bit and ARMv8 is 64-bit. It's ARM, it does some cool stuff in ARMv8. They simplify the architecture. But I won't be talking about that at all in this talk. So, I'm going to focus purely on ARMv7 and 32-bit. So, each of the ISAs also has various architecture profiles. So, there's the application profile, the real-time profile, and the microcontroller profile. I'm only going to talk about the application profile. So, the real-time microcontroller profiles are more for embedded systems, and they don't have full MMU support. So, as I said, ARMv7a is what we'll talk about. 32-bit ARM processors. They're called Cortex-A, if you're looking at models. They have full MMU support, and if you look at the ARM documents, they're designed for full feature operating systems. So, things like BSD and iOS and stuff like that, not really the kind of like embedded operating systems. So, the ARMv7a has two instruction sets, actually. So, there's the ARM instruction set, and there's the thumb instruction set. So, the ARM instruction set is bigger than the thumb instruction set. So, the thumb instruction set has a mix of 16 and 32-bit instructions, where ARM is just 32-bit. So, the 32-bit instructions were added with the thumb 2, and they added thumb 2 technology. So, the good thing about thumb is that you get better code density, and that's good for your caches. So, all the ARM CPUs are packaged up with other logic into a system on ship or SOC. So, you'll hear this SOC acronym a lot. So, what's in an SOC? So, you have your interrupt controllers, your timers, your UARTs, SDMMC controllers, SATA controllers, USB controllers, GPUs, and all kinds of peripherals. So, you might have a camera controller, or GPS controller, all kinds of stuff. All that stuff that goes into your phone is going to be on the SOC. So, the SOC is actually getting better for the ARM developers. So, ARM is actually kind of standardized some of these things, like the interrupt controller and the timers. Before, each SOC vendor would basically create their own set of timers and their own set of interrupt controllers. So, they had kind of hard for OS developers, since they had to write all these new drivers. But now ARM has this generic interrupt controller and generic timers, so it makes it a little bit easier. ARM is kind of building a platform. Okay. So, this is what an SOC looks like. This is the AM335X SOC. This is the SOC that's in the Beaglebone black. So, you see that the core is actually a small part of the SOC. There's a lot of other stuff in there. So, you've got a GPU here, and you've got a whole bunch of various buses. You've got the UR at SPI, I2C stuff, timers, real-time clock, ACs, and then you've got your GPIOs. So, there's a whole bunch of logic on there, including USB and an EMAC. So, the SOC has a whole bunch of stuff in addition to the core. So, as I mentioned, I'm using Beaglebone black. It can be yours for the low-low price of $45 US. It's a popular hobbyist board, and it's supported by FreeBSD, NetBSD, and OpenBSD. Dragonfly, I don't think, has an ARM port as far as I know, but I imagine if it did, it would support it as well. So, it's kind of a cool little board, as a lot of IO. Another popular board is the Beagleboard XM. It's a little more expensive. It's supported by FreeBSD in a branch for Google Summer Code. I don't think it was integrated back, but may sometime. NetBSD supports it. OpenBSD supports it. And the nice thing about this board is, if you don't want to buy it, QMU, Linara's version of QMU actually has a support for it. So, you could just boot it up in a virtual machine. So, that's kind of cool. Okay, so this is a bunch of different SOCs, and these are all supported by, and the boards that are associated with them. And these are all supported by one of the BSDs. So, the top is the OMAP3, OMAP4, the DaVinci, and Sitara. They're all from Texas Instruments, and they're kind of a family of SOCs. It's fairly similar. So, you've got the Beagleboard, and the Beagleboard, and there's a couple versions of the Beagleboard. There's a couple versions of the Beaglebone, like, Sean over there was talking about the Beaglebone white versus black. I have the black here. And Pandaboard has a couple versions, too. So, these are really popular developer boards. Then you have the Allwinner A10 and A20 that's used in the QB board and the QB truck, also a fairly popular board, especially now. The FreeScale IMX6 is in the WAN board. The Samsung X10S5 is in the Chromebook and the R&Dale board. That's a really high-end SOC with a really high-end R&P processors on them. So, the last one's actually really interesting. So, XilinXinq has these Cortex chips packaged up with some FPGA logic. So, if you want to do any hardware design, it's a pretty cool platform. So, these boards range from 45 to, like, 400, 500. I forget exactly. The Z board is the most expensive board. The MicroZet, I think, is fairly cheap. So, if you want to play around with that, it's pretty cool. So, this is all the Cortex CPUs. So, on the low end, you have the Cortex A5, the FreeScale Vibrid, as an SOC that supports the A5. That's for really kind of, like, embedded stuff. The Cortex A8 is more popular. You'll see that in most of these developer boards that are out. So, the OMAP 3 SOC, DaVinci, Sitara, so your BeagleBoards, BeagleBones, have this chip in there. So, it's the lower end. And they all win an A10. So, the Cortex A9 is sort of the mid-range CPU. You'll find in the OMAP 4, the FreeScale MX6, and the XilinXinq. So, this is what's in the panda board. The Cortex A15 is the high-end. As I mentioned, it's in the XNOS 5. You'll find it in a lot of the high-end phones and in the Chromebook. So, the A7 is actually a replacement for the A8. So, it's found in the XNOS 5 in the old winner, A20. So, it'll eventually, like, fully replace it. And the A12 and the A17 are mid-range CPUs. So, they'll replace the A9 over time. And I don't know what SOCs they're in, but I'm sure they'll be in a lot of SOCs, maybe some of you know. So, it's sort of, there's a lot of CPUs, so it's hard to kind of keep track of them all, and typically the one you'll see a lot is the A8 and A9 for the developer boards that you'll see. So, now that we've talked a little bit about the boards and the hardware, I'll talk a little bit about software. So, ARM has several versions of ABI's, and so an ABI is an application-finer interface. So, the ARM docs, the quote is that they are, rules that an ARM executable must adhere to. So, things like executable formats, calling conventions, alignment, what system calls look like, all that kind of stuff is set aside in the ABI. So, there's several ARM ABI's. There's the ARM EABI, the ARM Embedded ABI, and there's the ARM EABI HF, that's the ARM Embedded ABI with hardware floating point. So, those are kind of like the current ABI's. There's also the ARM OABI, and I don't actually know what it stands for, maybe old, I don't know, but it's sort of obsolete, but it's used for older versions of ARM. Does anybody know what OABI stands for? No? Okay. Just wondering. So, NEPI and FREME-AC both support EABI and EABI HF. So, when you're building this stuff, you want to make sure you have the right tool chain. So, you want a tool chain that's built for EABI or EABI HF, depending on what you want. Okay, so let's look a little bit into what the architecture looks like. So, the ARM has 16 general purpose registers. Some of them are used for other things or have kind of dedicated uses. So, the frame pointer, R11 is the frame pointer, R13 is the stack pointer, R14 is the link register. So, when you do a call instruction, the link register will save your current PC, so you have something to go back to, so that's the link register. And the program counter is R15. So, ARM also has some program status registers. There's also a floating point, it's called VFP instructions, and SIMD, that's the neon instruction set. I won't really talk about those, but if you're doing like, high vector code or GPU code or that kind of stuff, it'll be useful to you. So, in terms of this, ARM has these two program status registers that are fairly important when you're doing systems code. So, there's the current program status register, the CPSR, the saved program status register, the CPSR, which is used for exceptions. And it holds a number of important bits, like the processor mode, for instance, SVC is a mode, and we'll talk about what the different modes are soon. The interrupt mask bits, so if you want to disable interrupts, you'll set this IRQ bit. ARM versus thumb state, and INS is a big IND, a little IND, and various condition flags. So, here's what the assembly syntax looks like. We'll just add two numbers together, we'll add one plus two. So, you move one into R1, you move two into R2, then you add R1 and R2 and put in R3. So, the destination is on the left. So, that's a basic intro to data operations on ARM. So, the memory instructions, as I mentioned, it's a load store architecture, so if you're going to work on stuff or memory, you have to use the LDR, the load, and the store, STR instruction. So, you want to move what R1 points to into R0, and you want to store R0 into what R1 points to. There's also push and pop instructions that'll push multiple registers onto the stack, and pop multiple registers off the stack. So, these are actually aliases for ARM, has these instructions, LDM, load multiple, and STM, store multiple. They're the same thing if you have the right suffix and whatnot. And control flow. So, ARM, of course, has control flow. So, this is branch of 0, so bz loop, suspect. And the call instruction, as I mentioned, is the bl branch link instruction, and so this will label, and so this will save the current PC to the link register. If you're going to do a return, it's a bxlr, so branch exchange of the link register. I'll just jump back to whatever's in the LR, and in older versions of the ISA, this is how you do it. You just move the LR into the PC. That's, I think, deprecated in ARM v7, but it still works. They just don't recommend it, especially if you're doing thumb stuff. Okay, so that's kind of like the application level stuff. So, let's look into the OS relevant bits, because that's what we're doing. We're trying to get the OS going. So, there's actually more than this, but the two privileges I would care about are PL0 and PL1. So, PL0's unprivileged user code, PL1 is privileged kernel code, which is what it's used for. And there are nine operating modes, so there's one unprivileged user mode that runs at PL0, and there's eight privileged modes that we'll talk about, and they run at PL1 and above. So, the privileged modes are used mostly for exception in interrupt handling. So, it's a little bit complicated. ARM v8 cleans some of the stuff up, but if you're going to be doing kernel programming, especially at this level, it's good to know how this stuff works. So, here are the various modes. There's the supervisor mode, which handles, which sees for system calls, and it's also the initial mode that the processor starts up in. There's interrupt mode for normal interrupts, fast interrupt mode for higher priority interrupts, faster as well as you might expect from the name. There's abort mode for memory faults, undefined mode for illegal instructions. You can also use this to emulate instructions and software. System mode, which is a privileged mode for user mode registers. It doesn't really get used that much. Hypervisor mode for VMM support and monitor mode for trust zone. The modes that you really, that we'll really care about are supervisor mode, interrupt mode, abort mode, and undefined mode. Those are the ones that typically get used. So, there's one kind of interesting feature of the architecture. There's a thing called banked registers, and it's a little bit complicated, but it's good to know about. So, most registers are shared across all the modes. So, you have one PC that applies for all the modes. But there are some registers that are dedicated for each mode. So, you can have, and so there are duplicate registers for each of these modes. So, there's a separate stack for each mode, so you're going to want to have separate stack pointers. So, there's a stack pointer for user mode, a stack pointer for a supervisor mode, for instance. And the CPU will automatically set the appropriate banked register, depending on what mode you're in. So, in user mode, the CPU knows that you want the user mode stack pointer. In SVC mode, the CPU knows you want the SVC stack pointer. So, why do they add this stuff? So, it's important for exception handling. So, basically, we wonder how do we get back to the faulting instruction. So, we need to do that. We need to save the state at the time of the exception. So, the banked link register, so usually the link register saves the PC when you do a call. Here, it will save the program counter at exception time. And the saved program status register will save the current program status register at exception time. So, it basically saves your state. So, for instance, if you're doing a system call, LRSVC and SPRSVC, the link register and SVC mode, and the SPSR register and SVC mode, save the program counter and the current program status at the time of the SVC exception. So, from the user code. So, typically, when the processor changes modes, it will do that kind of automatically on an exception. So, it will switch to interrupt triggers. It will switch to IRQ mode on SVC instructions, also called SWI, or software interrupt. If you look through the BSE code, this is actually the name that gets used. But if you look at the new ARM docs, they tell you everything uses SVC, because I guess that's sort of the new name for the supervisor call or system call instruction. So, the SVC instruction will switch to SVC mode. So, if it's basically how it works, each exception, you go to the recording mode. So, the OS can also change the mode using the privileged CPS instruction. So, that's the change processor state instruction. So, the CPS instruction basically what it does is it modifies the mode field in the CPSR. So, this is basically, you'll find this when we talk a little bit later about what the instruction, the kind of early boot code looks like in FreeBSD and at BSD. It'll do this. So, this is how you switch to SVC mode if you want to use the CPS instruction. And if you want to switch to SVC mode and disable interrupts, this is the SVC mode. And you use the suffix ID, which is interrupt disable, and you tell which bits, the IRQ bit and the fast IRQ bit. So, back to the status registers. So, if you'll recall these hold the current mode, the interrupt disable and endiness. This is how you read and write them. So, this is if you want to read the CPSR and if you want to write the SPSR, it looks like this. The instructions are a little bit confusing. So, it's move to register from status is read, and then write is move to status from register. So, these are in the docs and you'll get used to them. But there is a gotcha here and actually this was fixed in FreeBSD a couple months ago. So, there's if you don't give the FSXC suffix, which basically lays out each of the parts of the status to write, you might not the compiler or the assembler won't necessarily do the right thing. So, there's an suffix underscore all and it doesn't actually mean all. So, depending on the version of gas you use, bad things can happen. So, there was this bug with the wrong Indian register restore bug and so this was because all didn't really mean all. So, if you're interested you can look into the revision ID, but it was kind of an interesting bug that was fixed and I think it took a long time for it because in LaPore it fixed it. So, tricky. So, let's look over view ARM Virtual Memory. It's 32 bit address on ARM v7. ARM v8 of course is 64 bit and if you use LPA it's actually 40 bits, but we'll talk about the 32 bit ARM v7 stuff. So, with that you get a 4 gigabyte virtual address space and there's paging support, two levels of paged tables. The TLBs are hardly managed so the MMU will do the paged table walk and the TLB miss. The commonly used page sizes are 4KB small page sizes and one megabyte sections. And if you want to know more, a lot more about ARM Virtual Memory there's a talk about transparency for pages tomorrow. So, you should go check that out. You'll learn a lot more about virtual memory because I'm just covering the basics. Okay, so the other kind of key thing in the architecture is coprocessor 15. So, if you're doing kernel hacking you're going to have to look at this. So, it's a system control coprocessor. Coprocessor is a little bit of a misleading term because it's actually an integral part of the architecture. You can't really take it out, but so it's heavily used for systems programming and it's used for things like setting up the processes page table. So, this is how you write the page table to the translation table base register. So, it's an MCR instruction, move to coprocessor from register and it's a little, I don't know, it doesn't, you have to read the docs to figure out exactly like which numbers to put in there and stuff. So, usually these things are wrapped up in inline assembly so that makes it a little bit easier. The other important thing this does is it holds the system control register. So, the system control register allows you to enable the M.U., the branch predictor, caching and it tells you where to put the exception. It allows the OS to tell the CPU where the exception vectors will go. So, yeah, reading, MRC, writing, MCR, so move to coprocessor from register and yeah, this stuff is in the docs but it takes a little getting used to. I kept screwing it up, but maybe you guys will be faster. So, this is probably the most important slide in the whole presentation. So, this is where you go to get more information about ARM. So, I'm just going to kind of like touch some of the basics, this is where you go. So, yeah, this is a quote from the NetBSD code and I'll read it out. It's, and thus spake the ARM ARM. So, what's the ARM ARM? It's the ARM architecture reference manual. And so, that is all the details about the instructions that architecture. It's 2,000 pages, or almost 2,000 pages. So, you're probably not going to read it cover to cover, but it has, it'll answer all the questions you have about the ARM architecture. So, you want the ARM V7A and ARM V7R edition of that. If you want kind of a quicker, kind of a lighter introduction to ARM, the ARM Cortex-A series programmers guide is really nice. It came out a couple years ago and they just updated it for this year. And the latest revision, they really updated and expanded a lot of the discussion. So, you should, if you're interested in ARM, you should download that like now. Both of these are free and it does require registration on the ARM website, but they're free. So, the other thing, the other resource that's good is the ARM System Developers Guide. This is actually a printed book and it's from 2004, so it's about a decade ago. And ARM moves pretty fast. So, it's definitely dated, but if you want to really dig into like, kind of like the systems level kernel stuff, like exactly exceptions work and some of that stuff, it's a good resource. In addition to the core ARM docs, you also want to look at the manuals for your processor, your SOC, and your board. So, for the Beagle board, it's a Cortex-A8. You want to look at the Cortex-A8 TRM, for the, sorry, the Beaglebone as well. And then for the Beaglebone, you want to look at the AM335X TRM. And then you want to look at the system reference manual for that. And it'll give you stuff basically like the device map. So, for the Beaglebone black, these are some of the key addresses that you need to know. 440900 is where the UART is. You write to the address, you can write to the UART. The interrupt controller is at this address, the timer, DM timer 1's clock is at this address, and the RAM's at this address. So, you want to look at your TRM to figure out exactly how to set up your board. ARM also has several migration guides. So, for the folks who are coming from Xe6 or MIPS or Power, these are really good guides. And even if you're just kind of interested in computer architecture, this is kind of cool. So, I saw some MIPS talks today, so I'm thinking about reading the MIPS to ARM guide, except I'm going to go from, maybe from ARM to MIPS as well. So, the IA32 guide has, like, for instance, it tells you about Agaccio, with the fact that characters are unsigned by default, and I can call this problem. There's a lot of useful tips in there. Okay. So, that's kind of a quick introduction to the ARM architecture. There's a lot of material, but as I said, there's a lot of guides that you can look into to get more details. So, now we'll start digging into the code a bit. So, I think George Neville Neill calls this code splunking in his ACM column, so we're going to start digging. So, the vast majority of the OS code is machine independent, but we're basically just going to dig into the machine-dependent code, so the ARM code. So, that's a mix of C in some assembly and some inline assembly, so that's why we quickly went over the assembly syntax earlier. And we'll see examples from FreeBSD and NEPISD, and by time, there would have been openBSD here too. Okay. So, FreeBSD and NEPISD ARM both have great ARM support. There's a few notable differences, really, so there are interesting differences to me. So, NEPISD has this build.sh strip that's used for cross-building, and it allows for cross-OS building, which is kind of cool. So, you can build from your Mac or your Linux machine, so that's kind of cool. FreeBSD uses Clang even on ARM to build the system, so that's kind of interesting. FreeBSD uses something called Device Tree for hardware configuration, while NEPISD uses the Autocon framework. FreeBSD also has an extra bootloader stage that you saw when we booted up. So, these are the key directories that you're going to want to look at if you want to dig into the code some more. So, these are the include directories. And so, it's interesting that this is ARM32 directory. You might think that happened, like, when ARMv8 was added, but it turns out ARM had a 26-bit version of the architecture a long time ago, so there's like 26-bit ARM support in some of the NEPISD files. And then these files are basically where you get the kind of core, like the source files, ARM, ARM32, and Cortex. So, in terms of the SOC and the Beagleboard, you want to look at the OMAP directory, and the EVB ARM, which stands for Evaluation Board ARM, is where the kind of platform board-specific stuff goes. So, all your Beagleboard machine-debt code goes there. So, the configuration files you want to look at, the files.arm file tells you which files will get built for the ARM port, and the files.cortex file also adds to that. STD.arm is the kind of like build options. So, you look at this stuff for the core files, and then the Beaglebone and the SOC files, you look at the OMAP2, the EVB ARM, the Beagle. And this is the kernel config file, the top-level config file that will pull in the rest of the stuff. So, FreeBSD is similar. It actually has fewer pads to look at. So, you look at the SysArm include and SysArm ARM, so that's the headers in the source for the FreeBSD ARM core support. And then for the SOC and the Beagle, you look at the TI directory that's shared code for all the TI family of SOCs. And then you look at the AM335X directory for the stuff that's specific to the Beagle. And then you've got the configuration file. These are the files that are for the core ARM support. This is Conf. And then you've got the Beagle stuff, so the TI family of SOC files, AM335X, the Beaglebone, and this is the top-level config file. So, if you want to dig into that stuff more, you'll kind of read through those files and see what's going on. So, it'll tell you exactly what's getting built. So, let's talk a little bit about bootloaders. So, I mean, what you saw earlier when I booted up, there was a lot of different bootloaders that happened. And so, we'll kind of go through what happens there. So, at a high level, what the bootloader does is it does sort of low-level hardware initialization. So, DRAM and serial. It'll pass and boot parameters to the kernel and it'll load the kernel. So, I mean, that sounds simple, but it actually does a lot under the hood. So, but at a high level, this is what it does. So, when you boot up, the first thing that happens is that the reset handler in the SSU ROM runs. And the first stage bootloader will run. So, that's MLO or SPL. It's a stripped-down version of Uboot that runs and it's needed since the DRAM isn't initialized. So, this is kind of the output from there. And so, you can tell that it's reading the Uboot image. So, then Uboot will run and that will read the configuration from the un.txt. You can see that here. And it'll run the Uboot loader on FreeBSD. It doesn't do this on NetBSD. And also, read this device tree blob that we'll talk about a little bit later. So, then the third stage bootloader runs. Again, only on NetBSD, on FreeBSD, rather. And this is an implementation of loader8, which is a loader for other architectures as well. And it will read the loader.conf configuration and then it will load the FreeBSD kernel. So, you can kind of see that happening here. And it's going to use that device tree blob. And if you're interested in what the sources are doing, that's where you go. So, device tree. This is used by FreeBSD on several platforms. I think it came from PowerPC originally. And basically what it is is a data structure that describes hardware configuration. So, all the device tree sources are in this directory, sysboot, FDT, DTS, ARM. For the Beagle on Black, you want to look at these files. The Beagle on Black.DTS and the AM335X.DTS.I. So, that's an include file. The Beagle on Black.DTS includes. So, most of the logic is actually in here. So, this is the configuration for the serial port. So, the SSE is the AM335X. And the serial port is at this address, which we saw earlier when I mentioned the device map. So, it's 44E0900. It tells you what kind of serial port it is, where the address range, the register shift. So, it's four byte access. 32 bit. And which interrupts clock frequency and other stuff. So, previously, so that DTS file that we just showed, there was a fragment from that. It'll go, the device tree compiler will take that and turn it into a blob. And so, that becomes the BBBone black.DTB for the Beagle on Black. And it's stored in a compressed format called flatten device tree. The blob can either be compiled into the kernel or the UB loader can load it. In this case, when we saw the output earlier, what's happening is the UB loader is actually loading it separately. It's not built into the kernel. And so, once that happens, the kernel will parse the DTB to learn the board's hardware configuration. And libftd handles the parsing in case you're curious on how that stuff works. So, an FBSD doesn't use device tree, it uses auto comps, the device auto configuration framework. Basically, what happens is the hardware config info is generated by the kernel configuration process. One config eight runs. And so, this is an example of that from the BeagleBones kernel configuration. So, this is the same UART configuration. You can tell the same address that we saw earlier, 44 E09000. And the size is the same. The same range, the same interrupt, and the registry of four are here, these multiple. So, it's just a different way to represent this information. So, at a high level, so we've talked a little bit about bootletters, at a high level, we'll talk about kernel initialization. So, the early kernel initialization is basically just kind of like the low level device stuff. So, it's kind of what mostly we'll be talking about in the coming examples. So, the first thing it does, it'll save off the boot parameters, it'll set initial page table and enable them, you. Then, it'll set up the exception vector table, the exception handlers, and the exception stacks. Then, it'll do some initializing of the devices, like the serial, the interrupt controller, the timers for the clock tick. Then, you'll get into your machine independent initialization, initialize various kernel subsystems, more device initialization. Then, you'll enable interrupts and then switch to user mode and run a knit. So, that's kind of like at a high level. It'll focus mostly on kind of like the early kernel initialization and the machine dependent stuff. So, these are the very first instructions that FreeBSD runs when you boot ARM. So, it's in sysarmarmlocore.s, so this is shared with all the ARM SSCs. So, you can tell that, so it uses the Linux boot API, or at least that's one of the options. There's some if depths around this code. So, in R0, you put 0. R1 gets the machine type, which gets passed in at ARM. And R2 gets the DTB image pointer. So, the first thing that this code does is that it will save off these parameters. The comment tells us that it gets put in ARM boot params and eventually passed in at ARM. And the next thing it does is that it will make sure that interrupts are disabled. Typically, Uboot does this for you, but it's good to do it anyways just to make sure. So, NetBSD, it's fairly similar. Each SSC board has a separate start routine, so that's a little bit different than previous days. So, there's a Beagle start routine that will be used on NetBSD as opposed to a common start routine on previous days. So, the first thing it does is that it will switch to SVC mode and it will disable interrupts. So, similar to previous days, except it also does the mode switch. Uboot should already put this processor in SVC mode, so this probably isn't necessary, but it's good to make sure. It also will save off the various parameters that it got from the boot loader. That's what this STMI-A thing is. It's basically a store multiple. And the moveW moveT thing, it doesn't really matter. It's basically because ARM can't do 32-bit immediate loads, so that's how you do that. But basically what it does is saving off the parameters it got from the boot loader. So, I'll continue walking through what NetBSD does as it boots up. FreeBSD is pretty similar and I'll talk about the similarities. NetBSD has a little bit more branching, so that's part of the reason why I went with that one. So, you can kind of follow through. So, the next thing that it's going to do is that it's going to set up the page table. So, BeagleStart calls this ARM bootL1PTinit function. So, that sets up an initial page table. And so, this is an L1 page table with one megabyte sections. And basically all it does is it identity maps the kernel. So, virtual address equals physical address. And then also map the serial. It also maps the serial, so you can get your debug output to the UART. So, once it sets that up, it'll call ARM CPU init. And ARM CPU init is what, if you read the comment, is what turns on the MMU and caches. So, the ARM CPU init function can be found in this directory. So, it's a little bit misleading because it says it's an A9 function, but it's running on a Cortex A8, but the code is actually the same. And basically what this does is it invalidates the caches and TLBs. And it'll also, after it does that, it'll enable the caches. It'll set the translation table base register, and then it'll enable the MMU. Okay. So, once that happens, the BeagleStart routine will jump to start in a locore.s. And this is actually a shared routine across all the ARM boards. And so, the ARM SoC will, here's the branch to start. And so, start what it does is basically sets up the environment for ccode. So, up until this point, everything you've seen is assembly code. And so, once that's done, start can call init.arm, which is the first ccode that runs. init.arm will call init.arm common. So, init.arm is actually a board specific. And there's an init.arm common, which is actually shared across all the boards. So, after that happens, after init.arm runs and init.arm common runs, start will return back to BeagleStart. And then you can call main, which is the first machine independent code. So, this is actually fairly similar to how previous D boots as well, except main, I think, is MI startup. And then there is also an init.arm function that does all the kind of heavy lifting to do the, to set up the processor and all that stuff. So, here's what init.arm looks like, or what it does basically. So, as I mentioned, init.arm is SOC specific. You can find it in Beagle, MacDep or Mashedep, I don't know. And then the init.arm common is ARM generic. You can find in the ARM32, ARM32boot.c. And so, it performs all the logic needed before main runs. So, it will set up the CPU func structure for the basic CPU functions. So, for the write-ins, in this case, for the ARM to the ARMv7 CPU functions. It will map the devices and initialize the console. It will set up a real page shable and switch to it. There's a lot of code to do that. And then it will also set up the exception vectors and stacks. And finally, it will also parse the boot arguments that it got from Uboot. So, once init.arm runs, you get back to Beagle start and then main runs. And that's the machine-independent code, which we won't really talk about, but it's pretty interesting. And there was a AsiaBSDcon talk about that that I'll reference later if you want to learn about how all that stuff works. Okay. So, let's talk a little bit about exception handling, since we kind of went through the machine-dependent code. So, basically as a kernel hacker, what you have to do is you have to set up the vector table. You have to set up the exception stack pointers. And you have to write handlers for each exception. So, here are the various exceptions that you have. You have the reset, undefined instructions, supervisor call for the SVC or SWI, prefetched abort and data abort. Those are memory faults. Interrupt, fast interrupt, hypervisor call for the SVC instruction. So, the exception vector table is a jump table with eight entries, one for each exception type. So, each entry holds one ARM instruction. So, that instruction can either be a branch to an exception handler or a PC load of the exception handler. And so, that's what that looks like. And there. So, this is previous to use exception vector table. And you can find that in ARM exception.s, so it's shared code. So, basically all this does is it's basically a branch instruction. So, you've got one for each of the various fault handlers. So, you're going to figure out exactly what the system call handler is doing. You look in SWI entry and see what that does. So, the OS has to tell the processor where the exception vector table is. There's a few options. So, you can put it at address zero or address 0xFFF0000. It's kind of the high vectors and where zero is the low vectors. And this is set based on the system control registers VBIT. You can also use the, in ARM V7, the vector base address register to program where the vector table goes. And NEPISD actually has an option for this if you want to do that. So, I think it supports all these options actually depending on which config, which if def you go through. So, this is what previous to use exception set of code looks like. NEPISD is also similar. But, so as I mentioned, previous to has an ARM function. And so, what it'll do is it'll allocate stacks for each of the modes. So, as I mentioned earlier, there's only a few modes that are really important to us. So, there's the IRQ stack for interrupts, the board stack for memory faults, the undefined stack for undefined instructions, and the kernel stack, which is used in SVC mode. So, once you've allocated all the stacks, you want to set the stack pointer for each of the modes. As I mentioned, there's those bank registers. So, you have to go into each mode and set up the stack pointers for each of these for IRQ mode, board mode, undefined mode, and SVC mode. So, once you do that, you will do call ARM vector in it. And you can tell, so it's ARM vectors high, so it's that 0xFFFF000 address. And this will set up the vectors that should go on the table. Okay. So, that's kind of like a high-level overview of how the machine-dependent code works on a BSD, specifically NEPISD and free BSD. So, there's a lot more stuff that happens in the machine-dependent code, or independent code. Some of the machine-independent code is going to call into machine-dependent stuff, like if you need to save context or switch and that kind of stuff. But that's kind of like the overall stuff. So, if you look at the other functions, if you follow each of the exceptions through, you can kind of look into how that stuff works. So, now we'll get into a few kind of practical things about developing BSD and ARM. BSD has actually really good cross-compilation support. So, it's kind of cool because you can cross-build the whole system. You've got the tool chain, the user land, all that stuff. It's all built-in, so it's kind of nice. You don't have to do that stuff yourself. You can also create bootable SD images. It's a project called Crochet for free BSD. And it makes it really easy to create these bootable SD images for these various boards. You can also use build.sh on FBSD to do the same thing. When I was building for the Big O'Bun Black, I used the eARM V7HF API. But you can choose whatever API it makes sense to you. One important thing here is that if you use this stuff, it'll give you the right version of Uboot. If you have an old version of Uboot or a different version of Uboot, it can be incompatible with your kernel. Sometimes I'd run a different version of Uboot or Uboot that came with the board, and sometimes the kernel wouldn't boot or panic or something. If you use this method where you just use the Uboot that came with the Crochet image or the build.sh image, you'll have an easier time. If you don't want to go through the hassle of building the image, you can also grab snapshots off the NetBSD and FreeBSD mirror. If you don't want to deal with burning SD images, you can also NetBoot. You can TFT boot the kernel, you can NFS mount the root file system. If you get tired of building SD images, that's one thing you can do. It might keep you from spending a lot of money on SD cards if you burn them out, if you keep re-imaging them. A few tips on debugging BSD ARM. One of the things you want to do is get printf as early as possible. Uboot usually will initialize a serial for you. If you write to that address, the right THR register, that 4x, 449 address, you can get stuff out to the UART. If you want to see early debug output, you can turn on verbose in it ARM on NetBSD. There's a debug macro on FreeBSD's MacDep.c. That stuff's good. It's good to see all that stuff come out on the console. Another thing that's useful is a JTAG debugger. Some are actually kind of affordable, like the fly swatter. Some are very, very expensive. You might need your company to buy it or something, but fly swatter is sort of reasonable. It's kind of useful so you can single step when you're doing low level port stuff. Especially if you're doing work with Uboot, then you really want a JTAG debugger. Kernel Debugger is also useful. DDB, it's nice to be able to have that as a debugging facility. QMU is also great. You can actually hack on ARM without having any ARM hardware. You need some hardware. You need to at least have a laptop, but you don't need any ARM hardware. Lenora's version of QMU has Beagleboard XM support and Mainline QMU has a QBboard support. The nice thing about this is that you can add debug code to your hardware. It's kind of cool. You can look at the code to figure out exactly what is the ARM process you're doing when, I don't know, exceptions happen. It's kind of nice if you don't want to dig through the ARM. There are a lot of talks on various ARM embedded topics. These are all from various VSC contours. How FreeBSD boots is great. It goes through all the machine-dependent and machine-independent stuff. It focuses on MIPS, most of its generic. It even mentions ARM in some places. That's worth reading if you want to get some more information. NetBSD on the Marvell or ModXP is great if you want to learn more about how to get NetBSD running on a modern kind of ARM SoC. FreeBSD and NetBSD on the APM is actually about PowerPC, but it's an interesting comparison about how you port on FreeBSD versus NetBSD. There are a lot of comparisons there that was kind of interesting. FreeBSD on latest ARM processors, EBA toolchain. That's a lot of interesting stuff about the new ABI and the toolchain support and how that happened. If you want to learn more about Flatten Device Tree, that's a good talk. Then interfacing FreeBSD with Uboot goes through all of the booting stuff in much more detail and talks a lot more about Ubuilder. I think maybe this talk kind of coincided with the creation of that. Porting NetBSD to a new ARM SoC is actually kind of an older document. It's just a web page of what we did in the presentation, but it goes into really heavy detail on how to get up a new SoC. Some of it's dated, but it has sort of the most complete information. If you want to port a new board, then that's where you'd go to. These are all really good presentations and overviews, so you should check them out if you want to do more hacking. As I said, there's a couple of cool ARM talks tomorrow. Transparent Superpages for FreeBSD on ARM and FreeBSD on BeagleBone Black, a robotic application. As I mentioned, this is the same board that the presenter will be using. I imagine a lot of it is probably a cooler demo than I did, so all I did was boot up FreeBSD. In summary, we discussed the basics of the ARM architecture, went into all the system-level aspects. I gave you guys a shopping list of all the boards you could get, you could try to collect them all if you want. It'll be kind of expensive, but you could do it. Then what the assembly looks like and all the system-level aspects. I also gave you the resources, so you should definitely grab the Cortex-A programmers guide and the ARM ARM. Then we looked at some of the machine-dependent BSD code, we kind of walked through what boot looks like. Sort of at a high level, but at the very least, you know which files to look at. You can kind of dig in a little bit more with Cisco or Global or whatever you use or CTags to figure out exactly what happens. There is a lot more detail there, I kind of went over it at a fairly high level. Then there were a couple tips and further resources I added for BSD hacking on ARM. Definitely, the crochet tool is really great and Buildus-A is great if you want to just build images. If you do a lot of hacking, definitely MFS and TFT boot is a good way to go. I want to thank all the BSD developers, especially all the ARM folks. There were a lot of really useful posts on the various mailing lists. Most of the developers have blogs, so there's a lot of really good information on there. I found that really useful when I was kind of getting up to speed on this stuff. If you just kind of Google something, you'll probably come across one of the developers' blogs. That's good stuff out there. I'm hoping, so it's a lot of information. It was kind of like drinking from the fire hose, but I hope that I gave you guys at least some interest and at least foundation so you can kind of start hacking on ARM for yourself. You can just grab some hardware off the shopping list and a BSD of your choice. Some of the things that you could do, you could port to a new board or add driver support or fix some bugs. I know that FreeBSD is working towards becoming a tier one, making ARM a tier one architecture. There was a wiki page about things they wanted done, so you could help out with that. NetBSD and OpenBSD are always looking for more driver support and board support. If you're really enterprising, you could port Dragonflight ARM. I think one of the developers is actually an ARM employee, so maybe you could talk to him. Actually, I think that would be cool. Some of those things I would be interested in. Another one would be getting a Beagleboard XM support and a mainline for FreeBSD. That would be cool. Crochet had, I think there was a wish list item on there. I think when you build a Crochet image, you have to do it as brute, so there was like a thread on that, so you could help out with that, but there's a lot of stuff you can do. There's definitely a lot more development happening on Linux on ARM just because it's a big ecosystem, but there's a lot happening on BST too, and you could be a part of it. If you have any questions about ARM, anyone talk to me, I'll be around, and you can email me. With that, I'm happy to take any questions. Two questions. I noticed you were using some, like what are you using for your serial connection? Yeah, so I'm using this cable from Adafruit. If you go there, after the talk, I can show you where to get it. I don't know, it's like a 10-buck cable or something. Second, do you have time after this, after you're talking to and talking privately about some plugs I'm having on ARM? Sure. Sure. Absolutely. Okay. One thing to be really nice is we actually got an outline of this paper as it getting started on ARM and the handbook or some of the previous data. Oh, cool. Sure. We should talk afterwards and other people who know how to talk to you. Okay, definitely. Sounds good. What other questions does it serve? Okay. I'll second that one. Cool. Second, awesome. So if you're building for different ARM-based devices, is there stuff like make-bar and the kernels, so should we just be trying to stuff on ARM or get that to the file and run on the actual, last year, or is that something? Yes, I mean a lot of those stuff is kind of dependent on like the configuration stuff. So some of the stuff is going to be just specific to your specific, so there's ARM generic stuff, there's Cortex stuff, then there's stuff that's for your specific SOC and for your specific board. So some of the code will be shared, some of it will be if-deft, a lot of it's driven from the configuration files. So if you look at the files that are ARM and STD to ARM, that basically defines which files will get built for each SOC and for each board. Cool. All right, thanks a lot. Thank you.
 Turn the light on.  Okay, we're good.  We're good?  Yeah.  Yeah, yeah.  Here we go.  Sorry about that.  So, is it okay if I get six minutes over?  Should I try to finish at 5.30 or is it okay if I go to 5.36?  Okay.  Cool.  Thanks.  Thanks.  You got both mics on?  Yeah, I haven't turned this one on yet.  They're both on?  Yeah, I just turned it on.  It's on now.  Okay, we'll just go grab some so we can start recording.  Okay, sure.  Okay, cool.  Is it on?  I don't know.  Okay.  Okay.  Hi, okay.  So, we've resolved the recording stuff, so we're good to go.  So, my name is Arun Thomas and I'm going to talk about BSD ARM kernel internals.  This is my second BSD conference.  I gave a talk at EuroBSDCon 2011 a few years ago and so it's good to be back at another  BSD conference.  They're always a good time.  So, let's talk about a BSD ARM.  So, I'm going to start off with a little bit of a quick demo.  So, this is a previous BSD current running I guess a few weeks ago when I built it and  a little bit of debug code.  And so, you can see it booting.  There's a mix of debug code and witnesses on, so it's going to be a little bit slower.  So, you'll see that there was, and I have some screenshots later that'll kind of, you  can actually see what's going on.  But there were a couple versions of Uboot that ran.  Uboot loader is running now and then it's going to load the kernel.  So, I'll take a couple seconds.  Okay, here we go.  Yes, I want to start now.  So, you've got all this kernel code.  It's booting up, it's doing some device stuff.  And then, shortly it'll boot up the rest of the way.  So, this is kind of, I'm going to kind of go through the flow exactly what's going on.  So, to get to this point there was a lot of like machine-dependent stuff that needed to  happen.  And so, here we go.  You're in user land and stuff's happening around the file systems and pretty soon you  get to root prompt.  So, I'm running it off of Beaglebone Black.  This is the board here.  It's pretty cheap.  It's a nice platform.  It's supported by all the BSTs that have ARM ports.  And so, yeah, it'll keep going.  And so, this is the black.  I have the white tube.  So, we're looking at an SSH session from your laptop.  No, no, this is just, I plugged in the serial.  There's a serial cable going on.  So, Adafruit has these handy little cables you can just plug in.  So, just plug in root.  Password, you get the free BST prompt.  So, this uses UBoot.  So, free BST, I think everyone uses UBoot on ARM boards basically.  So, that's that.  So, that's the basic process and then I'm going to kind of go into like what happens  when you're booting up.  I'm going to focus on the machine-dependent parts, the machine-independent parts that  I don't really talk about at all, but we'll look at some of the code and see what happens.  So, all right.  So, these are a couple of those screenshots so you can kind of see what happens.  So, there's a couple versions of UBoot running.  That's one.  That's another one.  And then, the free BST UBoot loader runs.  And then, there's something here about a DTB.  It's a device tree blob.  I'll talk about what that is.  And then, you get the copyright notice.  Before that happens, there's a whole bunch of machine-dependent stuff that needs to happen  before you get to this.  And I'll talk about all of that stuff that happens.  So, you can tell that it's free BST, 11 current.  And it's got a weird revision because I'm using git and it's dirty,  so because I had some local hacks.  Postname is imaginatively beasty.  And it's the Beaglebone kernel right there.  And I'm using crochet.  It's just who I'll talk about a little bit later to build this image.  And it's kind of cool for BSTs built with Clang.  And this is running in Cortex-A8.  I'll talk a little bit about the other processors that ARM has,  all the Cortex processors.  Here are some of the features.  And then, it talks a little bit about caches.  And then, this is the system on-ship, the SOC, the TI AM-3358.  And we'll talk a little bit about SOCs and all that stuff,  so if you're not familiar with ARM.  And then, there's some more stuff about device tree.  And this is basically how the UART gets mapped.  So, this is the memory range.  This is the IRQ.  And we'll go into all of this.  So, the goal of this talk is to get you hacking BST on ARM.  So, are there any people here who haven't done any hacking on any ARM hacking  or have done little hacking?  Raise your hands.  Excellent.  So, my goal is to get all of you, like, at least interested in hacking ARM.  So, that's my goal for this talk.  So, we'll start off with the ARM 101.  So, we'll talk a little bit about the basics of the ARM architecture,  what the assembly looks like,  what the system level stuff happens, looks like,  various boards that you could buy, the SOCs, all that stuff.  So, just like a quick bootstrap to the ARM architecture.  Then, we'll go through some of the kernel code.  We'll look at NetBSD and FreeBSD.  I just picked those because those are the ones I've hacked.  But, after watching some of the OpenBSD MIPS talks,  I'm hoping to get OpenBSD running on ARM.  Maybe in an updated version of this talk,  I'll add some OpenBSD stuff.  So, we'll focus on the machine-dependent stuff.  I won't look at the machine-independent stuff, as I mentioned earlier.  So, we'll look at how the kernel boots up  and does some exception handling  and sets up the initial page table and all of that stuff.  It'll still be kind of high-level, but you can at least,  I'll give you at least kind of like the files you want to look at.  It's sort of high-level, so you can dig into more detail later.  Then, there's a short section on tips for hacking on  BSDRM and debugging and more resources,  so you can dig in further.  I won't be able to get into everything,  but there's a lot of good material out there that you can look into.  So, there we go.  Okay, so the ARM architecture.  Hugely popular and embedded systems, you probably own several, each of you.  It's also moving into general-purpose computing.  So, you've got laptops, Sops netbooks.  Some of the Chromebooks have ARM chips in them.  They're pretty nice, actually.  There's servers.  HP was making ARM servers at one point.  It's also moving into high-performance computing.  GPGPU and Vida have some pretty cool SOCs coming out.  And the main reason for this is power efficiency.  So, it's a big push.  So, ARM has an interesting business model.  It doesn't manufacture chips.  It basically licensed the architecture and designs to various silicon vendors  like TI and Samsung and all that stuff.  So, they basically just come up with the architecture  and some of the processor designs, someone else will package it up into SOC  and fabricate it.  Okay, so ARM architecture.  So, ARM stands for Advanced Risk Machine.  It used to be a corn risk machine.  And since it's a risk machine, it's reduced instructions like computer,  fewer instructions, simpler instructions.  Because it's risk, it's load-store architecture.  So, if you want to operate on memory, you have to load it from memory into register,  operate it on there, and then store it back out to memory,  unlike, say, x86, which is not very risky.  So, it's big Indian or little Indian.  Little Indian is more common.  So, the current versions, there's several versions of the ISA.  The current version is ARMv7 and ARMv8.  So, ARMv7 is 32-bit and ARMv8 is 64-bit.  It's ARM, it does some cool stuff in ARMv8.  They simplify the architecture.  But I won't be talking about that at all in this talk.  So, I'm going to focus purely on ARMv7 and 32-bit.  So, each of the ISAs also has various architecture profiles.  So, there's the application profile, the real-time profile, and the microcontroller profile.  I'm only going to talk about the application profile.  So, the real-time microcontroller profiles are more for embedded systems,  and they don't have full MMU support.  So, as I said, ARMv7a is what we'll talk about.  32-bit ARM processors.  They're called Cortex-A, if you're looking at models.  They have full MMU support, and if you look at the ARM documents,  they're designed for full feature operating systems.  So, things like BSD and iOS and stuff like that,  not really the kind of like embedded operating systems.  So, the ARMv7a has two instruction sets, actually.  So, there's the ARM instruction set, and there's the thumb instruction set.  So, the ARM instruction set is bigger than the thumb instruction set.  So, the thumb instruction set has a mix of 16 and 32-bit instructions,  where ARM is just 32-bit.  So, the 32-bit instructions were added with the thumb 2,  and they added thumb 2 technology.  So, the good thing about thumb is that you get better code density,  and that's good for your caches.  So, all the ARM CPUs are packaged up with other logic into a system on ship or SOC.  So, you'll hear this SOC acronym a lot.  So, what's in an SOC?  So, you have your interrupt controllers, your timers, your UARTs,  SDMMC controllers, SATA controllers,  USB controllers, GPUs, and all kinds of peripherals.  So, you might have a camera controller, or GPS controller, all kinds of stuff.  All that stuff that goes into your phone is going to be on the SOC.  So, the SOC is actually getting better for the ARM developers.  So, ARM is actually kind of standardized some of these things,  like the interrupt controller and the timers.  Before, each SOC vendor would basically create their own set of timers  and their own set of interrupt controllers.  So, they had kind of hard for OS developers,  since they had to write all these new drivers.  But now ARM has this generic interrupt controller and generic timers,  so it makes it a little bit easier.  ARM is kind of building a platform.  Okay. So, this is what an SOC looks like.  This is the AM335X SOC.  This is the SOC that's in the Beaglebone black.  So, you see that the core is actually a small part of the SOC.  There's a lot of other stuff in there.  So, you've got a GPU here,  and you've got a whole bunch of various buses.  You've got the UR at SPI, I2C stuff,  timers, real-time clock, ACs,  and then you've got your GPIOs.  So, there's a whole bunch of logic on there,  including USB and an EMAC.  So, the SOC has a whole bunch of stuff in addition to the core.  So, as I mentioned, I'm using Beaglebone black.  It can be yours for the low-low price of $45 US.  It's a popular hobbyist board,  and it's supported by FreeBSD, NetBSD, and OpenBSD.  Dragonfly, I don't think, has an ARM port as far as I know,  but I imagine if it did, it would support it as well.  So, it's kind of a cool little board, as a lot of IO.  Another popular board is the Beagleboard XM.  It's a little more expensive.  It's supported by FreeBSD in a branch for Google Summer Code.  I don't think it was integrated back, but may sometime.  NetBSD supports it. OpenBSD supports it.  And the nice thing about this board is,  if you don't want to buy it, QMU,  Linara's version of QMU actually has a support for it.  So, you could just boot it up in a virtual machine.  So, that's kind of cool.  Okay, so this is a bunch of different SOCs,  and these are all supported by,  and the boards that are associated with them.  And these are all supported by one of the BSDs.  So, the top is the OMAP3, OMAP4, the DaVinci, and Sitara.  They're all from Texas Instruments, and they're kind of a family of SOCs.  It's fairly similar.  So, you've got the Beagleboard, and the Beagleboard,  and there's a couple versions of the Beagleboard.  There's a couple versions of the Beaglebone, like,  Sean over there was talking about the Beaglebone white versus black.  I have the black here.  And Pandaboard has a couple versions, too.  So, these are really popular developer boards.  Then you have the Allwinner A10 and A20 that's used in the QB board  and the QB truck, also a fairly popular board, especially now.  The FreeScale IMX6 is in the WAN board.  The Samsung X10S5 is in the Chromebook and the R&Dale board.  That's a really high-end SOC with a really high-end R&P processors on them.  So, the last one's actually really interesting.  So, XilinXinq has these Cortex chips packaged up with some FPGA logic.  So, if you want to do any hardware design, it's a pretty cool platform.  So, these boards range from 45 to, like, 400, 500.  I forget exactly. The Z board is the most expensive board.  The MicroZet, I think, is fairly cheap.  So, if you want to play around with that, it's pretty cool.  So, this is all the Cortex CPUs.  So, on the low end, you have the Cortex A5, the FreeScale Vibrid,  as an SOC that supports the A5.  That's for really kind of, like, embedded stuff.  The Cortex A8 is more popular.  You'll see that in most of these developer boards that are out.  So, the OMAP 3 SOC, DaVinci, Sitara, so your BeagleBoards, BeagleBones,  have this chip in there. So, it's the lower end.  And they all win an A10.  So, the Cortex A9 is sort of the mid-range CPU.  You'll find in the OMAP 4, the FreeScale MX6, and the XilinXinq.  So, this is what's in the panda board.  The Cortex A15 is the high-end.  As I mentioned, it's in the XNOS 5.  You'll find it in a lot of the high-end phones and in the Chromebook.  So, the A7 is actually a replacement for the A8.  So, it's found in the XNOS 5 in the old winner, A20.  So, it'll eventually, like, fully replace it.  And the A12 and the A17 are mid-range CPUs.  So, they'll replace the A9 over time.  And I don't know what SOCs they're in,  but I'm sure they'll be in a lot of SOCs, maybe some of you know.  So, it's sort of, there's a lot of CPUs,  so it's hard to kind of keep track of them all,  and typically the one you'll see a lot is the A8 and A9  for the developer boards that you'll see.  So, now that we've talked a little bit about the boards and the hardware,  I'll talk a little bit about software.  So, ARM has several versions of ABI's,  and so an ABI is an application-finer interface.  So, the ARM docs, the quote is that they are,  rules that an ARM executable must adhere to.  So, things like executable formats, calling conventions, alignment,  what system calls look like, all that kind of stuff is set aside in the ABI.  So, there's several ARM ABI's.  There's the ARM EABI, the ARM Embedded ABI,  and there's the ARM EABI HF,  that's the ARM Embedded ABI with hardware floating point.  So, those are kind of like the current ABI's.  There's also the ARM OABI,  and I don't actually know what it stands for, maybe old, I don't know,  but it's sort of obsolete, but it's used for older versions of ARM.  Does anybody know what OABI stands for?  No? Okay. Just wondering.  So, NEPI and FREME-AC both support EABI and EABI HF.  So, when you're building this stuff,  you want to make sure you have the right tool chain.  So, you want a tool chain that's built for EABI or EABI HF,  depending on what you want.  Okay, so let's look a little bit into what the architecture looks like.  So, the ARM has 16 general purpose registers.  Some of them are used for other things or have kind of dedicated uses.  So, the frame pointer, R11 is the frame pointer, R13 is the stack pointer,  R14 is the link register.  So, when you do a call instruction, the link register will save your current PC,  so you have something to go back to, so that's the link register.  And the program counter is R15.  So, ARM also has some program status registers.  There's also a floating point, it's called VFP instructions,  and SIMD, that's the neon instruction set.  I won't really talk about those,  but if you're doing like, high vector code or GPU code or that kind of stuff,  it'll be useful to you.  So, in terms of this, ARM has these two program status registers  that are fairly important when you're doing systems code.  So, there's the current program status register, the CPSR,  the saved program status register, the CPSR, which is used for exceptions.  And it holds a number of important bits, like the processor mode,  for instance, SVC is a mode, and we'll talk about what the different modes are soon.  The interrupt mask bits, so if you want to disable interrupts, you'll set this IRQ bit.  ARM versus thumb state,  and INS is a big IND, a little IND, and various condition flags.  So, here's what the assembly syntax looks like.  We'll just add two numbers together, we'll add one plus two.  So, you move one into R1, you move two into R2,  then you add R1 and R2 and put in R3.  So, the destination is on the left.  So, that's a basic intro to data operations on ARM.  So, the memory instructions, as I mentioned, it's a load store architecture,  so if you're going to work on stuff or memory, you have to use the LDR,  the load, and the store, STR instruction.  So, you want to move what R1 points to into R0,  and you want to store R0 into what R1 points to.  There's also push and pop instructions that'll push multiple registers onto the stack,  and pop multiple registers off the stack.  So, these are actually aliases for ARM, has these instructions,  LDM, load multiple, and STM, store multiple.  They're the same thing if you have the right suffix and whatnot.  And control flow. So, ARM, of course, has control flow.  So, this is branch of 0, so bz loop, suspect.  And the call instruction, as I mentioned, is the bl branch link instruction,  and so this will label, and so this will save the current PC to the link register.  If you're going to do a return, it's a bxlr, so branch exchange of the link register.  I'll just jump back to whatever's in the LR, and in older versions of the ISA,  this is how you do it. You just move the LR into the PC.  That's, I think, deprecated in ARM v7, but it still works.  They just don't recommend it, especially if you're doing thumb stuff.  Okay, so that's kind of like the application level stuff.  So, let's look into the OS relevant bits, because that's what we're doing.  We're trying to get the OS going.  So, there's actually more than this, but the two privileges I would care about are PL0 and PL1.  So, PL0's unprivileged user code, PL1 is privileged kernel code, which is what it's used for.  And there are nine operating modes, so there's one unprivileged user mode that runs at PL0,  and there's eight privileged modes that we'll talk about, and they run at PL1 and above.  So, the privileged modes are used mostly for exception in interrupt handling.  So, it's a little bit complicated.  ARM v8 cleans some of the stuff up, but if you're going to be doing kernel programming,  especially at this level, it's good to know how this stuff works.  So, here are the various modes.  There's the supervisor mode, which handles, which sees for system calls,  and it's also the initial mode that the processor starts up in.  There's interrupt mode for normal interrupts, fast interrupt mode for higher priority interrupts,  faster as well as you might expect from the name.  There's abort mode for memory faults, undefined mode for illegal instructions.  You can also use this to emulate instructions and software.  System mode, which is a privileged mode for user mode registers.  It doesn't really get used that much.  Hypervisor mode for VMM support and monitor mode for trust zone.  The modes that you really, that we'll really care about are supervisor mode, interrupt mode, abort mode,  and undefined mode. Those are the ones that typically get used.  So, there's one kind of interesting feature of the architecture.  There's a thing called banked registers, and it's a little bit complicated, but it's good to know about.  So, most registers are shared across all the modes.  So, you have one PC that applies for all the modes.  But there are some registers that are dedicated for each mode.  So, you can have, and so there are duplicate registers for each of these modes.  So, there's a separate stack for each mode, so you're going to want to have separate stack pointers.  So, there's a stack pointer for user mode, a stack pointer for a supervisor mode, for instance.  And the CPU will automatically set the appropriate banked register, depending on what mode you're in.  So, in user mode, the CPU knows that you want the user mode stack pointer.  In SVC mode, the CPU knows you want the SVC stack pointer.  So, why do they add this stuff?  So, it's important for exception handling. So, basically, we wonder how do we get back to the faulting instruction.  So, we need to do that. We need to save the state at the time of the exception.  So, the banked link register, so usually the link register saves the PC when you do a call.  Here, it will save the program counter at exception time.  And the saved program status register will save the current program status register at exception time.  So, it basically saves your state.  So, for instance, if you're doing a system call, LRSVC and SPRSVC, the link register and SVC mode,  and the SPSR register and SVC mode, save the program counter and the current program status at the time of the SVC exception.  So, from the user code.  So, typically, when the processor changes modes, it will do that kind of automatically on an exception.  So, it will switch to interrupt triggers. It will switch to IRQ mode on SVC instructions, also called SWI, or software interrupt.  If you look through the BSE code, this is actually the name that gets used.  But if you look at the new ARM docs, they tell you everything uses SVC,  because I guess that's sort of the new name for the supervisor call or system call instruction.  So, the SVC instruction will switch to SVC mode.  So, if it's basically how it works, each exception, you go to the recording mode.  So, the OS can also change the mode using the privileged CPS instruction.  So, that's the change processor state instruction.  So, the CPS instruction basically what it does is it modifies the mode field in the CPSR.  So, this is basically, you'll find this when we talk a little bit later about what the instruction,  the kind of early boot code looks like in FreeBSD and at BSD. It'll do this.  So, this is how you switch to SVC mode if you want to use the CPS instruction.  And if you want to switch to SVC mode and disable interrupts, this is the SVC mode.  And you use the suffix ID, which is interrupt disable, and you tell which bits, the IRQ bit and the fast IRQ bit.  So, back to the status registers.  So, if you'll recall these hold the current mode, the interrupt disable and endiness.  This is how you read and write them. So, this is if you want to read the CPSR and if you want to write the SPSR, it looks like this.  The instructions are a little bit confusing. So, it's move to register from status is read,  and then write is move to status from register. So, these are in the docs and you'll get used to them.  But there is a gotcha here and actually this was fixed in FreeBSD a couple months ago.  So, there's if you don't give the FSXC suffix, which basically lays out each of the parts of the status to write,  you might not the compiler or the assembler won't necessarily do the right thing.  So, there's an suffix underscore all and it doesn't actually mean all.  So, depending on the version of gas you use, bad things can happen.  So, there was this bug with the wrong Indian register restore bug and so this was because all didn't really mean all.  So, if you're interested you can look into the revision ID, but it was kind of an interesting bug that was fixed  and I think it took a long time for it because in LaPore it fixed it. So, tricky.  So, let's look over view ARM Virtual Memory. It's 32 bit address on ARM v7.  ARM v8 of course is 64 bit and if you use LPA it's actually 40 bits, but we'll talk about the 32 bit ARM v7 stuff.  So, with that you get a 4 gigabyte virtual address space and there's paging support, two levels of paged tables.  The TLBs are hardly managed so the MMU will do the paged table walk and the TLB miss.  The commonly used page sizes are 4KB small page sizes and one megabyte sections.  And if you want to know more, a lot more about ARM Virtual Memory there's a talk about transparency for pages tomorrow.  So, you should go check that out. You'll learn a lot more about virtual memory because I'm just covering the basics.  Okay, so the other kind of key thing in the architecture is coprocessor 15.  So, if you're doing kernel hacking you're going to have to look at this.  So, it's a system control coprocessor. Coprocessor is a little bit of a misleading term because it's actually an integral part of the architecture.  You can't really take it out, but so it's heavily used for systems programming and it's used for things like setting up the processes page table.  So, this is how you write the page table to the translation table base register.  So, it's an MCR instruction, move to coprocessor from register and it's a little, I don't know, it doesn't, you have to read the docs to figure out exactly like which numbers to put in there and stuff.  So, usually these things are wrapped up in inline assembly so that makes it a little bit easier.  The other important thing this does is it holds the system control register.  So, the system control register allows you to enable the M.U., the branch predictor, caching and it tells you where to put the exception.  It allows the OS to tell the CPU where the exception vectors will go.  So, yeah, reading, MRC, writing, MCR, so move to coprocessor from register and yeah, this stuff is in the docs but it takes a little getting used to.  I kept screwing it up, but maybe you guys will be faster.  So, this is probably the most important slide in the whole presentation.  So, this is where you go to get more information about ARM.  So, I'm just going to kind of like touch some of the basics, this is where you go.  So, yeah, this is a quote from the NetBSD code and I'll read it out.  It's, and thus spake the ARM ARM.  So, what's the ARM ARM?  It's the ARM architecture reference manual.  And so, that is all the details about the instructions that architecture.  It's 2,000 pages, or almost 2,000 pages.  So, you're probably not going to read it cover to cover, but it has, it'll answer all the questions you have about the ARM architecture.  So, you want the ARM V7A and ARM V7R edition of that.  If you want kind of a quicker, kind of a lighter introduction to ARM, the ARM Cortex-A series programmers guide is really nice.  It came out a couple years ago and they just updated it for this year.  And the latest revision, they really updated and expanded a lot of the discussion.  So, you should, if you're interested in ARM, you should download that like now.  Both of these are free and it does require registration on the ARM website, but they're free.  So, the other thing, the other resource that's good is the ARM System Developers Guide.  This is actually a printed book and it's from 2004, so it's about a decade ago.  And ARM moves pretty fast.  So, it's definitely dated, but if you want to really dig into like, kind of like the systems level kernel stuff,  like exactly exceptions work and some of that stuff, it's a good resource.  In addition to the core ARM docs, you also want to look at the manuals for your processor, your SOC, and your board.  So, for the Beagle board, it's a Cortex-A8.  You want to look at the Cortex-A8 TRM, for the, sorry, the Beaglebone as well.  And then for the Beaglebone, you want to look at the AM335X TRM.  And then you want to look at the system reference manual for that.  And it'll give you stuff basically like the device map.  So, for the Beaglebone black, these are some of the key addresses that you need to know.  440900 is where the UART is.  You write to the address, you can write to the UART.  The interrupt controller is at this address, the timer, DM timer 1's clock is at this address, and the RAM's at this address.  So, you want to look at your TRM to figure out exactly how to set up your board.  ARM also has several migration guides.  So, for the folks who are coming from Xe6 or MIPS or Power, these are really good guides.  And even if you're just kind of interested in computer architecture, this is kind of cool.  So, I saw some MIPS talks today, so I'm thinking about reading the MIPS to ARM guide,  except I'm going to go from, maybe from ARM to MIPS as well.  So, the IA32 guide has, like, for instance, it tells you about Agaccio,  with the fact that characters are unsigned by default, and I can call this problem.  There's a lot of useful tips in there.  Okay. So, that's kind of a quick introduction to the ARM architecture.  There's a lot of material, but as I said, there's a lot of guides that you can look into to get more details.  So, now we'll start digging into the code a bit.  So, I think George Neville Neill calls this code splunking in his ACM column, so we're going to start digging.  So, the vast majority of the OS code is machine independent, but we're basically just going to dig into the machine-dependent code, so the ARM code.  So, that's a mix of C in some assembly and some inline assembly, so that's why we quickly went over the assembly syntax earlier.  And we'll see examples from FreeBSD and NEPISD, and by time, there would have been openBSD here too.  Okay. So, FreeBSD and NEPISD ARM both have great ARM support.  There's a few notable differences, really, so there are interesting differences to me.  So, NEPISD has this build.sh strip that's used for cross-building, and it allows for cross-OS building, which is kind of cool.  So, you can build from your Mac or your Linux machine, so that's kind of cool.  FreeBSD uses Clang even on ARM to build the system, so that's kind of interesting.  FreeBSD uses something called Device Tree for hardware configuration, while NEPISD uses the Autocon framework.  FreeBSD also has an extra bootloader stage that you saw when we booted up.  So, these are the key directories that you're going to want to look at if you want to dig into the code some more.  So, these are the include directories.  And so, it's interesting that this is ARM32 directory.  You might think that happened, like, when ARMv8 was added, but it turns out ARM had a 26-bit version of the architecture a long time ago,  so there's like 26-bit ARM support in some of the NEPISD files.  And then these files are basically where you get the kind of core, like the source files, ARM, ARM32, and Cortex.  So, in terms of the SOC and the Beagleboard, you want to look at the OMAP directory,  and the EVB ARM, which stands for Evaluation Board ARM, is where the kind of platform board-specific stuff goes.  So, all your Beagleboard machine-debt code goes there.  So, the configuration files you want to look at, the files.arm file tells you which files will get built for the ARM port,  and the files.cortex file also adds to that.  STD.arm is the kind of like build options.  So, you look at this stuff for the core files, and then the Beaglebone and the SOC files,  you look at the OMAP2, the EVB ARM, the Beagle.  And this is the kernel config file, the top-level config file that will pull in the rest of the stuff.  So, FreeBSD is similar.  It actually has fewer pads to look at.  So, you look at the SysArm include and SysArm ARM, so that's the headers in the source for the FreeBSD ARM core support.  And then for the SOC and the Beagle, you look at the TI directory that's shared code for all the TI family of SOCs.  And then you look at the AM335X directory for the stuff that's specific to the Beagle.  And then you've got the configuration file.  These are the files that are for the core ARM support.  This is Conf.  And then you've got the Beagle stuff, so the TI family of SOC files, AM335X, the Beaglebone,  and this is the top-level config file.  So, if you want to dig into that stuff more, you'll kind of read through those files and see what's going on.  So, it'll tell you exactly what's getting built.  So, let's talk a little bit about bootloaders.  So, I mean, what you saw earlier when I booted up, there was a lot of different bootloaders that happened.  And so, we'll kind of go through what happens there.  So, at a high level, what the bootloader does is it does sort of low-level hardware initialization.  So, DRAM and serial.  It'll pass and boot parameters to the kernel and it'll load the kernel.  So, I mean, that sounds simple, but it actually does a lot under the hood.  So, but at a high level, this is what it does.  So, when you boot up, the first thing that happens is that the reset handler in the SSU ROM runs.  And the first stage bootloader will run.  So, that's MLO or SPL.  It's a stripped-down version of Uboot that runs and it's needed since the DRAM isn't initialized.  So, this is kind of the output from there.  And so, you can tell that it's reading the Uboot image.  So, then Uboot will run and that will read the configuration from the un.txt.  You can see that here.  And it'll run the Uboot loader on FreeBSD.  It doesn't do this on NetBSD.  And also, read this device tree blob that we'll talk about a little bit later.  So, then the third stage bootloader runs.  Again, only on NetBSD, on FreeBSD, rather.  And this is an implementation of loader8, which is a loader for other architectures as well.  And it will read the loader.conf configuration and then it will load the FreeBSD kernel.  So, you can kind of see that happening here.  And it's going to use that device tree blob.  And if you're interested in what the sources are doing, that's where you go.  So, device tree.  This is used by FreeBSD on several platforms.  I think it came from PowerPC originally.  And basically what it is is a data structure that describes hardware configuration.  So, all the device tree sources are in this directory, sysboot, FDT, DTS, ARM.  For the Beagle on Black, you want to look at these files.  The Beagle on Black.DTS and the AM335X.DTS.I.  So, that's an include file.  The Beagle on Black.DTS includes.  So, most of the logic is actually in here.  So, this is the configuration for the serial port.  So, the SSE is the AM335X.  And the serial port is at this address, which we saw earlier when I mentioned the device map.  So, it's 44E0900.  It tells you what kind of serial port it is, where the address range, the register shift.  So, it's four byte access.  32 bit.  And which interrupts clock frequency and other stuff.  So, previously, so that DTS file that we just showed, there was a fragment from that.  It'll go, the device tree compiler will take that and turn it into a blob.  And so, that becomes the BBBone black.DTB for the Beagle on Black.  And it's stored in a compressed format called flatten device tree.  The blob can either be compiled into the kernel or the UB loader can load it.  In this case, when we saw the output earlier, what's happening is the UB loader is actually loading it separately.  It's not built into the kernel.  And so, once that happens, the kernel will parse the DTB to learn the board's hardware configuration.  And libftd handles the parsing in case you're curious on how that stuff works.  So, an FBSD doesn't use device tree, it uses auto comps, the device auto configuration framework.  Basically, what happens is the hardware config info is generated by the kernel configuration process.  One config eight runs.  And so, this is an example of that from the BeagleBones kernel configuration.  So, this is the same UART configuration.  You can tell the same address that we saw earlier, 44 E09000.  And the size is the same.  The same range, the same interrupt, and the registry of four are here, these multiple.  So, it's just a different way to represent this information.  So, at a high level, so we've talked a little bit about bootletters,  at a high level, we'll talk about kernel initialization.  So, the early kernel initialization is basically just kind of like the low level device stuff.  So, it's kind of what mostly we'll be talking about in the coming examples.  So, the first thing it does, it'll save off the boot parameters,  it'll set initial page table and enable them, you.  Then, it'll set up the exception vector table, the exception handlers, and the exception stacks.  Then, it'll do some initializing of the devices, like the serial, the interrupt controller, the timers for the clock tick.  Then, you'll get into your machine independent initialization,  initialize various kernel subsystems, more device initialization.  Then, you'll enable interrupts and then switch to user mode and run a knit.  So, that's kind of like at a high level.  It'll focus mostly on kind of like the early kernel initialization and the machine dependent stuff.  So, these are the very first instructions that FreeBSD runs when you boot ARM.  So, it's in sysarmarmlocore.s, so this is shared with all the ARM SSCs.  So, you can tell that, so it uses the Linux boot API, or at least that's one of the options.  There's some if depths around this code.  So, in R0, you put 0.  R1 gets the machine type, which gets passed in at ARM.  And R2 gets the DTB image pointer.  So, the first thing that this code does is that it will save off these parameters.  The comment tells us that it gets put in ARM boot params and eventually passed in at ARM.  And the next thing it does is that it will make sure that interrupts are disabled.  Typically, Uboot does this for you, but it's good to do it anyways just to make sure.  So, NetBSD, it's fairly similar.  Each SSC board has a separate start routine, so that's a little bit different than previous days.  So, there's a Beagle start routine that will be used on NetBSD as opposed to a common start routine on previous days.  So, the first thing it does is that it will switch to SVC mode and it will disable interrupts.  So, similar to previous days, except it also does the mode switch.  Uboot should already put this processor in SVC mode, so this probably isn't necessary, but it's good to make sure.  It also will save off the various parameters that it got from the boot loader.  That's what this STMI-A thing is.  It's basically a store multiple. And the moveW moveT thing, it doesn't really matter.  It's basically because ARM can't do 32-bit immediate loads, so that's how you do that.  But basically what it does is saving off the parameters it got from the boot loader.  So, I'll continue walking through what NetBSD does as it boots up.  FreeBSD is pretty similar and I'll talk about the similarities.  NetBSD has a little bit more branching, so that's part of the reason why I went with that one.  So, you can kind of follow through.  So, the next thing that it's going to do is that it's going to set up the page table.  So, BeagleStart calls this ARM bootL1PTinit function.  So, that sets up an initial page table.  And so, this is an L1 page table with one megabyte sections.  And basically all it does is it identity maps the kernel.  So, virtual address equals physical address.  And then also map the serial.  It also maps the serial, so you can get your debug output to the UART.  So, once it sets that up, it'll call ARM CPU init.  And ARM CPU init is what, if you read the comment, is what turns on the MMU and caches.  So, the ARM CPU init function can be found in this directory.  So, it's a little bit misleading because it says it's an A9 function,  but it's running on a Cortex A8, but the code is actually the same.  And basically what this does is it invalidates the caches and TLBs.  And it'll also, after it does that, it'll enable the caches.  It'll set the translation table base register, and then it'll enable the MMU.  Okay.  So, once that happens, the BeagleStart routine will jump to start in a locore.s.  And this is actually a shared routine across all the ARM boards.  And so, the ARM SoC will, here's the branch to start.  And so, start what it does is basically sets up the environment for ccode.  So, up until this point, everything you've seen is assembly code.  And so, once that's done, start can call init.arm, which is the first ccode that runs.  init.arm will call init.arm common.  So, init.arm is actually a board specific.  And there's an init.arm common, which is actually shared across all the boards.  So, after that happens, after init.arm runs and init.arm common runs,  start will return back to BeagleStart.  And then you can call main, which is the first machine independent code.  So, this is actually fairly similar to how previous D boots as well, except main,  I think, is MI startup.  And then there is also an init.arm function that does all the kind of heavy lifting  to do the, to set up the processor and all that stuff.  So, here's what init.arm looks like, or what it does basically.  So, as I mentioned, init.arm is SOC specific.  You can find it in Beagle, MacDep or Mashedep, I don't know.  And then the init.arm common is ARM generic.  You can find in the ARM32, ARM32boot.c.  And so, it performs all the logic needed before main runs.  So, it will set up the CPU func structure for the basic CPU functions.  So, for the write-ins, in this case, for the ARM to the ARMv7 CPU functions.  It will map the devices and initialize the console.  It will set up a real page shable and switch to it.  There's a lot of code to do that.  And then it will also set up the exception vectors and stacks.  And finally, it will also parse the boot arguments that it got from Uboot.  So, once init.arm runs, you get back to Beagle start and then main runs.  And that's the machine-independent code, which we won't really talk about,  but it's pretty interesting.  And there was a AsiaBSDcon talk about that that I'll reference later  if you want to learn about how all that stuff works.  Okay.  So, let's talk a little bit about exception handling,  since we kind of went through the machine-dependent code.  So, basically as a kernel hacker, what you have to do is you have to set up the vector table.  You have to set up the exception stack pointers.  And you have to write handlers for each exception.  So, here are the various exceptions that you have.  You have the reset, undefined instructions, supervisor call for the SVC or SWI,  prefetched abort and data abort.  Those are memory faults.  Interrupt, fast interrupt, hypervisor call  for the SVC instruction.  So, the exception vector table is a jump table with eight entries,  one for each exception type.  So, each entry holds one ARM instruction.  So, that instruction can either be a branch to an exception handler  or a PC load of the exception handler.  And so, that's what that looks like.  And there.  So, this is previous to use exception vector table.  And you can find that in ARM exception.s, so it's shared code.  So, basically all this does is it's basically a branch instruction.  So, you've got one for each of the various fault handlers.  So, you're going to figure out exactly what the system call handler is doing.  You look in SWI entry and see what that does.  So, the OS has to tell the processor where the exception vector table is.  There's a few options.  So, you can put it at address zero or address 0xFFF0000.  It's kind of the high vectors and where zero is the low vectors.  And this is set based on the system control registers VBIT.  You can also use the, in ARM V7, the vector base address register  to program where the vector table goes.  And NEPISD actually has an option for this if you want to do that.  So, I think it supports all these options actually depending on which config,  which if def you go through.  So, this is what previous to use exception set of code looks like.  NEPISD is also similar.  But, so as I mentioned, previous to has an ARM function.  And so, what it'll do is it'll allocate stacks for each of the modes.  So, as I mentioned earlier, there's only a few modes that are really important to us.  So, there's the IRQ stack for interrupts, the board stack for memory faults,  the undefined stack for undefined instructions, and the kernel stack,  which is used in SVC mode.  So, once you've allocated all the stacks, you want to set the stack pointer for each of the modes.  As I mentioned, there's those bank registers.  So, you have to go into each mode and set up the stack pointers for each of these  for IRQ mode, board mode, undefined mode, and SVC mode.  So, once you do that, you will do call ARM vector in it.  And you can tell, so it's ARM vectors high, so it's that 0xFFFF000 address.  And this will set up the vectors that should go on the table.  Okay. So, that's kind of like a high-level overview of how the machine-dependent code  works on a BSD, specifically NEPISD and free BSD.  So, there's a lot more stuff that happens in the machine-dependent code, or independent code.  Some of the machine-independent code is going to call into machine-dependent stuff,  like if you need to save context or switch and that kind of stuff.  But that's kind of like the overall stuff.  So, if you look at the other functions, if you follow each of the exceptions through,  you can kind of look into how that stuff works.  So, now we'll get into a few kind of practical things about developing BSD and ARM.  BSD has actually really good cross-compilation support.  So, it's kind of cool because you can cross-build the whole system.  You've got the tool chain, the user land, all that stuff.  It's all built-in, so it's kind of nice.  You don't have to do that stuff yourself.  You can also create bootable SD images.  It's a project called Crochet for free BSD.  And it makes it really easy to create these bootable SD images for these various boards.  You can also use build.sh on FBSD to do the same thing.  When I was building for the Big O'Bun Black, I used the eARM V7HF API.  But you can choose whatever API it makes sense to you.  One important thing here is that if you use this stuff,  it'll give you the right version of Uboot.  If you have an old version of Uboot or a different version of Uboot,  it can be incompatible with your kernel.  Sometimes I'd run a different version of Uboot or Uboot that came with the board,  and sometimes the kernel wouldn't boot or panic or something.  If you use this method where you just use the Uboot that came with the Crochet image  or the build.sh image, you'll have an easier time.  If you don't want to go through the hassle of building the image,  you can also grab snapshots off the NetBSD and FreeBSD mirror.  If you don't want to deal with burning SD images, you can also NetBoot.  You can TFT boot the kernel, you can NFS mount the root file system.  If you get tired of building SD images, that's one thing you can do.  It might keep you from spending a lot of money on SD cards if you burn them out,  if you keep re-imaging them.  A few tips on debugging BSD ARM.  One of the things you want to do is get printf as early as possible.  Uboot usually will initialize a serial for you.  If you write to that address, the right THR register, that 4x, 449 address,  you can get stuff out to the UART.  If you want to see early debug output, you can turn on verbose in it ARM on NetBSD.  There's a debug macro on FreeBSD's MacDep.c.  That stuff's good. It's good to see all that stuff come out on the console.  Another thing that's useful is a JTAG debugger.  Some are actually kind of affordable, like the fly swatter.  Some are very, very expensive.  You might need your company to buy it or something, but fly swatter is sort of reasonable.  It's kind of useful so you can single step when you're doing low level port stuff.  Especially if you're doing work with Uboot, then you really want a JTAG debugger.  Kernel Debugger is also useful.  DDB, it's nice to be able to have that as a debugging facility.  QMU is also great.  You can actually hack on ARM without having any ARM hardware.  You need some hardware. You need to at least have a laptop, but you don't need any ARM hardware.  Lenora's version of QMU has Beagleboard XM support and Mainline QMU has a QBboard support.  The nice thing about this is that you can add debug code to your hardware.  It's kind of cool. You can look at the code to figure out exactly what is the ARM process you're doing  when, I don't know, exceptions happen.  It's kind of nice if you don't want to dig through the ARM.  There are a lot of talks on various ARM embedded topics.  These are all from various VSC contours.  How FreeBSD boots is great. It goes through all the machine-dependent and machine-independent stuff.  It focuses on MIPS, most of its generic. It even mentions ARM in some places.  That's worth reading if you want to get some more information.  NetBSD on the Marvell or ModXP is great if you want to learn more about how to get NetBSD running on a modern kind of ARM SoC.  FreeBSD and NetBSD on the APM is actually about PowerPC,  but it's an interesting comparison about how you port on FreeBSD versus NetBSD.  There are a lot of comparisons there that was kind of interesting.  FreeBSD on latest ARM processors, EBA toolchain.  That's a lot of interesting stuff about the new ABI and the toolchain support and how that happened.  If you want to learn more about Flatten Device Tree, that's a good talk.  Then interfacing FreeBSD with Uboot goes through all of the booting stuff in much more detail  and talks a lot more about Ubuilder. I think maybe this talk kind of coincided with the creation of that.  Porting NetBSD to a new ARM SoC is actually kind of an older document.  It's just a web page of what we did in the presentation, but it goes into really heavy detail on how to get up a new SoC.  Some of it's dated, but it has sort of the most complete information.  If you want to port a new board, then that's where you'd go to.  These are all really good presentations and overviews, so you should check them out if you want to do more hacking.  As I said, there's a couple of cool ARM talks tomorrow.  Transparent Superpages for FreeBSD on ARM and FreeBSD on BeagleBone Black, a robotic application.  As I mentioned, this is the same board that the presenter will be using.  I imagine a lot of it is probably a cooler demo than I did, so all I did was boot up FreeBSD.  In summary, we discussed the basics of the ARM architecture, went into all the system-level aspects.  I gave you guys a shopping list of all the boards you could get, you could try to collect them all if you want.  It'll be kind of expensive, but you could do it.  Then what the assembly looks like and all the system-level aspects.  I also gave you the resources, so you should definitely grab the Cortex-A programmers guide and the ARM ARM.  Then we looked at some of the machine-dependent BSD code, we kind of walked through what boot looks like.  Sort of at a high level, but at the very least, you know which files to look at.  You can kind of dig in a little bit more with Cisco or Global or whatever you use or CTags to figure out exactly what happens.  There is a lot more detail there, I kind of went over it at a fairly high level.  Then there were a couple tips and further resources I added for BSD hacking on ARM.  Definitely, the crochet tool is really great and Buildus-A is great if you want to just build images.  If you do a lot of hacking, definitely MFS and TFT boot is a good way to go.  I want to thank all the BSD developers, especially all the ARM folks.  There were a lot of really useful posts on the various mailing lists.  Most of the developers have blogs, so there's a lot of really good information on there.  I found that really useful when I was kind of getting up to speed on this stuff.  If you just kind of Google something, you'll probably come across one of the developers' blogs.  That's good stuff out there.  I'm hoping, so it's a lot of information.  It was kind of like drinking from the fire hose, but I hope that I gave you guys at least some interest and at least foundation  so you can kind of start hacking on ARM for yourself.  You can just grab some hardware off the shopping list and a BSD of your choice.  Some of the things that you could do, you could port to a new board or add driver support or fix some bugs.  I know that FreeBSD is working towards becoming a tier one, making ARM a tier one architecture.  There was a wiki page about things they wanted done, so you could help out with that.  NetBSD and OpenBSD are always looking for more driver support and board support.  If you're really enterprising, you could port Dragonflight ARM.  I think one of the developers is actually an ARM employee, so maybe you could talk to him.  Actually, I think that would be cool.  Some of those things I would be interested in.  Another one would be getting a Beagleboard XM support and a mainline for FreeBSD.  That would be cool.  Crochet had, I think there was a wish list item on there.  I think when you build a Crochet image, you have to do it as brute, so there was like a thread on that,  so you could help out with that, but there's a lot of stuff you can do.  There's definitely a lot more development happening on Linux on ARM just because it's a big ecosystem,  but there's a lot happening on BST too, and you could be a part of it.  If you have any questions about ARM, anyone talk to me, I'll be around, and you can email me.  With that, I'm happy to take any questions.  Two questions.  I noticed you were using some, like what are you using for your serial connection?  Yeah, so I'm using this cable from Adafruit.  If you go there, after the talk, I can show you where to get it.  I don't know, it's like a 10-buck cable or something.  Second, do you have time after this, after you're talking to and talking privately about some plugs I'm having on ARM?  Sure.  Sure.  Absolutely.  Okay.  One thing to be really nice is we actually got an outline of this paper as it getting started on ARM and the handbook or some of the previous data.  Oh, cool.  Sure.  We should talk afterwards and other people who know how to talk to you.  Okay, definitely.  Sounds good.  What other questions does it serve?  Okay.  I'll second that one.  Cool.  Second, awesome.  So if you're building for different ARM-based devices, is there stuff like make-bar and the kernels,  so should we just be trying to stuff on ARM or get that to the file and run on the actual, last year, or is that something?  Yes, I mean a lot of those stuff is kind of dependent on like the configuration stuff.  So some of the stuff is going to be just specific to your specific, so there's ARM generic stuff, there's Cortex stuff,  then there's stuff that's for your specific SOC and for your specific board.  So some of the code will be shared, some of it will be if-deft, a lot of it's driven from the configuration files.  So if you look at the files that are ARM and STD to ARM, that basically defines which files will get built for each SOC and for each board.  Cool.  All right, thanks a lot.  Thank you.
BSD_ARM_Kernel_Internals__480p__0_59.png BSD_ARM_Kernel_Internals__480p__60_1127.png BSD_ARM_Kernel_Internals__480p__1128_1235.png BSD_ARM_Kernel_Internals__480p__1236_1871.png BSD_ARM_Kernel_Internals__480p__1872_4655.png BSD_ARM_Kernel_Internals__480p__4656_5159.png BSD_ARM_Kernel_Internals__480p__5160_5387.png BSD_ARM_Kernel_Internals__480p__5388_8087.png BSD_ARM_Kernel_Internals__480p__8088_8483.png BSD_ARM_Kernel_Internals__480p__8484_9083.png BSD_ARM_Kernel_Internals__480p__9084_12191.png BSD_ARM_Kernel_Internals__480p__12192_13043.png BSD_ARM_Kernel_Internals__480p__13044_13103.png BSD_ARM_Kernel_Internals__480p__13104_14507.png BSD_ARM_Kernel_Internals__480p__14508_14627.png BSD_ARM_Kernel_Internals__480p__14628_14891.png BSD_ARM_Kernel_Internals__480p__14892_17507.png BSD_ARM_Kernel_Internals__480p__17508_20663.png BSD_ARM_Kernel_Internals__480p__20664_22391.png BSD_ARM_Kernel_Internals__480p__22392_22955.png BSD_ARM_Kernel_Internals__480p__22956_25955.png BSD_ARM_Kernel_Internals__480p__25956_26135.png BSD_ARM_Kernel_Internals__480p__26136_27383.png BSD_ARM_Kernel_Internals__480p__27384_33503.png BSD_ARM_Kernel_Internals__480p__33504_35267.png BSD_ARM_Kernel_Internals__480p__35268_35627.png BSD_ARM_Kernel_Internals__480p__35628_42047.png BSD_ARM_Kernel_Internals__480p__42048_42755.png BSD_ARM_Kernel_Internals__480p__42756_45503.png BSD_ARM_Kernel_Internals__480p__45504_46751.png BSD_ARM_Kernel_Internals__480p__46752_47507.png BSD_ARM_Kernel_Internals__480p__47508_47627.png BSD_ARM_Kernel_Internals__480p__47628_48131.png BSD_ARM_Kernel_Internals__480p__48132_49511.png BSD_ARM_Kernel_Internals__480p__49512_50627.png BSD_ARM_Kernel_Internals__480p__50628_51083.png BSD_ARM_Kernel_Internals__480p__51084_54503.png BSD_ARM_Kernel_Internals__480p__54504_59003.png BSD_ARM_Kernel_Internals__480p__59004_65219.png BSD_ARM_Kernel_Internals__480p__65220_66131.png BSD_ARM_Kernel_Internals__480p__66132_67511.png BSD_ARM_Kernel_Internals__480p__67512_68627.png BSD_ARM_Kernel_Internals__480p__68628_68759.png BSD_ARM_Kernel_Internals__480p__68760_70631.png BSD_ARM_Kernel_Internals__480p__70632_74003.png BSD_ARM_Kernel_Internals__480p__74004_77879.png BSD_ARM_Kernel_Internals__480p__77880_78131.png BSD_ARM_Kernel_Internals__480p__78132_78515.png BSD_ARM_Kernel_Internals__480p__78516_78899.png BSD_ARM_Kernel_Internals__480p__78900_79115.png BSD_ARM_Kernel_Internals__480p__79116_80255.png BSD_ARM_Kernel_Internals__480p__80256_80507.png BSD_ARM_Kernel_Internals__480p__80508_80879.png BSD_ARM_Kernel_Internals__480p__80880_81635.png BSD_ARM_Kernel_Internals__480p__81636_82007.png BSD_ARM_Kernel_Internals__480p__82008_82379.png BSD_ARM_Kernel_Internals__480p__82380_83003.png BSD_ARM_Kernel_Internals__480p__83004_83135.png BSD_ARM_Kernel_Internals__480p__83136_83759.png BSD_ARM_Kernel_Internals__480p__83760_84251.png BSD_ARM_Kernel_Internals__480p__84252_86199.png