libtrue
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Turner, Andrew
During a talk at EuroBSDCon 2016 it was mentioned FreeBSD had no GENERIC kernel for armv6. While this was true at the time the claim was made a GENERIC kernel configuration was committed before the talk finished. This talk will provide a history of how this kernel configuration, common on other architectures, was added to armv6. FreeBSD added support for the ARMv6 and ARMv7 architectures in 2012. At this time each System on Chip required its own kernel configuration, however there had the start of the work needed to get a single kernel with the requirement for Flatted Device Tree to enumerate hardware. We will look at why having a common kernel on 32-bit ARM is so difficult, and what was needed to make this possible. This will include support for Flattened Device Tree, the historical GENERIC kernel project branch, INTRNG, and the ARM platform and multi-delay work. With these changes a single FreeBSD can boot on at least six vendors SoCs, with more being added.
 I'll start at this comments. I'm going to be not talking about the picture, unfortunately. This is a lie. I will be talking about generic arm. In this case, I mean 32-bit arm, even though I'm known for 64-bit more recently. So we've got this problem. We've got arm and what we want to do is boot on all the things. And between true and everything. So quick introduction about me. I'm now a research associate in the University of Cambridge, in the computer lab, looking for Robert Watson. And I've also, up until fairly recently, been sort of still am, a freelance software engineer. So none of this was related to either of these positions. But I've been a free BSD committer for nearly seven years. Focusing on arm things. And I'm 64 mainly. I don't really know anything about Intel. People keep asking me, comparing between Intel and ARM. I say, you have to tell me about Intel first. I've done these two major projects. I break everything on everyone's arm binaries, and then I ran off to ARM 64. But this is, as I said, this is separate from these. This is my, I don't feel like doing work, so I'm doing something fun instead, and playing with more programming. I've done, so, a small history on ARM v6. We started 2011. So it was, the first ARM v6 CPUs were, they've been in the market since 2002, according to the ARM website. So we were fairly late to this. And even though ARM v6 existed, there wasn't really any hardware that was generally available at all. So it wasn't until ARM v7 came around that we were really the interest in the port. Started with a amount of XP and a few, some TI parts and video. And it's, you'll notice that, so these are all ARM v7 CPUs. So even though it's called ARM v6, it means ARM v6 in later, in 32 bit mode. Most of, many of these have actually been removed or are currently not really building properly. So some of the old code may have been a wee bit dead, but these were the original three ARM platforms. We've imported, so we imported into our version in 2012. That's been almost, we've had it almost five years. We have added a few more SoCs. Here's a few, so you can see the Vodcom chips are only really in Marsby pies, or for free BSD. There's some FreeScale, which I think is now in XP, and soon to be Qualcomm. Some Samsung parts, some Allwinner parts. I don't know how many Allwinner parts we've support, too many to list. We support Smintel, FPGA parts. The second one is the Amazon owned company now. Rockchip, Xilinx, FPGA, QMU, and GIMP5. So we support a reasonable number. We're not everything, but we've only got a number of people working on this, so if we don't want to speed ourselves too thin, we're supporting every part, every SoC. The problem is, we don't have a generic, so what is needed for generic? We need early hardware configuration. So we need something, as you boot in, you need to do a little bit of work fairly early on just to get to the point where you can get to some common code. You need to find some sort of device enumeration. So later on in the kernel, we're booting, we need to figure out what hardware do we actually have in this system we're booting on, because a lot of ARM hardware is not enumable without some sort of external information. So ACPI does provide some of this information on Intel, but also self-enumering buses like PCI Express help with us. But you can't rely on this being, you know, for all devices. And we've got these singleton functions, where each SoC implemented the same function in slightly different ways, or even in the same way. We need to break these a little and create some new interfaces. So early hardware is, yeah, so we're booting the kernel, we're booting, you've got Uboot, for example, as a first level bootloader, we then, that runs the Uboot loader, so UBLDR, and then jumps into the kernel. And then we need to do some sort of figuring out, you know, we're still running in physical addressing at this point. So we've got this problem of, you know, the kernels all had a hard-coded physical address they expected to be loaded in. It's great, except if you're wanting to run one kernel on one piece of hardware, and you can guarantee you're always loading at a fixed address. But even if you're not wanting to run same kernel multiple different types of hardware, just being able to say, you know, not have to hard code addresses is nice. So we came up with a solution. We do some web of runtime detection of what address are we actually running from, and then we can build a page tables around that. And then once we've done that, we're in the virtual address space, and we know we've got a virtual address, so we can force it to be the address we want. That has some issues, though, and it's not necessarily an easy thing to do. But luckily, so what we do, we have to figure out where we've been loaded, the physical address space address. Yeah, finally, we're in physical address space we've been loaded to. And then we have to, you know, build this page table. So how do we do that? It turns out ARM is really nice for this, and that 32-bit ARM, we have a register, which is, you can just use as a source register. And you can use it for move, as a source for move, so you can query what the value of the program counter is. Or you've got some program counter relative memory accesses. There is a slight issue with 32-bit ARM where the program counter doesn't point at the current instruction, but at what the current, at the start of the pipeline on early ARM processes would have been. And they were three-stage pipelines, that's two instructions after the current instruction that you're executing. So it's always, it's a wee bit confusing if you look at this, and suddenly everything's off by two instructions. So that's what, you know, we can, for a lot of this, we can handle that. And it's mostly using PC relative, and the assembler handles, knowing exactly what the PC will be at this given instruction. But then we need to, so we've got, we can figure out what the virtual address, the physical address will be, but we still need to actually build, we need to figure out, you know, we need to build this mapping, because we need to get into virtual address space somehow. So we have to create these two mappings. We have to create a large, I think, I think currently we're building a mapping for the entirety of the address space. And then within that, we create a small 64 megabyte mapping of where it is that the kernel will be running, ultimately be running from. But we use the information. We know the physical address that we wanted to create the map to. It's all, it's a wee bit hairy code, but it all works, and, you know, once you know a delta between virtual and physical address, it's relatively easy. And we can, you know, so we can do this, and we've, it survives. It's quite, if you're really interested in actually how we do it and issues I know in the code, you can come and talk to me afterwards. It's a lot of explanation to actually go through this. So I'm skipping over this a little bit at the moment. But that's fine. Okay, so we can now get into, we can start running C code. But we come to a second problem, and everyone just copies and pastes the code, and they made the minor changes. And so we, the first piece of C code we run in FeeBee's D on ARM is called Inner's ARM. I think we had, I'm not sure how many copies of that we had. I know. One of things, 50. I'm not sure if it was quite 50, but it was, it was many. Too many, you know, any more than one is too many. And I think we're down to two now, maybe a few more. So I came up, you know, because most of the ARMv6 are almost identical, but not quite, I came up with a, I did this, and I pulled out the common code and created some new, per-sot, you know, new functions to handle the per-sic code. This originally wasn't, it was just C function. So there's still the problem of singletons. So we still have, yeah, we need to, still need to implement these same functions. But we can, like, you know, it turns out per-pc has already done this. Yeah, that's how per-pc does it. We just create a K-Obj based, you know, FeeBee's D supports multiple device drivers using K-Obj. And it's a generic thing. So we can just use it as per-pc does in the early boot code. And this turns it to be surprisingly easy. It's, yeah, it gives you support of things like, even within a, if you've got two very similar SoCs, you can then even start running a kernel with these between them. The same kernel on two platforms, even if it's not entirely generic. So, yeah, this code is all called from the inner term, so the very first bit of C code. I started it in a, I created a project branch called Specific League as a play on generic arm. But it did prove that, you know, so this project branch proved this was possible because this was all, it's all sort of speculation that we thought we would do, this method would work. And I managed to at least show that with minor changes, I could run the same kernel on both PandaBoard and Raspberry Pi. The main issue was Raspberry Pi, that's Raspberry Pi 1, which is Arm V6 and PandaBoard is Arm V7, and I've got some issues with the cache, so you have to rebuild depending on exactly how you handle cache interactions with the cache. So that's why it's almost the same kernel, not the same kernel. So what does being a platform actually look like is, here's the absolute minimum SNP example I think we can provide. This is actually, this comes from the QMU support. So we've got QMU has a virtual machine type, which uses VertiO for everything, or mostly for devices, many devices at least. So what it does is, we've got a few things here, we've got a function that will map early device memory, mostly things like the UART, it also allows you to use larger pages for devices, which may help with TLE pressure. You've got a function to start the non-boot CPUs, we assume we start boot-a-gon CPU 0, some stage later in the future, we're going to want to start CPUs 1, for example, if there is a CPU 1. And you've got a function to tell the kernel, actually yes, there is a CPU 1, there's a CPU 2, etc. So it tells the kernel the maximum CPU ID and the number of CPUs. And then the kernel can then go and run the previous function, the start AP for each of these. Start AP, if this return tells the kernel yes, there's more than one CPU in the system. And there's a little bit of information down here at the bottom, which I'll explain a little bit later. So, yeah, platform is, so it's K-Obj-Base, so if you've done any device driver programming on FreeBSD, you'll be familiar with that. And because it's K-Obj-Base, we've got all these other functions, which you can implement, but you don't have to. So we can do things where, just after you've proved which, and you've got a successful probe of saying, yes, this object is for this piece of hardware, we can run the attach function. So the same simulator in the attach in the device. We've got things like finding the last address of kernel address base, find, initialising GPIOs, so if you need to handle something just before the console comes up, doing something after the console is ready, and the special one of when it's right at the end, rebooting. And then there is this platform probe, which I say, yeah, don't use. It's only as magic and it's used by the file descriptor table, the other flat and device tree tables. There you go. It's therefore, it handles the magic of figuring out which device, which K-Obj object to use. So I said, you know, there's this magic piece of macro, there's magic macro at the bottom. This handles the magic to figure out which platform object do we actually need to use. So it has, we've got vert, you know, so we're saying this is the QMU virtual machine object descriptor, where we say we will have this vert variable, which just magic means we'll use vert methods variable for it to find the methods. And, you know, we have a human readable name, because you can actually override which object you're planning on using, the kernel wants to use. So if you're wanting to test something, you can override it. They have a zero because we have to have a size of a soft C, even though we don't use it. We probably should just remove that and force it to zero. We have a compatibility string. So in the top of each device tree file, there is a, this device tree is compatible with this hardware. And so we can use this to select exactly which object we're using. And we have this magic one, which is a imaginary, if you've got a delay function, we do a busy loop. And early on when we don't know exactly what hardware we'll have, we need to know how many iterations through this loop we need to of just doing nothing, do we need to iterate for one microsecond. And because this is QMU and time is imaginary, we can just say one is the smallest allowed value. So there's a number of examples of these. We've got other slightly more complex versions if you need multiple different platforms pointing at the same methods. So the IMX6 from Frisco, Nari XP, Corkom users, I think they have three or four different compatibility strings for single core, dual core, quad core. So we can handle multiple options. But then we get to the next problem of, I've sort of briefly touched on device enumeration. We, yeah, so memory mapped devices are genuinely not non enumerable. So there's not an easy, you have to figure out what the base address and what the size of the mapping is somehow. You could potentially create a device at a known location that helps you enumerate, but that hasn't been done. So we have to give it in the constraints. We need to figure out where devices are in memory. The other problem was the kernel had a hard coded list of devices and their location in memory, which makes it difficult if you want to then suddenly change to another piece of hardware which has a different set of devices in different locations. So we could just have the firmware provided to us. When I say firmware, I also mean you could have the kernel build it in, but it makes it a bit less generic in that case, and you can only run on one piece of hardware. So a little bit of history. This was added prior to ARMv6. I believe it was funded by the Foundation 3CMH. So there's some early, some Maval, ARMv5 parts which were... You have a comment? Juniper funded it. Juniper funded. So it was added before ARMv6 to ARM. And there was an early requirement for ARMv6, so all kernel convicts have to use FDT. There's also a large list of every architecture that's not Spark64, I think. That's its optional one. Yes, but I think Juniper might be using it for AMD64. Yes. And we're trying to prod risk-5 people to adopt it as well, so that we don't have to have another format we have to pass in the kernel. So if you're used to ACPI, part of what ACPI has is similar to FDT, and that it has a big list of... Here's some devices that may not be innumerable in it. So if you're... It's more common on ARMv64, I think. Well, at least it's common on ARMv64 that you have this, because it has the same issue where you can't enumerate memory map devices without external information. So here's my very minimal example of FDT, of a DTS. So we have a few bits of information here. We have a unique board in SOC names, so this is what the platform code uses to detect which hardware it's running on. So all it is saying is that's a name value peer, where the values may be an array, which in this case it's an array of two strings. We have some RAM, so that's saying memory starts at that address and is this long. We have a device, and this is my SOC vendor's device, which is in their device. And then we have... As with the RAM, we have an address that the device is located in memory and a number of size of the memory that we should map. So this is... And there may be more information in here. There could be things like interrupts as well, but it's quite difficult to fit too much more onto a single slide. And yeah, so we have this memory range. So this gets compiled into a device blob, so it's a DTB, which is just a binary representation of the same information. So you can then decompile it back into... extract the source out again. And we normally then give this to Uboot to... or UBLD out to pass into the kernel on boot, so that we could have... The kernel then doesn't need to know about exactly what hardware it's running on before it can enumerate the hardware. So this still means you may have a... a hardware Uboot, but that's firmware, so we sort of have to live with that. And so a graphical overview of it is we have... we have some sort of bus, which this is... this could be a memory inside the bus, as it's mapped. And then they could have some... a simple bus inside and some devices. And then a completely separate one, simple bus. So we can have quite complex structures. It's all tree-based. And so you can build up exactly a description of the hardware. And so... writing device drivers, you just need to check with it. Because it came out of... I think it came out of PowerPC on Linux, where they wanted to convert... have a format where you could convert the device tree information to something... this flattened format. So everything is open-firmware based. So it looks like... it looks like open-firmware to the kernel. So we will see things like open-firmware... or if WBus data is okay, it's just checking that the... you haven't disabled that device. And that the device is compatible. So yeah, there's plenty of examples though. So this is just showing how you would attach. So we've got... we've managed to figure out... enumerate hardware. And we've got booted into... at least device... we've booted, we've started enumerating hardware. So we can... this is at the point now we can boot a full kernel. The problem is that we've still got these singleton functions where you can build a kernel and you might be able to run it on a few different, very quite closely related architectures. But you couldn't run it on something with, say, two different timers. Because you've got a new problem of... the delay function needs to be implemented by each timer. So this was... every time you wanted to create... if you wanted to create your own timer, you just said this delay function which weights some number of microseconds for... in a busy loop. So this is... so my solution was we just call back easy. We need to... it just needs to have another function that does this busy loop. We can... each timer then can register. And because you tend to only ever have one timer in your hard... your SAC, we just have a single... you just override whoever was the last to register. They're the current timer. They're the current implementation of delay. And that's... I don't think I've seen any hardware which has... this has even been the problem. We've got two timers that... we may do something like only call this on the first... on the first time you're attaching a timer, but... that seems to work. But, you know, it's... you have to then figure out... the problem is you still have to figure out... you still have to wait this time. So what this is just a quick example of is... you've got some... you've got a... you're figuring out... you've got some amount... number of counts you need to use... and you just have in a loop waiting until the timer is counted up to something large enough. I'm not particularly happy with this method of doing it, but I'm thinking I need to rework it slightly, but I'm not exactly happy with my reworking of having... a function that returns how many count... what the current count of value is. It doesn't work particularly well with a busy... the early boot loop. So I'm still thinking about a better way of doing this. And then we can then use this... so the early... yeah, the early boot loop, this one... is a wee bit more difficult if I wanted to rework it. But you'll notice... so... that delay count is the magic one I pointed out... in the platform definition. You probably... the other issue is we're not entirely sure... well, we need to be able to figure out how to calibrate this value... other than just choosing a random number like a hundred, which seems to be okay for most hardware. Yeah, this... it's... like I said, I'm not necessarily exactly happy with the current code, but it seems to work. And it's given us a chance to at least run on... even inside the family of the SACs, they often have different timers for slightly different... different SACs. But we've still got interrupts... but you're... there's even more of a problem for the issue of singletons... and that we're only ever able to handle a single interrupt controller. And even more so, we're only ever able to... sometimes interrupt controllers are chained. And so we now have to think about that. An example of that was the... the Samsung... the AMV4 Samsung, which I did... I did part of the work to port BBC2, had something like that, where you had an interrupt controller, then on certain interrupts, there were children interrupts, which was a way of a pain to handle. We had some special cases for these. But luckily, we have this new... new framework into the NG. So this was started in 2012 by Jacob, who is for the summer of code. He's sort of... I think he's got a reasonable amount of it working. Then we had a couple of developers working on it to get it into a committable state before his input in 2015. So it's still reasonably new in this, and it's only been in for a few... about two years. It's optional in MIPS. It's required in ARM64, and actually I have made a requirement to ARMV6 about two weeks ago. So it's optional on ARMV4 and 5, if one doesn't remove them. So this one is similar to... with devices, and you suddenly have a tree of interrupt controllers. They don't necessarily need to be children in the device tree, in the device sense. They could be... They can have cross links between an interrupt controller over here and an interrupt controller over there. And it doesn't actually have to be what you think of as an interrupt controller, a program interrupt controller. It could be something, for example, like a GPIO, where you'll rate that you want to have a function run when a GPIO toggles and pretend to just use that as an interrupt. So we've got a few... It's based on UBus, so you create some interfaces. These handle things like......mapping from FDT to internal value, for example, for the map interrupt, or binding assigns interrupts to a CPU. So I'll go through an example. So we... The user inserts her SD card. So this will toggle a GPIO, for example, in this case. The interrupt controller then sends... So the red there is the ARM assembler code. It then calls into our NG, because it needs to do a dispatch somehow. And it turns out in this device there are two possible devices it could use. In case the generic ARM interrupt controller, the JIC is the one that's attached and it's registered as the root controller. We can still build with the board com controller in because we no longer have a requirement for each controller to implement the exact same named function. So the interrupt gets passed into there. And then it passes it down to a GPIO driver. So the JIC driver doesn't actually know... It just knows that you've raised an interrupt on a GPIO. It doesn't know which GPIO, for example. So you may have 32 pins coming in. They all get multiplexed into a single interrupt. So the GPIO driver then is the one that knows exactly how those are... How to figure out which of those pins has actually caused the interrupt. And then this will pass just as a standard interrupt to the MMC driver. So this is... You can sort of see it's a standard flow. It's not an easy flow. These are the things that get called and that simplifies things a lot, I think, for handling of the interrupts in devices where this could be the case. It means we could potentially have a second interrupt JIC chained off the first one, for example, but there is support for that. I don't know of any hardware that, I think we see supports that do that, does that. There could also be other interesting layouts with, for example, with Beehive, where you could have a virtualised interrupt controller. So we need to then send... We've received the interrupt, we need to send it to a Beehive guest. But this code, into an engine, basically lets us write... Yeah, I do it... Move to runtime detection over compile time options. So, yeah, we've got all these bits, but just having lots of bits doesn't really mean you've got a generic kernel. You still have to put it all together. So, this was a UOBSD con last year. I did a test merge of... We've got the virtual kernel which runs on QMU, and we have the all-winner... Actually, it was the all-winner ECMP kernel, sorry, which runs on all-winner hardware, or some all-winner hardware. And I managed to boot... I got booted on QMU, and then Emmanuel got booted on the same... Exactly the same kernel on an all-winner board. So we've proved that we've got, you know, a generic two, isn't it? Once you can boot off two things, you can be done. And then later on in UOBSD con, Emmanuel gave a talk on the support of VBSD on all-winner, mentioned there's no generic kernel in R6 and subversion, so I committed it. In the middle of the talk... So just by the time the talk was done, the talk was out of date. Which I pointed out in the question section at the end. Yeah, so we had at least got something running. We got two kernels, two SoCs running with the same kernel config. Then we... Turns out two is not quite enough. You probably want to run on a few more. I think we've run on all......and V7 all-winner platforms at VBSD supports. We might even run on Pine64. I'm not sure... I'm not sure if anyone's tested it. We may also support that platform. We support the TI parts that we... I'm not sure... I definitely run the AM... AM335X is the Beaglebone Black. That requires... And OMAP4 is PandaBoard. I haven't actually... I've got a PandaBoard burn. I've never actually tested it on there. I just look... We support Raspberry Pi 2 and probably 3. I think people have actually booted that one into 32-bit mode. And we support the Nvidia Tegra. This is not the Tegra 2. This is a newer version of the hardware. As I said earlier, as a way of forcing people to clean up the code, I've now made into an engine a requirement on ARMv6. This will help... This is intended to help get more kernels and more SoC supported than generic. Also, it means that people will go... Hopefully, then, finally remove the old support on ARMv6, at least, for the pre-entry NG code. I've got almost all of the ARMv6 kernel conflicts using Platform... Platform ECMPs, just the ECMP support. My valve's a pain with that. I've got some ideas on how to support that. And many support multi-delay. So I think we're actually getting quite close to just having everything... Every ARMv7 kernel running being generic. Because of the issues, I see there are some differences in cache invalidation between ARMv6 and ARMv7. I don't think Raspberry Pi 1 will be on this list. It would be nice if we wanted to move towards an ARMv7 target arch. Then it definitely won't be. So, really, scripts have been updated. So I think the current snapshots, for everything that's not Beaglebone, for many of the all-in-the-parts of these, are now actually running off generic. So we could therefore then do updates for these, if we had in-package base, for example. And we've updated NanoBSD... Someone should do test Raspberry Pi 2. I think that that was the only one I remember. I think there might be NanoBSD support for Pandaboard as well, which should be updated if there is. I'm really excited about that, so I can do that. Yeah, I keep... One is just saying he maintains it, so he's going to do it. I think it's just a single line change anyway. It's easy. I think once we do that, we can start to remove old kernel configs so that people don't get confused as to which one they should use. There's still a few remaining issues. Not all kernel configs have been converted. There's a few that we either need platform support. As I said, it's been done for most, but it's not quite there yet for all of them. It's either a multi-delay or they just haven't been... We have got that support, but they just haven't been included in generic and tested. Old versions of UBoot don't necessarily support generic very well. They often assume a few things about how they're booting or the missing API or EFI support. So people therefore boot their kernels directly. If you're doing that, you're not passing in the major structure that loads of data passes in to the kernel so the kernel can't find its device tree. So you have to build the device tree into the kernel. Meno, don't hard code these. We've already had this problem once. I found this driver, this is a cache controller. I think I might have a fix for this, but I'm not entirely sure. We seem to do lots of magic on each SAC, which is slightly different even though they should probably be the same. I'm not sure, I don't know enough about each why people have chosen random values in their previous SAC configuration for the caching. So quick summary. We've shown generic and R6 as possible. The other BSDs, if you're wanting to do this, come and talk to me. I've worked through many of the issues. It's mostly just engineering. It's not just going through banging on things until they break and then trying to fix them. It would be nice to support more of these SACs and patches are welcome. Any questions? Is this just ahead at the moment? The question is, is this just in head? It is. It's all came in. I only did it last year and we branched early in the middle of last year, I think. We've only done it in head. It's probably going to be only new in 12. My question is related. Should we use the CR2 in S of MV6 to MFC that? Should we use an advantage that MV6 is CR2 to MFC that to 11? Should we just say, MV6 is tier 2. So we can just see it. It's not changing any interfaces. There's just a lot of bits that may... Someone may have to go and track down all the commits that have been made. So there's quite a few different changes in there. Have you given any thoughts on having a generic U-Boot? So generic U-Boot... It's impossible. Apparently. I think what we could do is a generic... We might be able to do a generic image that you then may need to have some install step for U-Boot. Yeah, I have this. So we could do that. We just need to work out making sure we've got the correct gaps for different versions. Some versions of U-Boot you have to put on DD to the right part of an SD card. Some you need to put into a FAT file system. So we could... I think we could do it. It's just not an easy task. So what in the release stream is most compatible with the future generic, like you're going to say, as a testing target into QMU or something like that as an image? So what is most compatible with generic is all the all-winner SoCs that should be currently supported. But if you don't have one, a QMU, there's a nanoBSD script to build it. And we do support... there's a command line in the configuration file to tell you how to run it. The only thing is you need UEFI to download, which you can download from the narrow. Any more? Thank you.
 I'll start at this comments.  I'm going to be not talking about the picture, unfortunately.  This is a lie.  I will be talking about generic arm.  In this case, I mean 32-bit arm,  even though I'm known for 64-bit more recently.  So we've got this problem.  We've got arm and what we want to do is boot on all the things.  And between true and everything.  So quick introduction about me.  I'm now a research associate in the University of Cambridge,  in the computer lab, looking for Robert Watson.  And I've also, up until fairly recently, been sort of still am,  a freelance software engineer.  So none of this was related to either of these positions.  But I've been a free BSD committer for nearly seven years.  Focusing on arm things.  And I'm 64 mainly.  I don't really know anything about Intel.  People keep asking me,  comparing between Intel and ARM.  I say, you have to tell me about Intel first.  I've done these two major projects.  I break everything on everyone's arm binaries,  and then I ran off to ARM 64.  But this is, as I said, this is separate from these.  This is my, I don't feel like doing work,  so I'm doing something fun instead,  and playing with more programming.  I've done, so, a small history on ARM v6.  We started 2011.  So it was, the first ARM v6 CPUs were,  they've been in the market since 2002,  according to the ARM website.  So we were fairly late to this.  And even though ARM v6 existed,  there wasn't really any hardware that was generally available at all.  So it wasn't until ARM v7 came around  that we were really the interest in the port.  Started with a amount of XP  and a few, some TI parts and video.  And it's, you'll notice that,  so these are all ARM v7 CPUs.  So even though it's called ARM v6,  it means ARM v6 in later, in 32 bit mode.  Most of, many of these have actually been removed  or are currently not really building properly.  So some of the old code may have been a wee bit dead,  but these were the original three ARM platforms.  We've imported, so we imported into our version in 2012.  That's been almost, we've had it almost five years.  We have added a few more SoCs.  Here's a few, so you can see the Vodcom chips  are only really in Marsby pies, or for free BSD.  There's some FreeScale, which I think is now in XP,  and soon to be Qualcomm.  Some Samsung parts, some Allwinner parts.  I don't know how many Allwinner parts  we've support, too many to list.  We support Smintel, FPGA parts.  The second one is the Amazon owned company now.  Rockchip, Xilinx, FPGA, QMU, and GIMP5.  So we support a reasonable number.  We're not everything, but we've only got a number of people  working on this, so if we don't want to speed ourselves too  thin, we're supporting every part, every SoC.  The problem is, we don't have a generic,  so what is needed for generic?  We need early hardware configuration.  So we need something, as you boot in,  you need to do a little bit of work fairly early on  just to get to the point where you can get to some common code.  You need to find some sort of device enumeration.  So later on in the kernel, we're booting,  we need to figure out what hardware do we actually have  in this system we're booting on,  because a lot of ARM hardware is not enumable  without some sort of external information.  So ACPI does provide some of this information on Intel,  but also self-enumering buses like PCI Express help with us.  But you can't rely on this being, you know,  for all devices.  And we've got these singleton functions,  where each SoC implemented the same function  in slightly different ways, or even in the same way.  We need to break these a little  and create some new interfaces.  So early hardware is, yeah,  so we're booting the kernel,  we're booting, you've got Uboot, for example,  as a first level bootloader,  we then, that runs the Uboot loader, so UBLDR,  and then jumps into the kernel.  And then we need to do some sort of figuring out, you know,  we're still running in physical addressing at this point.  So we've got this problem of, you know,  the kernels all had a hard-coded physical address  they expected to be loaded in.  It's great, except if you're wanting to run one kernel  on one piece of hardware, and you can guarantee  you're always loading at a fixed address.  But even if you're not wanting to run same kernel  multiple different types of hardware,  just being able to say, you know,  not have to hard code addresses is nice.  So we came up with a solution.  We do some web of runtime detection of what address  are we actually running from,  and then we can build a page tables around that.  And then once we've done that,  we're in the virtual address space,  and we know we've got a virtual address,  so we can force it to be the address we want.  That has some issues, though,  and it's not necessarily an easy thing to do.  But luckily, so what we do,  we have to figure out where we've been loaded,  the physical address space address.  Yeah, finally, we're in physical address space  we've been loaded to.  And then we have to, you know,  build this page table.  So how do we do that?  It turns out ARM is really nice for this,  and that 32-bit ARM, we have a register,  which is, you can just use as a source register.  And you can use it for move, as a source for move,  so you can query what the value of the program counter is.  Or you've got some program counter relative memory accesses.  There is a slight issue with 32-bit ARM  where the program counter doesn't point at the current instruction,  but at what the current, at the start of the pipeline  on early ARM processes would have been.  And they were three-stage pipelines,  that's two instructions after the current instruction  that you're executing.  So it's always, it's a wee bit confusing  if you look at this, and suddenly everything's off  by two instructions.  So that's what, you know, we can,  for a lot of this, we can handle that.  And it's mostly using PC relative,  and the assembler handles,  knowing exactly what the PC will be  at this given instruction.  But then we need to, so we've got,  we can figure out what the virtual address,  the physical address will be,  but we still need to actually build,  we need to figure out, you know,  we need to build this mapping,  because we need to get into virtual address space somehow.  So we have to create these two mappings.  We have to create a large,  I think, I think currently we're building a mapping  for the entirety of the address space.  And then within that, we create a small 64 megabyte mapping  of where it is that the kernel will be running,  ultimately be running from.  But we use the information.  We know the physical address  that we wanted to create the map to.  It's all, it's a wee bit hairy code,  but it all works, and, you know,  once you know a delta between virtual and physical address,  it's relatively easy.  And we can, you know, so we can do this,  and we've, it survives.  It's quite, if you're really interested in actually how we do it  and issues I know in the code,  you can come and talk to me afterwards.  It's a lot of explanation to actually go through this.  So I'm skipping over this a little bit at the moment.  But that's fine.  Okay, so we can now get into, we can start running C code.  But we come to a second problem,  and everyone just copies and pastes the code,  and they made the minor changes.  And so we, the first piece of C code we run  in FeeBee's D on ARM is called Inner's ARM.  I think we had, I'm not sure how many copies of that we had.  I know.  One of things, 50.  I'm not sure if it was quite 50, but it was, it was many.  Too many, you know, any more than one is too many.  And I think we're down to two now, maybe a few more.  So I came up, you know,  because most of the ARMv6 are almost identical, but not quite,  I came up with a, I did this, and I pulled out the common code  and created some new, per-sot, you know,  new functions to handle the per-sic code.  This originally wasn't, it was just C function.  So there's still the problem of singletons.  So we still have, yeah, we need to,  still need to implement these same functions.  But we can, like, you know, it turns out per-pc has already done this.  Yeah, that's how per-pc does it.  We just create a K-Obj based, you know,  FeeBee's D supports multiple device drivers using K-Obj.  And it's a generic thing.  So we can just use it as per-pc does in the early boot code.  And this turns it to be surprisingly easy.  It's, yeah, it gives you support of things like,  even within a, if you've got two very similar SoCs,  you can then even start running a kernel with these between them.  The same kernel on two platforms,  even if it's not entirely generic.  So, yeah, this code is all called from the inner term,  so the very first bit of C code.  I started it in a, I created a project branch called  Specific League as a play on generic arm.  But it did prove that, you know, so this project branch proved  this was possible because this was all,  it's all sort of speculation that we thought we would do,  this method would work.  And I managed to at least show that with minor changes,  I could run the same kernel on both PandaBoard and Raspberry Pi.  The main issue was Raspberry Pi, that's Raspberry Pi 1,  which is Arm V6 and PandaBoard is Arm V7,  and I've got some issues with the cache,  so you have to rebuild depending on exactly how you handle  cache interactions with the cache.  So that's why it's almost the same kernel,  not the same kernel.  So what does being a platform actually look like  is, here's the absolute minimum SNP example  I think we can provide.  This is actually, this comes from the QMU support.  So we've got QMU has a virtual machine type,  which uses VertiO for everything,  or mostly for devices, many devices at least.  So what it does is, we've got a few things here,  we've got a function that will map early device memory,  mostly things like the UART, it also allows you to use  larger pages for devices, which may help with TLE pressure.  You've got a function to start the non-boot CPUs,  we assume we start boot-a-gon CPU 0,  some stage later in the future, we're going to want to start  CPUs 1, for example, if there is a CPU 1.  And you've got a function to tell the kernel,  actually yes, there is a CPU 1, there's a CPU 2, etc.  So it tells the kernel the maximum CPU ID  and the number of CPUs.  And then the kernel can then go and run the previous function,  the start AP for each of these.  Start AP, if this return tells the kernel yes,  there's more than one CPU in the system.  And there's a little bit of information down here at the bottom,  which I'll explain a little bit later.  So, yeah, platform is, so it's K-Obj-Base,  so if you've done any device driver programming on FreeBSD,  you'll be familiar with that.  And because it's K-Obj-Base, we've got all these other functions,  which you can implement, but you don't have to.  So we can do things where, just after you've proved  which, and you've got a successful probe of saying,  yes, this object is for this piece of hardware,  we can run the attach function.  So the same simulator in the attach in the device.  We've got things like finding the last address of kernel address base,  find, initialising GPIOs, so if you need to handle  something just before the console comes up,  doing something after the console is ready,  and the special one of when it's right at the end, rebooting.  And then there is this platform probe, which I say,  yeah, don't use.  It's only as magic and it's used by the file descriptor table,  the other flat and device tree tables.  There you go. It's therefore, it handles the magic of figuring out  which device, which K-Obj object to use.  So I said, you know, there's this magic piece of macro,  there's magic macro at the bottom.  This handles the magic to figure out which platform object  do we actually need to use.  So it has, we've got vert, you know, so we're saying this is the  QMU virtual machine object descriptor,  where we say we will have this vert variable,  which just magic means we'll use vert methods variable  for it to find the methods.  And, you know, we have a human readable name,  because you can actually override which object you're planning  on using, the kernel wants to use.  So if you're wanting to test something, you can override it.  They have a zero because we have to have a size of a soft C,  even though we don't use it.  We probably should just remove that and force it to zero.  We have a compatibility string.  So in the top of each device tree file,  there is a, this device tree is compatible with this hardware.  And so we can use this to select exactly which object we're using.  And we have this magic one, which is a imaginary,  if you've got a delay function, we do a busy loop.  And early on when we don't know exactly what hardware we'll have,  we need to know how many iterations through this loop we need to  of just doing nothing, do we need to iterate for one microsecond.  And because this is QMU and time is imaginary,  we can just say one is the smallest allowed value.  So there's a number of examples of these.  We've got other slightly more complex versions if you need  multiple different platforms pointing at the same methods.  So the IMX6 from Frisco, Nari XP, Corkom users,  I think they have three or four different compatibility strings  for single core, dual core, quad core.  So we can handle multiple options.  But then we get to the next problem of,  I've sort of briefly touched on device enumeration.  We, yeah, so memory mapped devices are genuinely not non enumerable.  So there's not an easy, you have to figure out what the base address  and what the size of the mapping is somehow.  You could potentially create a device at a known location  that helps you enumerate, but that hasn't been done.  So we have to give it in the constraints.  We need to figure out where devices are in memory.  The other problem was the kernel had a hard coded list of devices  and their location in memory, which makes it difficult  if you want to then suddenly change to another piece of hardware  which has a different set of devices in different locations.  So we could just have the firmware provided to us.  When I say firmware, I also mean you could have the kernel build it in,  but it makes it a bit less generic in that case,  and you can only run on one piece of hardware.  So a little bit of history.  This was added prior to ARMv6.  I believe it was funded by the Foundation 3CMH.  So there's some early, some Maval, ARMv5 parts which were...  You have a comment?  Juniper funded it.  Juniper funded.  So it was added before ARMv6 to ARM.  And there was an early requirement for ARMv6,  so all kernel convicts have to use FDT.  There's also a large list of every architecture that's not Spark64, I think.  That's its optional one.  Yes, but I think Juniper might be using it for AMD64.  Yes.  And we're trying to prod risk-5 people to adopt it as well,  so that we don't have to have another format we have to pass in the kernel.  So if you're used to ACPI,  part of what ACPI has is similar to FDT,  and that it has a big list of...  Here's some devices that may not be innumerable in it.  So if you're...  It's more common on ARMv64, I think.  Well, at least it's common on ARMv64 that you have this,  because it has the same issue where you can't enumerate memory map devices  without external information.  So here's my very minimal example of FDT, of a DTS.  So we have a few bits of information here.  We have a unique board in SOC names,  so this is what the platform code uses to detect which hardware it's running on.  So all it is saying is that's a name value peer,  where the values may be an array,  which in this case it's an array of two strings.  We have some RAM,  so that's saying memory starts at that address and is this long.  We have a device,  and this is my SOC vendor's device, which is in their device.  And then we have...  As with the RAM,  we have an address that the device is located in memory  and a number of size of the memory that we should map.  So this is...  And there may be more information in here.  There could be things like interrupts as well,  but it's quite difficult to fit too much more onto a single slide.  And yeah, so we have this memory range.  So this gets compiled into a device blob, so it's a DTB,  which is just a binary representation of the same information.  So you can then decompile it back into... extract the source out again.  And we normally then give this to Uboot to...  or UBLD out to pass into the kernel on boot,  so that we could have...  The kernel then doesn't need to know about exactly what hardware it's running on  before it can enumerate the hardware.  So this still means you may have a...  a hardware Uboot, but that's firmware,  so we sort of have to live with that.  And so a graphical overview of it is we have...  we have some sort of bus, which this is...  this could be a memory inside the bus, as it's mapped.  And then they could have some... a simple bus inside  and some devices.  And then a completely separate one, simple bus.  So we can have quite complex structures.  It's all tree-based.  And so you can build up exactly a description of the hardware.  And so...  writing device drivers, you just need to check with it.  Because it came out of...  I think it came out of PowerPC on Linux,  where they wanted to convert... have a format where  you could convert the device tree information  to something... this flattened format.  So everything is open-firmware based.  So it looks like... it looks like open-firmware to the kernel.  So we will see things like open-firmware...  or if WBus data is okay,  it's just checking that the...  you haven't disabled that device.  And that the device is compatible.  So yeah, there's plenty of examples though.  So this is just showing how you would attach.  So we've got... we've managed to figure out...  enumerate hardware.  And we've got booted into... at least device...  we've booted, we've started enumerating hardware.  So we can... this is at the point now we can boot a full kernel.  The problem is that we've still got these singleton functions  where you can build a kernel and you might be able to run it on  a few different, very quite closely related architectures.  But you couldn't run it on something with,  say, two different timers.  Because you've got a new problem of...  the delay function needs to be implemented by each timer.  So this was...  every time you wanted to create...  if you wanted to create your own timer,  you just said this delay function which weights some number of microseconds  for... in a busy loop.  So this is...  so my solution was we just call back easy.  We need to... it just needs to have another function that  does this busy loop.  We can... each timer then can register.  And because you tend to only ever have one timer in your hard...  your SAC, we just have a single...  you just override whoever was the last to register.  They're the current timer.  They're the current implementation of delay.  And that's... I don't think I've seen any hardware which has...  this has even been the problem.  We've got two timers that...  we may do something like only call this on the first...  on the first time you're attaching a timer, but...  that seems to work.  But, you know, it's...  you have to then figure out...  the problem is you still have to figure out...  you still have to wait this time.  So what this is just a quick example of is...  you've got some... you've got a...  you're figuring out...  you've got some amount... number of counts you need to use...  and you just have in a loop waiting until the timer is counted up  to something large enough.  I'm not particularly happy with this method of doing it,  but I'm thinking I need to rework it slightly,  but I'm not exactly happy with my reworking of having...  a function that returns how many count...  what the current count of value is.  It doesn't work particularly well with a busy...  the early boot loop.  So I'm still thinking about a better way of doing this.  And then we can then use this...  so the early... yeah, the early boot loop, this one...  is a wee bit more difficult if I wanted to rework it.  But you'll notice... so...  that delay count is the magic one I pointed out...  in the platform definition.  You probably... the other issue is we're not entirely sure...  well, we need to be able to figure out how to calibrate this value...  other than just choosing a random number like a hundred,  which seems to be okay for most hardware.  Yeah, this... it's...  like I said, I'm not necessarily exactly happy with the current code,  but it seems to work.  And it's given us a chance to at least run on...  even inside the family of the SACs,  they often have different timers for slightly different...  different SACs.  But we've still got interrupts...  but you're...  there's even more of a problem for the issue of singletons...  and that we're only ever able to handle a single interrupt controller.  And even more so, we're only ever able to...  sometimes interrupt controllers are chained.  And so we now have to think about that.  An example of that was the...  the Samsung... the AMV4 Samsung,  which I did...  I did part of the work to port BBC2,  had something like that,  where you had an interrupt controller,  then on certain interrupts, there were children interrupts,  which was a way of a pain to handle.  We had some special cases for these.  But luckily, we have this new...  new framework into the NG.  So this was started in 2012 by Jacob,  who is for the summer of code.  He's sort of...  I think he's got a reasonable amount of it working.  Then we had a couple of developers working on it  to get it into a committable state  before his input in 2015.  So it's still reasonably new in this,  and it's only been in for a few... about two years.  It's optional in MIPS.  It's required in ARM64,  and actually I have made a requirement to ARMV6  about two weeks ago.  So it's optional on ARMV4 and 5,  if one doesn't remove them.  So this one is similar to...  with devices,  and you suddenly have a tree of interrupt controllers.  They don't necessarily need to be children  in the device tree, in the device sense.  They could be...  They can have cross links between an interrupt controller over here  and an interrupt controller over there.  And it doesn't actually have to be  what you think of as an interrupt controller,  a program interrupt controller.  It could be something, for example, like a GPIO,  where you'll rate that you want to  have a function run when a GPIO toggles  and pretend to just use that as an interrupt.  So we've got a few...  It's based on UBus, so you create some interfaces.  These handle things like... ...mapping from FDT to internal value,  for example, for the map interrupt,  or binding assigns interrupts to a CPU.  So I'll go through an example.  So we...  The user inserts her SD card.  So this will toggle a GPIO, for example, in this case.  The interrupt controller then sends...  So the red there is the ARM assembler code.  It then calls into our NG,  because it needs to do a dispatch somehow.  And it turns out in this device  there are two possible devices it could use.  In case the generic ARM interrupt controller,  the JIC is the one that's attached  and it's registered as the root controller.  We can still build with the board com controller in  because we no longer have a requirement  for each controller to implement  the exact same named function.  So the interrupt gets passed into there.  And then it passes it down to a GPIO driver.  So the JIC driver doesn't actually know...  It just knows that you've raised an interrupt on a GPIO.  It doesn't know which GPIO, for example.  So you may have 32 pins coming in.  They all get multiplexed into a single interrupt.  So the GPIO driver then is the one that knows  exactly how those are...  How to figure out which of those pins  has actually caused the interrupt.  And then this will pass just as a standard interrupt  to the MMC driver.  So this is... You can sort of see it's a standard flow.  It's not an easy flow.  These are the things that get called  and that simplifies things a lot, I think,  for handling of the interrupts in devices  where this could be the case.  It means we could potentially have a second interrupt  JIC chained off the first one, for example,  but there is support for that.  I don't know of any hardware that,  I think we see supports that do that, does that.  There could also be other interesting layouts  with, for example, with Beehive,  where you could have a virtualised interrupt controller.  So we need to then send... We've received the interrupt,  we need to send it to a Beehive guest.  But this code, into an engine, basically lets us write...  Yeah, I do it...  Move to runtime detection over compile time options.  So, yeah, we've got all these bits,  but just having lots of bits  doesn't really mean you've got a generic kernel.  You still have to put it all together.  So, this was a UOBSD con last year.  I did a test merge of...  We've got the virtual kernel which runs on QMU,  and we have the all-winner...  Actually, it was the all-winner ECMP kernel,  sorry, which runs on all-winner hardware,  or some all-winner hardware.  And I managed to boot... I got booted on QMU,  and then Emmanuel got booted on the same...  Exactly the same kernel on an all-winner board.  So we've proved that we've got, you know,  a generic two, isn't it?  Once you can boot off two things, you can be done.  And then later on in UOBSD con,  Emmanuel gave a talk on the support of VBSD on all-winner,  mentioned there's no generic kernel in R6 and subversion,  so I committed it.  In the middle of the talk...  So just by the time the talk was done,  the talk was out of date.  Which I pointed out in the question section at the end.  Yeah, so we had at least got something running.  We got two kernels, two SoCs running with the same kernel config.  Then we...  Turns out two is not quite enough.  You probably want to run on a few more.  I think we've run on all... ...and V7 all-winner platforms at VBSD supports.  We might even run on Pine64.  I'm not sure...  I'm not sure if anyone's tested it.  We may also support that platform.  We support the TI parts that we...  I'm not sure...  I definitely run the AM...  AM335X is the Beaglebone Black.  That requires...  And OMAP4 is PandaBoard.  I haven't actually...  I've got a PandaBoard burn.  I've never actually tested it on there.  I just look...  We support Raspberry Pi 2 and probably 3.  I think people have actually booted that one into 32-bit mode.  And we support the Nvidia Tegra.  This is not the Tegra 2.  This is a newer version of the hardware.  As I said earlier,  as a way of forcing people to clean up the code,  I've now made into an engine a requirement on ARMv6.  This will help...  This is intended to help get more kernels and more SoC supported than generic.  Also, it means that people will go...  Hopefully, then, finally remove the old support on ARMv6, at least,  for the pre-entry NG code.  I've got almost all of the ARMv6 kernel conflicts using Platform...  Platform ECMPs, just the ECMP support.  My valve's a pain with that.  I've got some ideas on how to support that.  And many support multi-delay.  So I think we're actually getting quite close to just having everything...  Every ARMv7 kernel running being generic.  Because of the issues, I see there are some differences in cache invalidation  between ARMv6 and ARMv7.  I don't think Raspberry Pi 1 will be on this list.  It would be nice if we wanted to move towards an ARMv7 target arch.  Then it definitely won't be.  So, really, scripts have been updated.  So I think the current snapshots, for everything that's not Beaglebone,  for many of the all-in-the-parts of these,  are now actually running off generic.  So we could therefore then do updates for these,  if we had in-package base, for example.  And we've updated NanoBSD...  Someone should do test Raspberry Pi 2.  I think that that was the only one I remember.  I think there might be NanoBSD support for Pandaboard as well,  which should be updated if there is.  I'm really excited about that, so I can do that.  Yeah, I keep...  One is just saying he maintains it, so he's going to do it.  I think it's just a single line change anyway. It's easy.  I think once we do that, we can start to remove old kernel configs  so that people don't get confused as to which one they should use.  There's still a few remaining issues.  Not all kernel configs have been converted.  There's a few that we either need platform support.  As I said, it's been done for most, but it's not quite there yet for all of them.  It's either a multi-delay or they just haven't been...  We have got that support, but they just haven't been included in generic and tested.  Old versions of UBoot don't necessarily support generic very well.  They often assume a few things about how they're booting  or the missing API or EFI support.  So people therefore boot their kernels directly.  If you're doing that, you're not passing in the major structure  that loads of data passes in to the kernel  so the kernel can't find its device tree.  So you have to build the device tree into the kernel.  Meno, don't hard code these. We've already had this problem once.  I found this driver, this is a cache controller.  I think I might have a fix for this, but I'm not entirely sure.  We seem to do lots of magic on each SAC, which is slightly different  even though they should probably be the same.  I'm not sure, I don't know enough about each  why people have chosen random values in their previous SAC configuration for the caching.  So quick summary.  We've shown generic and R6 as possible.  The other BSDs, if you're wanting to do this,  come and talk to me.  I've worked through many of the issues.  It's mostly just engineering.  It's not just going through banging on things until they break  and then trying to fix them.  It would be nice to support more of these SACs  and patches are welcome.  Any questions?  Is this just ahead at the moment?  The question is, is this just in head?  It is.  It's all came in.  I only did it last year  and we branched early in the middle of last year, I think.  We've only done it in head.  It's probably going to be only new in 12.  My question is related.  Should we use the CR2 in S of MV6 to MFC that?  Should we use an advantage that MV6 is CR2 to MFC that to 11?  Should we just say, MV6 is tier 2.  So we can just see it.  It's not changing any interfaces.  There's just a lot of bits that may...  Someone may have to go and track down all the commits that have been made.  So there's quite a few different changes in there.  Have you given any thoughts on having a generic U-Boot?  So generic U-Boot...  It's impossible.  Apparently.  I think what we could do is a generic...  We might be able to do a generic image that you then may need to have some install step for U-Boot.  Yeah, I have this.  So we could do that. We just need to work out making sure we've got the correct gaps for different versions.  Some versions of U-Boot you have to put on DD to the right part of an SD card.  Some you need to put into a FAT file system.  So we could... I think we could do it.  It's just not an easy task.  So what in the release stream is most compatible with the future generic, like you're going to say,  as a testing target into QMU or something like that as an image?  So what is most compatible with generic is all the all-winner SoCs that should be currently supported.  But if you don't have one, a QMU, there's a nanoBSD script to build it.  And we do support... there's a command line in the configuration file to tell you how to run it.  The only thing is you need UEFI to download, which you can download from the narrow.  Any more?  Thank you.
Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_0_755.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_756_839.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_840_923.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_924_2171.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_2172_2543.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_2544_4799.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_4800_6131.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_6132_6611.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_6612_7331.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_7332_9635.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_9636_11135.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_11136_17387.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_17388_19931.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_19932_20759.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_20760_21455.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_21456_22523.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_22524_22655.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_22656_30527.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_30528_31727.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_31728_41495.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_41496_41675.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_41676_41759.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_41760_41855.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_41856_42311.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_42312_43499.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_43500_48791.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_48792_49163.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_49164_49319.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_49320_50339.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_50340_50855.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_50856_50915.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_50916_51011.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_51012_51059.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_51060_51239.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_51240_52739.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_52740_53039.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_53040_53123.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_53124_53387.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_53388_53735.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_53736_54167.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54168_54227.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54228_54323.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54324_54407.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54408_54527.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54528_54575.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54576_54827.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54828_54875.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_54876_55223.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_55224_56783.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_56784_61979.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_61980_69119.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_69120_69515.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_69516_69671.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_69672_70067.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_70068_70895.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_70896_71207.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_71208_74003.png Andrew_Turner__GENERIC_ARM_-_BSDCan_2017_gkYjvrFvPJ0_74004_74722.png