Compiling the Linux kernel with LLVM tools
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Desaulniers, Nick
Wendling, Bill
The Linux kernel codebase has co-evolved with GCC and binutils over its lifetime, but LLVM is now capable of producing production ready Linux kernels. Come learn what some of the challenges were in porting the codebase from GCC/binutils to Clang/LLVM, and what the latest efforts are in both codebases to work towards this goal. The Linux kernel codebase has co-evolved with GCC and binutils over its lifetime, but LLVM is now capable of producing production ready Linux kernels. Come learn what some of the challenges were in porting the codebase from GCC/binutils to Clang/LLVM, and what the latest efforts are in both codebases to work towards this goal. We plan to cover some of the previous challenges, current challenges, issue tracker, various continuous integration setups, topics of future interest, usage in production, and how folks interested in helping out can participate. This talk will be from a kernel developer's (slash recent LLVM contributor's) perspective.
 Hi everyone, my name is Nick. I work on the Linux kernel and the LLVM compiler project at Google in Mountain View, California. Hi, my name is Bill and I also work on Linux, but like on the prod kernel side, and also with LLVM as well. So some of the goals of this project, I would say that whenever you have a maybe a newish compiler that you're pointing at, at a code base that hasn't used that compiler before, kind of both the compiler and the code base stand to benefit from that. So on the source side of things, the Linux kernel benefits by getting kind of an additional warning coverage. I would say there's quite a few patches that we've sent, as far as just trying to drive the warning count when compiling the kernel with Clang to 0. So we're still have work to do there, but we've already upstreamed over a few hundreds of patches related to that. Trying to reduce undefined behavior in the kernel, there's not a large amount of it, but there's a lot of code. So you always have issues there. The earlier talk on static analysis is actually kind of why I got into this whole project was I was just trying to run scan build on the compiler. Unfortunately, static analysis needs to know how exactly are things defined via the pre-pre-processor, and what symbols get included or not via dash D flags and stuff. So you basically need to be able to compile the source code in the first place to do then the static analysis. So I basically got into this rabbit hole of, let's get it building first and then we can go do the static analysis later. So maybe at some point in the future, I'll be able to revisit that. But some of the other stuff I'm really excited about is the thread safety annotations. We use quite heavily in Google 3 C++ code for statically verifying thread safety. You can annotate your code and say, you must acquire this mutex or these mutexes in this order to protect these members of the struct or point to values and stuff like that. Otherwise, the dynamic analyses via like address sanitizer, undefined behavior sanitizer, thread sanitizer all have kernel variants as well that are implemented in LLVM are really nice. There's some cool new-ish stuff that we're looking to do in the kernel. A lot of issues, we're still working through a lot of them. But things like link time optimization and even lately, there's been some research post link optimization stuff that we're playing around with a little bit on our kernels. And ideally, we want to lower the switching cost between compilers. Just like going back to economics, you have this notion of substitute goods. And in order to have substitute compilers, you need to lower the friction between swapping them in and out. And so that puts this constraint on client that it needs to compile almost the same code as GCC. Not exactly, but you can't even compete. You can't come to the table unless you can compile the same code. So if we can lower the cost of switching compilers, that makes it easy for other people to then report bugs in the compiler and try things out and see how it works for them. For improving LLVM, we basically have a whole brand new customer to both give us feature requests and help us figure out what features we're missing from either the C language standard or the various new extensions. We can't compete with GCC if we can't build the same code kind of thing. So that's kind of a non-starter. And then finally, just having more and more code to throw at your compiler helps you find more compiler bugs. So some of the things that we're trying to do is actually create Linux distributions that are entirely built with Clang and LLVM. So Android is in the process of moving there. If you have a pixel phone, those are already all entirely built with Clang, even the kernel. So we're in the process of working with Android OEMs and vendors now. I'm getting the whole ecosystem moved over to Clang-built kernels as well. Chrome OS is entirely all Clang-built as well. OpenMandriva is more of, I guess, traditional Linux packages and is a project working through trying to get all of that code ported over and buildable with Clang. And then Bill will talk a little bit about some of the work he's been doing on Google's production servers that are serving your traffic when you visit various Google properties. So the earliest history I could find was there's some project called LLL project that has a few commits in 2011. I don't really know too much about it, but that's like some proof that someone here was trying to build the Linux kernel with Clang. So they have patches on top of a 2.6 kernel, which is like, I don't know, ancient history to me kind of thing. Then from 2012 to 2016 was the LLVM Linux project. They have a large set of patches that is actually a lot of the work that I started doing was kind of based like finding their patches and applying those, tracing them, cleaning them up, and then in the process kind of upstreaming them as well. And they did a lot of really good work and found a lot of kind of standing issues in Clang and LLVM. I think part of the issue that they were having was both upstreaming of patches into the kernel itself and then also getting fixes in on the Clang and LLVM side of things. So around 2016, my coworker Greg Hackman and I were looking into kind of getting this up and running on the Google Pixel, right? I was looking into it for static analysis of the kernels. And around that time, Greg came to me and was like, hey, I got this up and running, like it's building, can you test and see if it's booting kind of thing. And at that same time, I noticed Matthias Kalki on the Chrome OS kernel team submitting fixes upstream to LKML for like fixing warnings reported by Clang. And so I kind of went over to Matthias and showed him like what we were trying to do and we started kind of collaborating on this stuff and we kind of figured out like, hey, there's a lot of really good patches from the LLVM Linux project. We should actually get those upstreamed or work through whatever issues upstream maintainers have and get that all up and running. So we actually had it up and running for Pixel 1, but due to the way feature releases and product cycles work, we were asked to kind of like give it more soak time. So we ended up shipping it and Pixel 2 was the first device. I think we used Clang 4 to build the full is an ARM64 Linux kernel for that. So that was the first one, then 2018 Chrome OS. I have two links in there. The first one's like, I think in March was they started flipping it on for their various LTS kernels. And then by October flipped it to be the default compiler. So all kernels going forward for Chrome OS are built with Clang as well. In 2018, for Pixel 3, we got LTO working in the kernel and then control flow integrity analysis up and running. That CFI helps with trying to prevent rock chains in the kernel. And then so 2019, what are we working on? I would say LLD support is imminent, at least for ARM64. On the day I was flying out here, project manager was kind of bashing me over the head saying like, how come these patches aren't in yet? And I'm like, oh, I got to clean up the commit messages. And there's still some work I need to do to properly kind of upstream them and get them reviewed properly kind of thing. But it's there, it's working a little bit more work for X86, but ARM64 is ready to go. As I'm go to the patches are posted. This is a feature that's used in the Linux kernel. X86 kind of requires it. So support patches up are up for both the Clang side, the LLVM side. I have been hammering on them nonstop in the past two weeks. Using C reduce, I'll talk about later to basically find bugs in the implementation and make sure we have a high quality implementation before it lands in a bug free one. I would say Clang's integrated assembler. So LLVM tools, right, has a LLVM has a lot of kind of substitutes, not just for the compiler GCC, but also for bin utils. So the assembler and the linker and then NM and Redelph and all these other, like, obj tool, obj dump, strip kind of thing. So there's definitely a long tail of all the tools. I would say integrated assembler is the one I'm most worried about. I think has basically the longest tail of things that we need. We're in the process of moving all of Android over. I'm working with OEMs on issues that they're seeing with it. Prodkernel, which Bill will talk a little bit more about. The other part of CFI is shadow call stack is another part of the, the rock chain prevention. Third safety analysis, trying to get an intern this summer to help out, see if we can get this working in the kernel because that would be nice. A lot of the bugs that I was fixing when I worked primarily on the Nexus and Pixel kernel team came down to like concurrency bugs in third party drivers. Auto FDO is kind of like the next generation of PGO. We want to do kind of low overhead sampling for ARM that requires ETM or on x86, it's Glass Branch Records. Then Bolt is like this cool thing that just came out of Facebook research. It's post link optimization. There's some other things people are doing, trying to see can we move that into the compiler or does it need to be like done after the link stage and one of the issues there. And then trying to, trying to get Clang integrated into the upstream Linux kernel continuous integrations system. So kernel CI, I would say support for Clang is imminent. They just rearchitected it to support multiple different compilers in order to report bugs or regressions in different versions of GCC. And then for them now it's generic enough to just add Clang to it. Zero day bot, we were like this close from getting it integrated and then x86 maintainers like forced the use of Asm go to the week I was talking with the zero day bot team. So that was a little unfortunate. But once we have Asm go to landed wheels start up those talks again. If you want to try it out, usually when you build your kernel, you end up doing like some, you play around with your kernel configuration, like a make local mod config is kind of the basic target. But make has variables you can set a command line, you set cc equals Clang and you're off to the races. You can set LD is equal to LD dot LLD to start trying to link. There's some issues I need to fix upstream, but that's going to be the command, right, how to invoke LLD. And then when you cross compile one of the things I like about kind of Clang is kind of the default build of Clang is has all of the different back ends built into it. So with bin utils and GCC, you typically install like a cross tool version that has a prefix in front of it. And so because we're not using clang's integrated assembler, we still kind of shell out to GNU AS. Then when you're cross compiling, you need to know that target triple. So if you were going to cross compile for arm 64, which I do commonly for my x86 workstation, those are the environmental variables you set kind of thing to do the cross compile. These are very rough measurements. I don't know if GitGrep supports Perl, RegX's probably would have been nicer, but these are like very rough counts of commit messages that mention Clang or LLVM in the kernel. Probably lots of the ones that just mentioned Clang may be related to EVPF. I didn't really do any very scientific measurements there. But then LLVM as well has quite a few commits now in it that say like, hey, this is something that we fixed or implemented because the Linux kernel is making use of this. My turn. Hi. So first off, I'm going to apologize. I'm finding a cold. So I may start coughing. So the production kernel is basically a different beast from what he's working on, which is Android. The Android tends to work more towards the top of tree Linux while the production kernel has various different, what do you call them, long term. Different LTS branches you're based off of? Yeah, LTS branches. So the one that I'm using is kind of an ancient one. I think it's like pre 4.5. But anyway, it's all kind of a more of an experimental type situation right now. Well, asterisks, it's more than just experimental. We really want to do it, but we need to get like performance there, for instance. And I'll go into that in a bit here. So what I did is I went to the team about a year ago, and luckily the Android team had already done LTO on Linux. And so I was able to use their patches and get our kernel to compile with Linux. I'm sorry, with LTO. Basically, it just took a bunch of changes in the build system. Mostly it has to do with the fact that during LTO you generate LVM IR files instead of ELF files. So you really don't generate ELF files until you start linking the VM Linux.o file. And that's when you do all of the LTO optimizations. So the built-in.o files that are generated are not really, well, they're just not really archives at all. So basically what you're doing is you're creating a thin archive, so that just lists all of the.o files that you want to then shove to the linker. And then the linker will take all of those and do its magic and create the.o file. I started by using the gold linker because that's what you're supposed to use. And then in order to get the actual final VM Linux, I linked it with the BFD linker. There is some issues, however, with the gold linker and our teams internally at Google no longer support the gold linker. So basically they said use LLD and so about a month ago I went ahead and switched us over to LLD. Excuse me. So anyway, like I said, the gold linker had bugs. In fact, it was asserting on me a lot. LLD worked great, however, it turns out that when you use LTO and you're also linking LF.o files with it, the LTO files will basically just be generated as one huge.o file,.o L file. And then the other ones are kind of scattered here. And that wouldn't be necessarily an issue, however, with Linux they have this section called init data. And in there they have a list of init calls and those have to be in a specific order because they're initializing structures and so on before other ones. And we were just not able to replicate that. So I basically had to come up with a kind of a hack to get around it. So first off, this is kind of where you define an init call here. And so what I do is I'm popping it into a section that's named after the file that it occurs in. And then during linking I create a really massive linker script that will go through and reorder all of the sections basically so that they are in the quote unquote correct order as they appear on the command line. This is absolutely horrible. It actually causes the compilation or the linking to increase from about 1.2 minutes to over three. And there are kind of ways I can maybe get around that, but in general it's just kind of bad. So please don't tell anybody. And so now we're going to discuss a few interesting issues that we kind of came across while developing this. So I'll let him go first. So just in the interest of time, I won't get into all these, but the very first one which was interesting was, so if you see this statement here, you're declaring a variable foo. It's a long, long. So we had this issue in Clang on 32-bit hosts where according to their ABI, long, long should be 64 bits. And so you're saying like please use register edx for instance, so you can guess which architecture this is. Please store it in a register kind of thing. The thing that's curious is if you're on a 32-bit host and you're saying use edx, which is a 32-bit register, and you're saying you want a long, long. Well, where's the other 32 bits of that 64-bit variable? Does anyone know? No? Okay, well, basically you have to implement bug for bug compatibility with GCC on how does it choose the next 32-bit register? And if you say like ESP, it'll just like crash. So yeah, that was kind of interesting. One of the bugs that we ran to with AR64 that was kind of clever, the kernel few text code, which is like a synchronization primitive, would explicitly dereference null to see like how does the hardware perform and how does it behave and select this implementation. Otherwise this implementation and AR64 has an explicit zero register, which is nice. So Clang was trying to generate code that said like dereference the XZR is like a zero register. The issue is that there is no invalid encoding for that. So like you can write that in assembly and then your assembler will choke and say like there's no bit pattern to represent this. So that was kind of funny. GNU inline is like, if someone asks you like what are the semantic differences between C89 and C99, like how XZR inline works, that's one. The kernel makes use of this. I wrote the documentation on this and like have the patch in the kernel for this. I'm not really proud of it, but that's something you can go look up if you're interested. Single clashes with the C standard library, it's like header file like, I don't know if you were to pound include stdio or something. And like the kernel itself doesn't have a C runtime. It implements its own kind of helper functions. And then there was like collisions there. Custom calling conventions, VLAIS is a variable length array. And then you can also put a variable length array in the middle of a struct, not at the end of a struct, but in the middle kind of thing was an issue. I think the one that I would be most interested in showing you is this one. So quick interview question. How can you write code that is valid at 02 and not at 00? Think about it. Okay, did anyone get this? Might be hard to see here, but basically we have a static inline function. Inline function has an inline assembly. It has some constraints on it saying this is going to be an immediate value like the number 42. And so when we compile this at 02, everything's semantically valid. But if we don't inline that static inline function, it is no longer semantically valid kind of thing. The kernel is using this in a few places. Coincidentally, everywhere where it uses asm go to is wrapped in this pattern. So it's making testing asm go to very difficult, especially if you want to make sure like inlining or, especially because the initial instance of patches of asm go to skip inlining support. So one of the things that's kind of interesting is if you use attribute always inline, it doesn't mean always inline because the compiler needs to know how to do the inlining. And a lot of times you'll see like fix me to do implement later, not needed for correctness kind of thing. So like these need to be macros if you want it to always be inlined kind of thing. And I don't know, Bill, we'll talk about some of these two other ones in a little bit. Oh, there we go. Oh, cool. So there were a couple of issues. One of them, actually, this would have not been a problem if I was using a more recent kernel because the patch as what was already existing in an upstream kernel. But basically what was happening is we, during one of our tests, the machine would just simply assert. But, and I whittled the test cases down and I found out that like in one of the crypto, yeah, crypto library, our.o files, if you compiled it with O1, everything went fine. If you compiled it with O2, everything crashed. And so it happened, it had to do with inlining actually. And it wasn't a bug in Clang, which is what I thought initially. It was some weird, some weird aligning issues. Basically, because Linux does just horrible things with pointers, it has this thing called SGSetBuff. And what this does is you are basically setting a buffer inside of this structure. And it's going to be aligned at a certain place and so on. So inside of SGSetBuff, it's, the offset here is 12 bit aligned, as you can see. So when it's not inlined, Clang is looking at that and it doesn't really try and calculate the value or anything like that. So it just leaves the 0FFF alone. However, when you do inline it, you get the bottom 4 bits turned into 0s. And because you have this variable length array being defined like this, and S&D are actually defined towards the end, there's one variable afterwards, but it's not allocated any stack. So basically, D here is the last thing on the stack right here. And what was happening is after it set this buffer like this, it would put a whole bunch of data in there, of course. And it would end up overriding the return address because of the misalignment, basically. All the lower bits there would just be cleared out and it would be off by 8 and everything would blow up. So that was kind of an interesting bug to find. But let's see. Oh, I'm sorry. The resolution is basically to specify the stack alignment on the command line, which, like I said, had been fixed by a patch probably a couple of years ago or something like that. But it just hadn't made it into one of our LTS kernels yet. I think that was Matthias' patch? I think so, yeah. You went by my other ones. Okay. He had mentioned earlier that there had been kind of like lacking support for some of the GCC features in Clang. And one of the big ones was built-in constant P. It wasn't that we were getting it wrong. It's just that GCC was, oh, geez, doing what GCC does. I mean, a lot of built-ins aren't super portable and aren't super well-defined in the first place, kind of thing. Well, basically, the definition of what a built-in does for GCC is what does GCC do? And so that's it. So basically, after inlining happens, GCC is still able to say, oh, is this, you know, is the variable, or the argument to the built-in constant P is an integer? Are you, you know, a constant? If it is, we'll just go ahead and say, yes, it is. Otherwise, no. And in the vast majority of cases, this is used to take the slow path as opposed to the quick path. In one of the newer kernels, it turns out that kernel hardening actually kind of fails. You would get this, you know, scary message here without it. So actually, both James Knight and I kind of tackled this. James Knight did the L of MIR transfer, our addition using this is constant intrinsic. And I went ahead and modified the Clang front-end, which is not actually the code base that I'm most used to, but I became used to it because, my God, I had to touch every part of it. So Richard Smith asked me to, one of the main maintainers of Clang asked me to implement a, more or less, a wrapper class called constant expert. And it is the correct way of doing it. It's just the long way of doing it. But the benefit of it is that it's kind of a more natural way of doing it. And also, it allows us to support C++ features in the future. And essentially, what it does is kind of what the name says is, if you are in a constant expression, then you are assuming that you are in a constant context. And that can determine whether you're going to evaluate it early in the Clang front-end or late in the L of MIR, our middle end. And so, yeah, we were able to get past that and now we are, but compatible at least with that and with our DCC. So, and I think you are, that's you. Yeah, so if you're interested in how you might be able to help, so I highly recommend this project. This is how I was able to start kind of contributing to the upstream Linux kernel and to LLVM itself. So we have a bug tracker. I try to like highlight like good first issues. So we're happy to hand out bugs. If you're looking to either start contributing to either one of these code bases, I'm more than happy to help you. Shoot me an email is on the first slide. I'm more than, that's, you know, just my thing is like how do I get people involved in contributing to the open source software that they love and use every day is something that's personally important to me. We're running continuous integration for various architectures. So we have ARM64, X8664, ARMV7, ARMV6, ARMV5, PowerPC64, little endian and PowerPC32 bit running continuously. The mainline Linux kernel, Linux Next, LTS branches 419, 414, 4944. So it's a poor man CI like cron job running every night kind of thing, but for all these various like, I don't know, 30 something targets kind of thing. So you can check that out. It's just running on Travis. We have a link to other talks and other kind of material if you're interested in finding out kind of more about the history and the project and stuff. Godbolt.org is amazing. That's the link I showed you earlier with the code that was only in lineable. That's like, if you want to communicate to a compiler developer, a bug, like send them a Godbolt link. Okay, that is like the shared language that compiler people understand. Like here is a clear case. Here's what GCC does. This is what I want in Clang or here's what I code. And here's how it crashes Clang, right? Kind of thing. C reduce. I just wrote a blog post on it. It's incredible. I have it. I have like at least 10 compiler bugs that I've found with it. You kind of give it a shell script where you put whatever you want in it and then you say like in a source file and it mutates the source file, pairing it down until you have like a nice concise reproducer. So I found like errors at link time. You can say like this script returns a different return code based on if this symbol is found after running an M or not or something on a binary or does it crash the compiler or does a compiler produce strange output or strangest assembly bear is this utility that hooks make for and it kind of spits out a compile commands that can be used in a different file which you can then feed into static analysis tools like scan build Clang static analyzer or was it Clang CPP check, right? It was the other utility. I think that uses it. Or alternatively, the kernel will spit out these.o.cmd files that have all the commands for each translation unit when you build. The issue with that is it only produces them, I believe, if compilation is successful, which we could probably fix that in K build. But that's helpful as well. Just because like when C reducing a bug, you sometimes need to know exactly like what flags were passed to the compiler to reproduce a given issue. And so like getting those flags out of compile commands.json or the.o.cmd file is critical. Same thing with stack analysis is different command line flags will change your translation unit kind of once it's been pre-processed kind of thing. Make sure you grab for the right error message. Yes. Yep. So. So, this is a code where I just say like grep for like a crash and then it paired it down to some like bizarro C syntax thing that like and kind of kind of the Clang's maintainer is like, well, do we need to fuzz this, this, this C front end? Like if it's not valid C code, like if the compiler crashes or not, right, is it really a bug? I don't know. Cool. So, that's all we have. Thank you very much. And we're happy to take questions now this time. Or sorry, I guess it's time up. Oh, yeah, yeah. Yeah. So, the question was around what set of kernel configuration options do we use? The kernel itself has a build system called K build that is highly configurable, right? So, it depends on the distribution, right? Like you'll have distributions of Linux where there's no guarantee what kind of hardware they're going to run on. So, they're going to compile a big kernel image that has tons of drivers in it because someone's going to try to put Debian on their toaster or their faucet or something, right? So, just turn everything on for Android, particularly for Pixel, you know, we have one hardware configuration, right? So, we're going to pair our kernels down and know exactly what we're running, but then, like, we may have out of tree drivers that are problematic and things as well. And so, that's definitely an issue of does it build or not, right? So, someone might come to me and say, like, well, can it build a kernel outright? And my question is usually like, well, what are your configs, right? Because the hard thing with the kernel, like, baseline, what we run in CI is we want to make sure that the dev configs build. So, each architecture has a recommended kind of set of configurations. So, that's kind of what we cover continuously. From there, we try to run all yes config builds every so often. The issue with all yes config, which turns on lots of configs, is the kernel has a lot of configs that are mutually exclusive. And so, like, you get this implementation of this function or that depending on the config. So, all yes doesn't mean, like, all the code in the kernel. So, there's probably code hidden somewhere in the kernel that will crash Clang or Clang can't compile and it's like a matter of finding it. So, the kernel has something called ran config, which kernel CI does run. So, once they have Clang support, they'll probably help us find it. It's just a random coin flip. So, you have to flip that coin a lot to find all the bugs and stuff. They're on fabricator. I link to them from the slides. If not, if you go to our issue tracker, they're like posted over there kind of thing as well. So, if you can pull them down on top of mainline Clang, rebuild Clang, and then try it out on the kernel, you'll run into the issue with the static always inline. I have a patch for X86 that converts those all to macros. For arm64, that rabbit hole got really deep and I never finished that patch. But it's a starting point if you're interested. So, can I build on X86.5? You'll still need the out of tree patch because inlining support is not implemented in the asmgo2 patches yet. So, the plan is to land, yes, that is the plan. Once Clang has support for asmgo2, then we're going to work on getting it to be able to inline basic blocks that have asmgo2 statements in them kind of thing. And then, the goal is theoretically there should be no other bugs, but sometimes you don't have visibility beyond like one bug. So, you're like, yes, I fixed it and then you try it out and then something else is broken. So, I can't promise you, but I think it should work kind of thing after that. Oh, I mean, alternatively like, alternatively you can revert the kernel's use of asmgo2 forcing it kind of thing. So, it's not a real solution, but you know, you still get a working kernel kind of thing. One of the big problems we had in LLVNX was that we had to track three different patch sources. LLVN, all is running, kernel, all is running, and our own patches always flipping between the two. And because we had like just two or three build blocks like X86 and one ARM and one ARM is running like in the KMU, it was hard enough to keep track of all of these three sources in a single configuration. And you guys have like 30 something configuration. How do you find where the bug is in LLVN or in the kernel or in your own patch? So, eventually we're looking to like get help from kernel CI and Zero Day Bot for spotting these regressions. And kernel CI is doing like, they pin the kernel version and then try with different versions of clang like top of tree clang. And like they'll report to us when people break the clang in LLVN trees, which is like pretty frequently, which is kind of frustrating. But it's good that we're getting eyes and coverage and stuff. So usually like I'll kind of forward these reports to LLVN developers saying, hey, you probably already know this. Someone probably poked you, but in case they didn't, you're breaking the Linux kernel right now kind of thing. Then I would say on the kernel side, like we're trying to do like always have patches upstream, have Zero, or like minimize the delta as much as possible so we're not carrying patches around kind of thing. So it's like as soon as we have a fix, it goes upstream and we work with the kernel maintainers to make sure we get a fix and kind of thing. And this is no inline asm thing because there's a lot of inline asm problems. So we disable no integrated AS kind of thing for our builds for now. The kernel makes use of assembly, both external assembly files, but then also the inline assembly, which is even more complicated because it has a whole constraint language with it. And so like kernel abuse extensively of non-standard assembly. They have C code generated by the inline assembly that is then recompiled as an assembly file by the C compiler. Abuse is an accurate term when talking about the code base. Okay, any other questions? What's your plan for plugins? Good question. So the question is about what is the plan for plugins? So the kernel has, I think, checked into the source even like some GCC plugins and then some are maintained externally. So LLVM does have a plugin system. When I think back to the earlier talk today, that was an excellent talk. That was about tooling and how to support kind of this ABI instability in LLVM. I think that that's probably like a good thing to point to as far as like now you have a plugin, but how do you guarantee it works with the given compiler version that a user has, especially if the compiler doesn't have a stable ABI itself. So from a plugin maintenance perspective, it makes my skin crawl a little bit, but it's not impossible. It does have a plugin system. And I think plugins are useful. I think anyone who thinks of reaching for a plugin as a tool needs to sit down and really think hard, should this be implemented as a plugin or in the compiler itself? So for instance, I'll give a case, a recent talk at Linux Conf.U was saying, hey, we use a GCC plugin to default initialize all variables. I think personally, I think that's better implemented just in the compiler. And there's actually discussion about this for C++ and there's actually disagreement within the LLVM community. People saying, oh, you're going to fork C++ and create another dialect. You can't do this. And I feel like saying, well, for C, which development has kind of stopped on, like, can we please have this since we already have our own dialect anyways that we use in the kernel. Like kind of every dash F flag you add. Like to me, this could be another dash F default initialize, right? And there's no need for a plugin. Plugins are great if you want to like hack something up and play with the compiler and get something working. I would say the best case where I say plugins make the most sense is when you have, you need compiler technology for one given project. Because then if you were going to say like to submit this to the compiler vendor and say this is only used in this one code, code base, they're going to tell you to get lost. They're not going to want to maintain this plugin system because whatever, whatever maintenance overhead there is kind of thing. There's only works if the compiler version is fixed, which in GCC is normally what happens. They have GCC five forever. In LLVM, you know, fixing in GCC in LLVM six or LLVM seven doesn't make sense because it just progressed so quickly that you can't get a plugin that working seven don't care its drug. So it's a completely different way of doing it. Okay. Thank you so much everyone. We appreciate that.
 Hi everyone, my name is Nick.  I work on the Linux kernel and  the LLVM compiler project at Google in Mountain View, California.  Hi, my name is Bill and I also work on Linux,  but like on the prod kernel side,  and also with LLVM as well.  So some of the goals of this project,  I would say that whenever you have  a maybe a newish compiler that you're pointing at,  at a code base that hasn't used that compiler before,  kind of both the compiler and  the code base stand to benefit from that.  So on the source side of things,  the Linux kernel benefits by getting kind of  an additional warning coverage.  I would say there's quite a few patches that we've sent,  as far as just trying to drive  the warning count when compiling the kernel with Clang to 0.  So we're still have work to do there,  but we've already upstreamed  over a few hundreds of patches related to that.  Trying to reduce undefined behavior in the kernel,  there's not a large amount of it,  but there's a lot of code.  So you always have issues there.  The earlier talk on static analysis is actually kind of  why I got into this whole project was I was just trying to  run scan build on the compiler.  Unfortunately, static analysis needs to know  how exactly are things defined via the pre-pre-processor,  and what symbols get included or not via dash D flags and stuff.  So you basically need to be able to compile  the source code in the first place to do then the static analysis.  So I basically got into this rabbit hole of,  let's get it building first and then we can go do  the static analysis later.  So maybe at some point in the future,  I'll be able to revisit that.  But some of the other stuff I'm really excited about  is the thread safety annotations.  We use quite heavily in Google 3 C++ code  for statically verifying thread safety.  You can annotate your code and say,  you must acquire this mutex or these mutexes in this order  to protect these members of the struct or point to values and stuff like that.  Otherwise, the dynamic analyses via  like address sanitizer, undefined behavior sanitizer,  thread sanitizer all have kernel variants as well  that are implemented in LLVM are really nice.  There's some cool new-ish stuff that we're looking to do in the kernel.  A lot of issues, we're still working through a lot of them.  But things like link time optimization and even lately,  there's been some research post link optimization stuff  that we're playing around with a little bit on our kernels.  And ideally, we want to lower the switching cost between compilers.  Just like going back to economics,  you have this notion of substitute goods.  And in order to have substitute compilers,  you need to lower the friction between swapping them in and out.  And so that puts this constraint on client  that it needs to compile almost the same code as GCC.  Not exactly, but you can't even compete.  You can't come to the table unless you can compile the same code.  So if we can lower the cost of switching compilers,  that makes it easy for other people to then report bugs in the compiler  and try things out and see how it works for them.  For improving LLVM, we basically have a whole brand new customer  to both give us feature requests  and help us figure out what features we're missing from either  the C language standard or the various new extensions.  We can't compete with GCC if we can't build the same code kind of thing.  So that's kind of a non-starter.  And then finally, just having more and more code to throw at your compiler  helps you find more compiler bugs.  So some of the things that we're trying to do  is actually create Linux distributions that are entirely  built with Clang and LLVM.  So Android is in the process of moving there.  If you have a pixel phone, those are already all entirely built with Clang,  even the kernel.  So we're in the process of working with Android OEMs and vendors now.  I'm getting the whole ecosystem moved over to Clang-built kernels as well.  Chrome OS is entirely all Clang-built as well.  OpenMandriva is more of, I guess, traditional Linux packages  and is a project working through trying to get all of that code  ported over and buildable with Clang.  And then Bill will talk a little bit about some of the work he's been doing  on Google's production servers that are serving your traffic  when you visit various Google properties.  So the earliest history I could find was there's some project called LLL project  that has a few commits in 2011.  I don't really know too much about it,  but that's like some proof that someone here was trying to build the Linux kernel  with Clang.  So they have patches on top of a 2.6 kernel, which is like, I don't know,  ancient history to me kind of thing.  Then from 2012 to 2016 was the LLVM Linux project.  They have a large set of patches that is actually a lot of the work that I started doing  was kind of based like finding their patches and applying those,  tracing them, cleaning them up, and then in the process kind of upstreaming them as well.  And they did a lot of really good work and found a lot of kind of standing issues  in Clang and LLVM.  I think part of the issue that they were having was both upstreaming of patches  into the kernel itself and then also getting fixes in on the Clang and LLVM side of things.  So around 2016, my coworker Greg Hackman and I were looking into kind of getting this up  and running on the Google Pixel, right?  I was looking into it for static analysis of the kernels.  And around that time, Greg came to me and was like, hey, I got this up and running,  like it's building, can you test and see if it's booting kind of thing.  And at that same time, I noticed Matthias Kalki on the Chrome OS kernel team  submitting fixes upstream to LKML for like fixing warnings reported by Clang.  And so I kind of went over to Matthias and showed him like what we were trying to do  and we started kind of collaborating on this stuff and we kind of figured out like,  hey, there's a lot of really good patches from the LLVM Linux project.  We should actually get those upstreamed or work through whatever issues upstream maintainers have  and get that all up and running.  So we actually had it up and running for Pixel 1,  but due to the way feature releases and product cycles work,  we were asked to kind of like give it more soak time.  So we ended up shipping it and Pixel 2 was the first device.  I think we used Clang 4 to build the full is an ARM64 Linux kernel for that.  So that was the first one, then 2018 Chrome OS.  I have two links in there.  The first one's like, I think in March was they started flipping it on for their various LTS kernels.  And then by October flipped it to be the default compiler.  So all kernels going forward for Chrome OS are built with Clang as well.  In 2018, for Pixel 3, we got LTO working in the kernel  and then control flow integrity analysis up and running.  That CFI helps with trying to prevent rock chains in the kernel.  And then so 2019, what are we working on?  I would say LLD support is imminent, at least for ARM64.  On the day I was flying out here, project manager was kind of bashing me over the head  saying like, how come these patches aren't in yet?  And I'm like, oh, I got to clean up the commit messages.  And there's still some work I need to do to properly kind of upstream them  and get them reviewed properly kind of thing.  But it's there, it's working a little bit more work for X86,  but ARM64 is ready to go.  As I'm go to the patches are posted.  This is a feature that's used in the Linux kernel.  X86 kind of requires it.  So support patches up are up for both the Clang side, the LLVM side.  I have been hammering on them nonstop in the past two weeks.  Using C reduce, I'll talk about later to basically find bugs in the implementation  and make sure we have a high quality implementation before it lands in a bug free one.  I would say Clang's integrated assembler.  So LLVM tools, right, has a LLVM has a lot of kind of substitutes,  not just for the compiler GCC, but also for bin utils.  So the assembler and the linker and then NM and Redelph and all these other,  like, obj tool, obj dump, strip kind of thing.  So there's definitely a long tail of all the tools.  I would say integrated assembler is the one I'm most worried about.  I think has basically the longest tail of things that we need.  We're in the process of moving all of Android over.  I'm working with OEMs on issues that they're seeing with it.  Prodkernel, which Bill will talk a little bit more about.  The other part of CFI is shadow call stack is another part of the, the rock chain prevention.  Third safety analysis, trying to get an intern this summer to help out,  see if we can get this working in the kernel because that would be nice.  A lot of the bugs that I was fixing when I worked primarily on the Nexus and Pixel kernel team  came down to like concurrency bugs in third party drivers.  Auto FDO is kind of like the next generation of PGO.  We want to do kind of low overhead sampling for ARM that requires ETM  or on x86, it's Glass Branch Records.  Then Bolt is like this cool thing that just came out of Facebook research.  It's post link optimization.  There's some other things people are doing, trying to see can we move that into the compiler  or does it need to be like done after the link stage and one of the issues there.  And then trying to, trying to get Clang integrated into the upstream Linux kernel  continuous integrations system.  So kernel CI, I would say support for Clang is imminent.  They just rearchitected it to support multiple different compilers in order to report bugs  or regressions in different versions of GCC.  And then for them now it's generic enough to just add Clang to it.  Zero day bot, we were like this close from getting it integrated and then x86 maintainers  like forced the use of Asm go to the week I was talking with the zero day bot team.  So that was a little unfortunate.  But once we have Asm go to landed wheels start up those talks again.  If you want to try it out, usually when you build your kernel, you end up doing like some,  you play around with your kernel configuration, like a make local mod config is kind of the basic target.  But make has variables you can set a command line, you set cc equals Clang and you're off to the races.  You can set LD is equal to LD dot LLD to start trying to link.  There's some issues I need to fix upstream, but that's going to be the command, right, how to invoke LLD.  And then when you cross compile one of the things I like about kind of Clang is kind of the default build of Clang is has all of the different back ends built into it.  So with bin utils and GCC, you typically install like a cross tool version that has a prefix in front of it.  And so because we're not using clang's integrated assembler, we still kind of shell out to GNU AS.  Then when you're cross compiling, you need to know that target triple.  So if you were going to cross compile for arm 64, which I do commonly for my x86 workstation,  those are the environmental variables you set kind of thing to do the cross compile.  These are very rough measurements.  I don't know if GitGrep supports Perl, RegX's probably would have been nicer,  but these are like very rough counts of commit messages that mention Clang or LLVM in the kernel.  Probably lots of the ones that just mentioned Clang may be related to EVPF.  I didn't really do any very scientific measurements there.  But then LLVM as well has quite a few commits now in it that say like, hey, this is something that we fixed or implemented because the Linux kernel is making use of this.  My turn.  Hi.  So first off, I'm going to apologize.  I'm finding a cold.  So I may start coughing.  So the production kernel is basically a different beast from what he's working on, which is Android.  The Android tends to work more towards the top of tree Linux while the production kernel has various different, what do you call them, long term.  Different LTS branches you're based off of?  Yeah, LTS branches.  So the one that I'm using is kind of an ancient one.  I think it's like pre 4.5.  But anyway, it's all kind of a more of an experimental type situation right now.  Well, asterisks, it's more than just experimental.  We really want to do it, but we need to get like performance there, for instance.  And I'll go into that in a bit here.  So what I did is I went to the team about a year ago, and luckily the Android team had already done LTO on Linux.  And so I was able to use their patches and get our kernel to compile with Linux.  I'm sorry, with LTO.  Basically, it just took a bunch of changes in the build system.  Mostly it has to do with the fact that during LTO you generate LVM IR files instead of ELF files.  So you really don't generate ELF files until you start linking the VM Linux.o file.  And that's when you do all of the LTO optimizations.  So the built-in.o files that are generated are not really, well, they're just not really archives at all.  So basically what you're doing is you're creating a thin archive, so that just lists all of the.o files that you want to then shove to the linker.  And then the linker will take all of those and do its magic and create the.o file.  I started by using the gold linker because that's what you're supposed to use.  And then in order to get the actual final VM Linux, I linked it with the BFD linker.  There is some issues, however, with the gold linker and our teams internally at Google no longer support the gold linker.  So basically they said use LLD and so about a month ago I went ahead and switched us over to LLD.  Excuse me.  So anyway, like I said, the gold linker had bugs. In fact, it was asserting on me a lot.  LLD worked great, however, it turns out that when you use LTO and you're also linking LF.o files with it, the LTO files will basically just be generated as one huge.o file,.o L file.  And then the other ones are kind of scattered here.  And that wouldn't be necessarily an issue, however, with Linux they have this section called init data.  And in there they have a list of init calls and those have to be in a specific order because they're initializing structures and so on before other ones.  And we were just not able to replicate that.  So I basically had to come up with a kind of a hack to get around it.  So first off, this is kind of where you define an init call here.  And so what I do is I'm popping it into a section that's named after the file that it occurs in.  And then during linking I create a really massive linker script that will go through and reorder all of the sections basically so that they are in the quote unquote correct order as they appear on the command line.  This is absolutely horrible. It actually causes the compilation or the linking to increase from about 1.2 minutes to over three.  And there are kind of ways I can maybe get around that, but in general it's just kind of bad.  So please don't tell anybody.  And so now we're going to discuss a few interesting issues that we kind of came across while developing this.  So I'll let him go first.  So just in the interest of time, I won't get into all these, but the very first one which was interesting was, so if you see this statement here, you're declaring a variable foo.  It's a long, long.  So we had this issue in Clang on 32-bit hosts where according to their ABI, long, long should be 64 bits.  And so you're saying like please use register edx for instance, so you can guess which architecture this is.  Please store it in a register kind of thing.  The thing that's curious is if you're on a 32-bit host and you're saying use edx, which is a 32-bit register, and you're saying you want a long, long.  Well, where's the other 32 bits of that 64-bit variable? Does anyone know?  No?  Okay, well, basically you have to implement bug for bug compatibility with GCC on how does it choose the next 32-bit register?  And if you say like ESP, it'll just like crash.  So yeah, that was kind of interesting.  One of the bugs that we ran to with AR64 that was kind of clever, the kernel few text code, which is like a synchronization primitive, would explicitly dereference null to see like how does the hardware perform and how does it behave and select this implementation.  Otherwise this implementation and AR64 has an explicit zero register, which is nice.  So Clang was trying to generate code that said like dereference the XZR is like a zero register.  The issue is that there is no invalid encoding for that.  So like you can write that in assembly and then your assembler will choke and say like there's no bit pattern to represent this.  So that was kind of funny.  GNU inline is like, if someone asks you like what are the semantic differences between C89 and C99, like how XZR inline works, that's one.  The kernel makes use of this.  I wrote the documentation on this and like have the patch in the kernel for this.  I'm not really proud of it, but that's something you can go look up if you're interested.  Single clashes with the C standard library, it's like header file like, I don't know if you were to pound include stdio or something.  And like the kernel itself doesn't have a C runtime.  It implements its own kind of helper functions.  And then there was like collisions there.  Custom calling conventions, VLAIS is a variable length array.  And then you can also put a variable length array in the middle of a struct, not at the end of a struct, but in the middle kind of thing was an issue.  I think the one that I would be most interested in showing you is this one.  So quick interview question.  How can you write code that is valid at 02 and not at 00?  Think about it.  Okay, did anyone get this?  Might be hard to see here, but basically we have a static inline function.  Inline function has an inline assembly.  It has some constraints on it saying this is going to be an immediate value like the number 42.  And so when we compile this at 02, everything's semantically valid.  But if we don't inline that static inline function, it is no longer semantically valid kind of thing.  The kernel is using this in a few places.  Coincidentally, everywhere where it uses asm go to is wrapped in this pattern.  So it's making testing asm go to very difficult, especially if you want to make sure like inlining or,  especially because the initial instance of patches of asm go to skip inlining support.  So one of the things that's kind of interesting is if you use attribute always inline,  it doesn't mean always inline because the compiler needs to know how to do the inlining.  And a lot of times you'll see like fix me to do implement later, not needed for correctness kind of thing.  So like these need to be macros if you want it to always be inlined kind of thing.  And I don't know, Bill, we'll talk about some of these two other ones in a little bit.  Oh, there we go.  Oh, cool.  So there were a couple of issues.  One of them, actually, this would have not been a problem if I was using a more recent kernel  because the patch as what was already existing in an upstream kernel.  But basically what was happening is we, during one of our tests, the machine would just simply assert.  But, and I whittled the test cases down and I found out that like in one of the crypto, yeah, crypto library, our.o files,  if you compiled it with O1, everything went fine.  If you compiled it with O2, everything crashed.  And so it happened, it had to do with inlining actually.  And it wasn't a bug in Clang, which is what I thought initially.  It was some weird, some weird aligning issues.  Basically, because Linux does just horrible things with pointers, it has this thing called SGSetBuff.  And what this does is you are basically setting a buffer inside of this structure.  And it's going to be aligned at a certain place and so on.  So inside of SGSetBuff, it's, the offset here is 12 bit aligned, as you can see.  So when it's not inlined, Clang is looking at that and it doesn't really try and calculate the value or anything like that.  So it just leaves the 0FFF alone.  However, when you do inline it, you get the bottom 4 bits turned into 0s.  And because you have this variable length array being defined like this, and S&D are actually defined towards the end,  there's one variable afterwards, but it's not allocated any stack.  So basically, D here is the last thing on the stack right here.  And what was happening is after it set this buffer like this, it would put a whole bunch of data in there, of course.  And it would end up overriding the return address because of the misalignment, basically.  All the lower bits there would just be cleared out and it would be off by 8 and everything would blow up.  So that was kind of an interesting bug to find.  But let's see.  Oh, I'm sorry.  The resolution is basically to specify the stack alignment on the command line,  which, like I said, had been fixed by a patch probably a couple of years ago or something like that.  But it just hadn't made it into one of our LTS kernels yet.  I think that was Matthias' patch?  I think so, yeah.  You went by my other ones.  Okay.  He had mentioned earlier that there had been kind of like lacking support for some of the GCC features in Clang.  And one of the big ones was built-in constant P.  It wasn't that we were getting it wrong.  It's just that GCC was, oh, geez, doing what GCC does.  I mean, a lot of built-ins aren't super portable and aren't super well-defined in the first place, kind of thing.  Well, basically, the definition of what a built-in does for GCC is what does GCC do?  And so that's it.  So basically, after inlining happens, GCC is still able to say, oh, is this, you know,  is the variable, or the argument to the built-in constant P is an integer?  Are you, you know, a constant?  If it is, we'll just go ahead and say, yes, it is.  Otherwise, no.  And in the vast majority of cases, this is used to take the slow path as opposed to the quick path.  In one of the newer kernels, it turns out that kernel hardening actually kind of fails.  You would get this, you know, scary message here without it.  So actually, both James Knight and I kind of tackled this.  James Knight did the L of MIR transfer, our addition using this is constant intrinsic.  And I went ahead and modified the Clang front-end, which is not actually the code base that I'm most used to,  but I became used to it because, my God, I had to touch every part of it.  So Richard Smith asked me to, one of the main maintainers of Clang asked me to implement a,  more or less, a wrapper class called constant expert.  And it is the correct way of doing it.  It's just the long way of doing it.  But the benefit of it is that it's kind of a more natural way of doing it.  And also, it allows us to support C++ features in the future.  And essentially, what it does is kind of what the name says is,  if you are in a constant expression, then you are assuming that you are in a constant context.  And that can determine whether you're going to evaluate it early in the Clang front-end or late in the L of MIR, our middle end.  And so, yeah, we were able to get past that and now we are, but compatible at least with that and with our DCC.  So, and I think you are, that's you.  Yeah, so if you're interested in how you might be able to help, so I highly recommend this project.  This is how I was able to start kind of contributing to the upstream Linux kernel and to LLVM itself.  So we have a bug tracker.  I try to like highlight like good first issues.  So we're happy to hand out bugs.  If you're looking to either start contributing to either one of these code bases, I'm more than happy to help you.  Shoot me an email is on the first slide.  I'm more than, that's, you know, just my thing is like how do I get people involved in contributing to the open source software that they love and use every day is something that's personally important to me.  We're running continuous integration for various architectures.  So we have ARM64, X8664, ARMV7, ARMV6, ARMV5, PowerPC64, little endian and PowerPC32 bit running continuously.  The mainline Linux kernel, Linux Next, LTS branches 419, 414, 4944.  So it's a poor man CI like cron job running every night kind of thing, but for all these various like, I don't know, 30 something targets kind of thing.  So you can check that out.  It's just running on Travis.  We have a link to other talks and other kind of material if you're interested in finding out kind of more about the history and the project and stuff.  Godbolt.org is amazing.  That's the link I showed you earlier with the code that was only in lineable.  That's like, if you want to communicate to a compiler developer, a bug, like send them a Godbolt link.  Okay, that is like the shared language that compiler people understand.  Like here is a clear case.  Here's what GCC does.  This is what I want in Clang or here's what I code.  And here's how it crashes Clang, right?  Kind of thing.  C reduce.  I just wrote a blog post on it.  It's incredible.  I have it.  I have like at least 10 compiler bugs that I've found with it.  You kind of give it a shell script where you put whatever you want in it and then you say like in a source file and it mutates the source file, pairing it down until you have like a nice concise reproducer.  So I found like errors at link time.  You can say like this script returns a different return code based on if this symbol is found after running an M or not or something on a binary or does it crash the compiler or does a compiler produce strange output or strangest assembly bear is this utility that hooks make for and it kind of spits out a compile commands that  can be used in a different file which you can then feed into static analysis tools like scan build Clang static analyzer or was it Clang CPP check, right?  It was the other utility.  I think that uses it.  Or alternatively, the kernel will spit out these.o.cmd files that have all the commands for each translation unit when you build.  The issue with that is it only produces them, I believe, if compilation is successful, which we could probably fix that in K build.  But that's helpful as well.  Just because like when C reducing a bug, you sometimes need to know exactly like what flags were passed to the compiler to reproduce a given issue.  And so like getting those flags out of compile commands.json or the.o.cmd file is critical.  Same thing with stack analysis is different command line flags will change your translation unit kind of once it's been pre-processed kind of thing.  Make sure you grab for the right error message.  Yes.  Yep.  So.  So, this is a code where I just say like grep for like a crash and then it paired it down to some like bizarro C syntax thing that like and kind of kind of the Clang's  maintainer is like, well, do we need to fuzz this, this, this C front end?  Like if it's not valid C code, like if the compiler crashes or not, right, is it really a bug?  I don't know.  Cool.  So, that's all we have.  Thank you very much.  And we're happy to take questions now this time.  Or sorry, I guess it's time up.  Oh, yeah, yeah.  Yeah.  So, the question was around what set of kernel configuration options do we use?  The kernel itself has a build system called K build that is highly configurable, right?  So, it depends on the distribution, right?  Like you'll have distributions of Linux where there's no guarantee what kind of hardware they're going to run on.  So, they're going to compile a big kernel image that has tons of drivers in it because someone's going to try to put Debian on their toaster or their faucet or something, right?  So, just turn everything on for Android, particularly for Pixel, you know, we have one hardware configuration, right?  So, we're going to pair our kernels down and know exactly what we're running, but then, like, we may have out of tree drivers that are problematic and things as well.  And so, that's definitely an issue of does it build or not, right?  So, someone might come to me and say, like, well, can it build a kernel outright?  And my question is usually like, well, what are your configs, right?  Because the hard thing with the kernel, like, baseline, what we run in CI is we want to make sure that the dev configs build.  So, each architecture has a recommended kind of set of configurations.  So, that's kind of what we cover continuously.  From there, we try to run all yes config builds every so often.  The issue with all yes config, which turns on lots of configs, is the kernel has a lot of configs that are mutually exclusive.  And so, like, you get this implementation of this function or that depending on the config.  So, all yes doesn't mean, like, all the code in the kernel.  So, there's probably code hidden somewhere in the kernel that will crash Clang or Clang can't compile and it's like a matter of finding it.  So, the kernel has something called ran config, which kernel CI does run.  So, once they have Clang support, they'll probably help us find it.  It's just a random coin flip.  So, you have to flip that coin a lot to find all the bugs and stuff.  They're on fabricator.  I link to them from the slides.  If not, if you go to our issue tracker, they're like posted over there kind of thing as well.  So, if you can pull them down on top of mainline Clang, rebuild Clang, and then try it out on the kernel, you'll run into the issue with the static always inline.  I have a patch for X86 that converts those all to macros.  For arm64, that rabbit hole got really deep and I never finished that patch.  But it's a starting point if you're interested.  So, can I build on X86.5?  You'll still need the out of tree patch because inlining support is not implemented in the asmgo2 patches yet.  So, the plan is to land, yes, that is the plan.  Once Clang has support for asmgo2, then we're going to work on getting it to be able to inline basic blocks that have asmgo2 statements in them kind of thing.  And then, the goal is theoretically there should be no other bugs, but sometimes you don't have visibility beyond like one bug.  So, you're like, yes, I fixed it and then you try it out and then something else is broken.  So, I can't promise you, but I think it should work kind of thing after that.  Oh, I mean, alternatively like, alternatively you can revert the kernel's use of asmgo2 forcing it kind of thing.  So, it's not a real solution, but you know, you still get a working kernel kind of thing.  One of the big problems we had in LLVNX was that we had to track three different patch sources.  LLVN, all is running, kernel, all is running, and our own patches always flipping between the two.  And because we had like just two or three build blocks like X86 and one ARM and one ARM is running like in the KMU,  it was hard enough to keep track of all of these three sources in a single configuration.  And you guys have like 30 something configuration. How do you find where the bug is in LLVN or in the kernel or in your own patch?  So, eventually we're looking to like get help from kernel CI and Zero Day Bot for spotting these regressions.  And kernel CI is doing like, they pin the kernel version and then try with different versions of clang like top of tree clang.  And like they'll report to us when people break the clang in LLVN trees, which is like pretty frequently, which is kind of frustrating.  But it's good that we're getting eyes and coverage and stuff.  So usually like I'll kind of forward these reports to LLVN developers saying, hey, you probably already know this.  Someone probably poked you, but in case they didn't, you're breaking the Linux kernel right now kind of thing.  Then I would say on the kernel side, like we're trying to do like always have patches upstream, have Zero,  or like minimize the delta as much as possible so we're not carrying patches around kind of thing.  So it's like as soon as we have a fix, it goes upstream and we work with the kernel maintainers to make sure we get a fix and kind of thing.  And this is no inline asm thing because there's a lot of inline asm problems.  So we disable no integrated AS kind of thing for our builds for now.  The kernel makes use of assembly, both external assembly files, but then also the inline assembly,  which is even more complicated because it has a whole constraint language with it.  And so like kernel abuse extensively of non-standard assembly.  They have C code generated by the inline assembly that is then recompiled as an assembly file by the C compiler.  Abuse is an accurate term when talking about the code base.  Okay, any other questions?  What's your plan for plugins?  Good question. So the question is about what is the plan for plugins?  So the kernel has, I think, checked into the source even like some GCC plugins and then some are maintained externally.  So LLVM does have a plugin system.  When I think back to the earlier talk today, that was an excellent talk.  That was about tooling and how to support kind of this ABI instability in LLVM.  I think that that's probably like a good thing to point to as far as like now you have a plugin,  but how do you guarantee it works with the given compiler version that a user has,  especially if the compiler doesn't have a stable ABI itself.  So from a plugin maintenance perspective, it makes my skin crawl a little bit,  but it's not impossible. It does have a plugin system.  And I think plugins are useful.  I think anyone who thinks of reaching for a plugin as a tool needs to sit down and really think hard,  should this be implemented as a plugin or in the compiler itself?  So for instance, I'll give a case, a recent talk at Linux Conf.U was saying,  hey, we use a GCC plugin to default initialize all variables.  I think personally, I think that's better implemented just in the compiler.  And there's actually discussion about this for C++ and there's actually disagreement within the LLVM community.  People saying, oh, you're going to fork C++ and create another dialect.  You can't do this. And I feel like saying, well, for C, which development has kind of stopped on,  like, can we please have this since we already have our own dialect anyways that we use in the kernel.  Like kind of every dash F flag you add.  Like to me, this could be another dash F default initialize, right?  And there's no need for a plugin. Plugins are great if you want to like hack something up and play with the compiler and get something working.  I would say the best case where I say plugins make the most sense is when you have,  you need compiler technology for one given project.  Because then if you were going to say like to submit this to the compiler vendor and say this is only used in this one code,  code base, they're going to tell you to get lost.  They're not going to want to maintain this plugin system because whatever,  whatever maintenance overhead there is kind of thing.  There's only works if the compiler version is fixed, which in GCC is normally what happens.  They have GCC five forever.  In LLVM, you know, fixing in GCC in LLVM six or LLVM seven doesn't make sense because it just progressed so quickly that you can't get a plugin that  working seven don't care its drug.  So it's a completely different way of doing it.  Okay. Thank you so much everyone. We appreciate that.
Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_0_143.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_144_803.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_804_887.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_888_3083.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_3084_7403.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_7404_15911.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_15912_15995.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_15996_16139.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_16140_16187.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_16188_16511.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_16512_16919.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_16920_17015.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_17016_17519.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_17520_18383.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_18384_22943.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_22944_22991.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_22992_23039.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23040_23111.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23112_23171.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23172_23219.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23220_23267.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23268_23315.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23316_23363.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23364_23411.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23412_23471.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23472_23543.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23544_23615.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23616_23663.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23664_23711.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23712_23795.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23796_23891.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23892_23939.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23940_23987.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_23988_24071.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_24072_24191.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_24192_24323.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_24324_24455.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_24456_25931.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_25932_26015.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_26016_30095.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30096_30143.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30144_30251.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30252_30299.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30300_30347.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30348_30395.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_30396_31667.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_31668_32207.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_32208_32303.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_32304_37583.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_37584_37643.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_37644_42311.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_42312_48167.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_48168_48215.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_48216_48299.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_48300_53267.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_53268_53363.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_53364_60899.png Compiling_the_Linux_kernel_with_LLVM_tools_m4ktATnz0co_60900_61020.png