Everything You Always Wanted to Know About "Hello, World"*
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Davis, Brooks
I've been working on a new system call ABI and the required runtime support for a C variant with spacial memory safety. Along the way I've encountered lots of interesting bits and pieces required to implement a simple C "Hello, World" program. I found the process fascinating so and this talk brings all that knowledge together in one place. The first example in the classic "The C Programming Language" by Kernighan and Ritchie is in fact a remarkably complete test of the C programming language. This talk provides a guided tour a slightly more complex program where printf() is called with multiple arguments. Along the way from the initial processes' call to exec() to the final _exit(), we'll tour the program loading code in the kernel and the dynamic linker, the basics of system call implementation, the implementation of the memory allocator, and of course printf(). We'll also touch on localization and a little on threading support. Where appropriate, I'll discuss portions of the system that need changing to accommodate memory safe versions of C like the version we are developing for our CHERI CPU. This talk will assume some knowledge of a language with C-like syntax (C, C++, Java, and PHP should all be fine).
 Good morning. I'm going to talk to you today about Hello World, which is a deceptively simple program. It is, as this slide shows, the first example in Kerrigan and Richie's, the C programming language. It's on the first page. It's about the simplest thing you could do, or so you might think. It turns out, actually, that talking to several people who have implemented languages in run times that printf is usually the last part of the language you implement because it requires all of the rest of the language. And in fact, a slightly more complicated version of Hello World that I'll show you shortly is essentially a complete test suite for C. So here, again, is the first example. So well, to start with, we have to correct it a little bit because it turns out the example in K and R is not valid C in any modern C dialect. So we had to add a couple of void statements. But so that's a bit of an interesting thing. To get all the way to the example I'm going to use today, I'm going to add a couple more things and slightly complicate matters. So what I've done here is I've added a static string rather than just having a printf so that we're actually using the Verodak arguments functions of printf, which is one of those horrible features that no language should ever have, at least not one with a register-based calling convention. But all modern architectures do this. And this is one of the reasons why printf is really a test suite. Before that, though, let's talk about the minimal version of printf, of Hello World. This is the smallest thing you could write in C that emits the string. It uses the right system call and just splats it out by length. And then it calls exit, which is also a system call. It's very simple, very straightforward. And you might think that it would turn into a piece of assembly code much like this. Don't worry, if you don't know assembly, this is the last you'll see in this talk. But it is very, very simple. So we load the one argument into the first argument register. We look up the address of the string Hello. We load it into the second argument register. We load the length incorrectly into the third argument register. We load the system call number into the system call argument register, and we trigger a system call trap. This goes into the kernel. The kernel does its thing. It reads the data. It writes it out. Magic happens. And then we return. And we do basically the same thing again, except that we load the return value of zero and the system call number for exit. And that's it. This is, in fact, the compiler DLA is a pseudo instruction, so it actually does two whole instructions when it's expanded. But really, this is it. So you might think that's what happens. But in fact, the minimal C version is about a half megabyte stripped. It turns out that this is mostly malloc and localization, despite the fact that you're not using either of those features. So you might wonder why that happened. How did we get all this stuff in a program that doesn't use any memory management functionality at all? Well, for some hints at that, let's look at program linkage. So here's how you might link a simple program where you have an object file and you want to turn it into an executable. Most programmers probably know that this link's in the C standard library. That's expected. But what you might not know is here's the actual link command, quite simplified, that's generated by the compiler. I've removed all the quoting and I've simplified the paths and I've chopped a few things out that weren't interesting. But you notice all these little object files. These are linked in this order because they contain initialization code. So here's the list of them again in a more sorted order. So CRT1, this is the beginning of all things. It contains the underscore underscore start function, which is the thing that actually gets run first in any sort of conventional POSIX C program. The name doesn't matter, but by convention, that's always the name. And its job is to set up the environment that C expects and then call main. In following that, there's CRTI. CRTI's job is to ensure, is to call a number of initializers and also set up some destructors. The things that it sets up are horrible. It's just a genuinely terrible idea. So there's these underscore init and underscore finet functions. What appears in CRTI is some assembly for a function entry for each of these in a different section. Then any random object file that's linked later on could contain some other machine code which is concatenated together. Hopefully everything goes right. And no one misaligns the stack on the way through. No one corrupts the state of the stack in other ways and all that. And then way at the end, CRTN adds some return from that function. And again, hopefully everything went well. All that's just sort of mushed together by the linker to create some functions. It's lovely. Then there's CRTBegin. There's three different versions in FreeBSD. They just differ in what kind of linkage you're doing and they're compiled slightly differently. But they all do the same thing. They support yet another kind of constructor. These are C++, Ctor, Dtor constructors. There are another set of sections. These are at least halfway sane in that they are in fact a section which contains an array of function pointers. So you just call the functions. That makes sense. And it's fairly reasonable. And in addition to declaring those arrays, they declare some functions to actually call what's in them. So the layout of the arrays is not part of the ABI in this case. So again, CRTN terminates those. They need to be null terminated. So each program might contain a bunch of function pointers. Object might contain a bunch of function pointers that get put into these arrays and then they're terminated. So that's fine. And there's actually two more kinds of the constructor and destructor that we'll get to later. And these are where all this stuff gets pulled in. We're going to do a walk through the code or through the execution, a little bit of the code, mostly just looking at the stack. And I'll talk about what various things do. And along the way, I'll show you where we hit all these things and why stuff gets pulled in that you wouldn't expect. A quick note, most of the rest of my slides are going to be screen caps of Zoom mobile SVGs. If you look at this URL, either one, they both go to the same place. You can see the actual SVGs. And so you can explore on your own if you want to. These were generated by taking an instruction trace out of QMU, doing a bunch of processing with AUK, and then feeding that into a slightly hacked version of Brendan Gregg's Flamegraph program so that you can see stacks. I think it's kind of a neat visualization. I wouldn't want to try it on a really complicated program, but as you see, it's kind of a neat way to look at things. So we're going to start not in printf itself, but in the kernel, at the sys execve call. Execve's job is to take an existing process and replace all of its contents with whatever program you want to run, in this case, Hello World. So let's take a look here. The first thing we get in execve is execve copy in args. Its job is to allocate some memory and then copy in the program path and the other arguments that were passed by reference to the execve system call. It's not very exciting, but it is one of the first things that's involved in setting up the exec call. The next thing we're going to look at is current execve. This is the actual implementation of execve, and it's used by all the different system call avis in previous Ds. So if you call a Linux binary, it does a bit of work to deal with Linux enus, and then calls current execve to actually execute the process. So the first thing that happens in current execve is it looks up the path that you've passed. I'm not going to go into details because I don't understand name i, but it's magic. You give it a path, it finds it. It's all good. And as George has been telling me all week, names are an illusion, so we don't want to know anyway. Next part is a little function which checks the permissions of the file, makes sure it's executable by the user who owns the process. It seems like a good thing. And as a side effect opens the file, then exec map first page does what you might expect. It maps the first page of the program so that it can examine the header and other parts and other things, other image activators can look at that header and say, is this mine? Is this something I can run? Then we get into the meat of things. So this traces on a 64-bit MIPS machine, obviously because that's where I've been doing all my work, and because I have nice instruction level tracing. So on there, the first image activator we find is the one we want, which is the 64-bit Elf execution environment, image activator. So it's going to try and load the program, get memory setup for it, and then set up registers so it can return into the program. So the first function we get is exec new namespace. Its job is to set up the memory mapping of the process in the initial state. First thing it does is it deletes all the pages from the memory mapping because it doesn't care about whatever the contents were from the previous process that forked. The next thing is that it maps a stack into the address space. This was something that actually surprised me a little. I had expected that stacks might be placed on a per system call interface basis, but in fact, they're always set up here. They can move around, so 1x could put the stack in a different place, but in practice looking at the system, it's always the same on every architecture and free BSD, which is a little surprising. So now we have a stack and an otherwise empty address space. So the next thing we're going to do is we're going to load some sections into memory. So a typical simple statically linked program, which is what we're talking about, has two sections in the file. There's the text section, which is the executable bit of code, so it's all the instructions and some other non-instruction things, but it's basically all the instructions that will be executed. And then there's the data section. The data section is all the static data, all the dynamic values, or all the global variables, all the initialization values for various things. And then there's BSS, which is all the global and static variables that default to zero. And so those aren't in the file, that saves space, but those pages are mapped at the same time. So back into current exec VE. The next step is exec copy out strings. This is a bit of an oddity. It's actually, despite the fact that we're on MIPS, it does a bunch of things according to the SCO I3D6 ABI, which happens to define how every elf executable, which is the standard file format on every UNIX anyone cares about, except macOS. So it defines how the process is laid out and where to find things. So for instance, where to find the argument vector, where to find the environment vector. And I'll get to the how this is laid out. But what's interesting is it's actually just plopped on top of the stack. And why? Historical reasons. But it's an interesting thing there. And then finally, there is exec set regs. Exec set regs, its job is to set up the registers in the trap frame so that when the system call returns, it says though it returns to where the start process is, with the right arguments. And I'll talk about what those arguments are when I get to start. So here we are. We've returned back from Kern exec VE. We go to sys exec VE and we get the standard system call return path, which is out of scope today. And we return to user space. So the recap, at this point, the stack is mapped into the address space. The program is mapped into the address space. And a bunch of strings and whatnot are set up. And then finally, the register status set up to call the start function. So now to talk a bit about what's on the stack. There's a whole bunch of things. Their order is very strict, is in part strictly defined and in part loosely defined. The top is PS strings. It is used by the debugger and it points and by programs like PS to find program names that are too long. It doesn't appear to actually be necessary in modern software, but we're stuck with it for existing ABI's. Another oddity, SIG code is where signal hander rollers return to so that they can jump back into the kernel. And for some reason, we decided to put it here long, long ago. And then there's a bunch of strings. There's the stack canary. There's some arrays of things. Those are accessed by the elf auxiliary arguments, by the environment's array or via argv. And finally, the last thing at the bottom of the top set of the stack is the argument count. So this is all the things that you need to pass to main in order to run a program with standard arguments. One oddity here is argc in C is defined as an integer, but in the ABI it's a long so that it's always pointer aligned. And so that stack aligned is preserved. So lots of weird history here. And so I would say what I forgot to tell you in the introduction is the reason that I'm here is that I'm looking at all these things and finding all this history is that I'm in the process of writing a new ABI for a memory safe variant of C with hardware enforcement. So it turns out all this history and all these odd things that you just know you can find them at the top of the stack, for instance, that's not OK if you want memory safety. So we have to take all these assumptions, find them, tear them apart. So here we are. So you end up with the stack pointer at the top of the stack. Additionally, the first argument to start is also that stack pointer. And it's used for things. I go into this again, but it's used to magically find things on the stack. So this layout is all part of the application binary interface. And it's part of how the process works. So now let's get into start. One thing you might observe, I'm looking at this, is that most of the arguments have these JEE, or most of the functions have JEE something under them. That's because, in fact, this program spends almost all of its time in malloc. Something like 90%, 95% of cycles in Hello World are spent in malloc. So part of that, so most of that is because JEE malloc is a malloc designed to support giant multi-threaded programs. One of the optimization test cases, in the first case, was Firefox. These days it is whatever Facebook's PHP workload looks like. So it is a big, heavyweight implementation designed for real problems. So it doesn't really matter. Hello World's not a performant thing. You don't really care how long it takes. But it's interesting that there's all this overhead. And in fact, for some program, if you wanted a trivial program like this to be super fast, you would definitely want to use a simple malloc. And actually one of the things we found in our research is we were using a trivial malloc in some of our benchmarking. We realized we were doing a lot better than we should be. There should have been more overhead compared to the unmodified environment. It turned out, oh right, we hadn't turned JEE malloc on yet. So we made that change in our tree recently. So let's take a look at start. I've condensed start onto two slides by stripping out all the conditional cases for dynamic programming and dynamic libraries and all the conditionals for profiling. So it's just the minimal thing. So first up, we're setting up the argument vector. So we have AP, which as you recall is that pointer to the top of the stack. We know that that is in fact a pointer to argc. So we get argc out of there. And then we say, oh, well we know the next thing. That's argv. And then we know that the environment is just past that plus a null pointer. And we buy argc. And we'll get later in a moment to an even more bizarre example of finding things on the stack. The next thing that we do is there's this function handle argv. Its job is to set the environment variable. So it sets it to the pointer to the argument array. And it sets the program name variable. It's kind of a minimal thing. Next up is init tls. So this is in fact why that example program with just write was so bloated. It turns out that we need to set up thread local storage. And we do this unconditionally. We need to do it in all cases because we don't know whether or not we'll be threaded. So in a static program, we probably could avoid it. But in a dynamic program, we must always set up tls because we might load a library later that uses threads. So we need to have thread local storage set up. So for those who don't know, thread local storage is global or static variables which are per thread. So each thread has its own copy of that variable. Again unsurprisingly, most of the time here is spent in malloc. So let's talk about what init tls does. So the first thing it does is it finds the elfxillary arguments vector. It does that by this cute bit of code, which is it takes the environment array and it knows that the environment array is null terminated. And it knows the elfxillary arguments vector is just past it. So it walks right off the end of the array on purpose. And then it says, oh, that's the elfxillary arguments vector. Good. I'll just cast it and use it. Great. So it uses that to find the program headers. So one of the many things in the elfxillary arguments in addition to things like the stack canary and the array of page sizes and whatnot is a pointer to the maps in programs program headers. So that's the thing that tells you what all the sections are in the program, in the program's binary. It uses that to find the plt tls section. This section contains the default values for all of the tls variables that aren't zero by default. So you might have a tls variable that's one because you want to do something, you know, you want a condition to be true. That will be stored in here. And that's the beginning of the tls section is all those things. So that can just be splattered right over the top. Once it has that information and it's stored it so each thread can use it in the future, it calls this libc allocate tls which allocates space and initializes values as necessary. And this is where JML comes in. So it needs to allocate space and that space needs to be freeable later by calling free because you might start another thread and kill the original one. So you can't just leak that and you don't want to have to record that maybe I'll replace you know, you don't want to record that this one's special in some way. So what happens is there's a special interface into JML which doesn't use thread local storage and which does all the initialization bits but it avoids all the things that haven't been set up yet because as I said JML is designed for a threaded environment so of course it needs per thread pools of storage and whatnot. So it makes use of tls. And finally the tls pointer is set on a per thread basis. On MIPS this is calling a special sysarc syscall which says set the tls pointer to this value and then when you want to use it you call the syscall again usually via an instruction that traps, that generates a trap because it's typically unimplemented and you get the value back. In CherryBSD we actually have implemented an optimization so we call the syscall to set the value but then we set a user local register which is a special register readable by the program or readable from user space. Yes. I think that's the first time you mentioned the name CherryBSD. Oh sure. So Cherry is our processor with memory safe extensions and CherryBSD is our fork of free BSD. I talked about it last year so I think there's probably a talk online if you're interested. It is basic so it adds bounds checked pointers so we can do all the things that ASLR and control flow integrity except we can do them in forced and hardware and fast. So yeah so we've implemented some optimizations that one we could probably merge back to free BSD. I think it works on the entrader light so we probably should. So back to start again. We've initialized TLS and now we get into a handle static in it. I'm not going to go dive deep into the code because it is a chain of boring and hard to read code but basically this is where all the constructors are called and where destructors are registered. So as I said earlier there are four different kinds because people added them at different times and wanted different things. So I did it in Finae were the first things historically and I believe an array was added next so that's an array of function pointers that's the bottom one that runs last and it's called it's a sensible interface more or less. And then I assume that people decided oh but we need to run some things early. Some library setup stuff that might be called by other initializers. So we got pre an array just because why not have a consistent and sensible interface when you can have something dumb. Actually what's really going on here is it's hard to get all the world to change their linker and to add new features to all the linkers in the world so therefore you have to write something that works with the old cruddy linker you have, not the new shiny linker you would like to have and then you encode it into the ABI as is done here you make it part of all the standard interfaces and then you're stuck with it forever and ever. I think we can kill in it in Finae. We're doing it in CherryBSD we're going to see what happens. And there's also the C++ constructors, the Ctors section and that's weirdly in FreeBSD I think actually due to a GNU thing called through in it and that's the only thing that gets called through in it. That's a trivial fix but I haven't done it. In any case what all these do is they call a bunch of functions which set various things up. The thing that actually matters here is that in the process the finalized destructor which calls Finae and which calls there's a Finae array at exit get set up. And then finally we're ready to call main. In FreeBSD's start that is main and whatever main returns gets passed to exit and the exit system call gets run. So we'll get into main now. Okay so here it is. Let's see and I can barely read it. So again you know it is mostly malloc. So and in fact although it is also mostly VF printf. VF printf is called by printf down here and then VF printf starts doing some work. The first thing it does is it has to get the locale which is how it can look up things like which decimal point is used. And we'll get to the, that is in fact the reason it's used in this case. But get locale has a bit of interesting machinery to it. So locales are per thread because you might want to write a server where one person connected is in a country where the decimal separator is a comma and another one in a period. I don't understand exactly why you would want to structure your application this way but this is in fact something you do. So locales are per thread. So we look up the locale and then we call this once function which is an interesting bit of machinery. It's job is as you might expect to do something once. Either once per thread or once per execution. So you can look up a value and then you can save it. So what it's doing here is it's trying to find the locale path. It looks at the environment, tries to find the path. In the end it's not set in the environment so it uses the default. So all that work to copy a string. But it's very general. And that's one of the amusing things about looking through printf. As printf is trying to solve, as you walk through all this code, it's trying to solve the whole big problem and that's why it's such a big beast to make two system calls. So back into VF printf. Now let's look at underscore VF printf. Under score VF printf we get here by passing the locale that we got before to VF printf L which takes an explicit locale and then we call VF printf which does all the actual work of printing a string. So I'm going to ignore all the stuff on this side because what this is is it calls into the file object for standard out and it looks at it and says, oh, you're a buffered file and you don't have a buffer yet. I guess I better allocate one. And so that's what all this code is, is basically allocating an array. And I don't have time to get into jmalloc. So we won't do that. Now this is the part that's kind of the actual meat of printing. First, this was a bit of a surprise and I think needs to be optimized out. Printf no matter what it's printing, what's passed to it, always spends a bit of time here finding out what the decimal point is on entry because you might print a floating point number so you should look that up. Next is a fairly simple thing. As you recall, the string that we were parsing was a percent s, space percent d slash n. So a string, a number, and a bit of white space. So the way vfprintf works is for each block of data up to a format code, it assembles a buffer of things to print out and then flushes those out at the end. So in this case, it first finds the percent s string format. So it puts that and the string in. In here it looks up the length of the string, sets that in the buffer. Here it adds a little bit that says, okay, there's going to be a space and then it says, oh, there's going to be a number and it calls this function whose name I can't quite remember whose job it is to write out the representation of the integer string. One thing that I found was interesting when I was reading this code and modifying it for some other reasons is it actually works backwards. So it starts at the end of the buffer, of the temporary buffer and writes the first digit or the least significant digit, the next, the next, the next, which is an easy way to do it and I thought that makes a lot of sense. And then I stumbled across three other implementations of printf in the tree, all of which do it the other way. In fact, we have at least four implementations of printf in the tree. The runtime linker has its own, libc has its own, there's one in the kernel, and there's I think one in the boot loader and I'm pretty sure there's some more. All right, jmalloc has one too, because printf, as you've noticed, uses malloc and jmalloc wanted to have a printf so they could print debugging information and then, you know, deadlock is bad. So there's a copy of printf in there. And then we finally get to the last bit, so there's the new line at the end and we'll zoom in on that because that's where the actual write system call goes. So we have this single buffer to be written out. At the beginning, when it's scanning the buffers, it first copies them all into one place and then it looks at them and says, it scans the mall and says, oh, is there a new line anywhere in this output? Because if there's a new line and it's a buffered file descriptor, then we need to actually call write when we hit that new line. But it finds the new line, which means that this fflush call gets made later. fflush in turn does a bunch of things, digs down, digs down, digs down. And finally, we write the string. So there we are. Now that's the basic static hello world. I said in my abstract that I would do dynamic hello world, so I will talk a little bit about dynamic hello world. Unfortunately, I was assuming that I would have finished modifying the runtime linker for our project to support our new AVI and it being a software project, I didn't get it done in time, so I don't actually understand runtime linking yet. But I understand the basics, so I'll give you a quick tour of it. Oops, wait, I'm sorry. I am not there yet. I have one more tiny bit. We've returned from main. We're back into start. We get to exit. And exit isn't actually call the system call directly. It's a little more complicated than that. Inside exit is a function. Its job is to call destructors. So this CXA finalize is a function whose job it is to call all the destructors that were registered through the at exit function. So finalize is one such destructor. So that gets called, that calls FinA, cleans up all the, does a bunch of deallocation and clean up. In practice, it doesn't really do much. The next part is there is this clean up function, which could use a more descriptive name. So what it actually is, is it's a function that's called only if a C file object was used during the course of the program. It's used to flush out any remaining IO. Then finally, we call the exit system call. So okay, now dynamic libraries. So here's a different trace, which you might notice is you can't see where main is. Main is this little bit on the far, far right. So for those in the recording, it's a little bit over here. So dynamic library, dynamic linking has quite a lot of overhead. There's also a few changes in the process layout. So the first, so we'll start here at the beginning. This is the same memory layout that you saw before. You see the program on the left, you see the stack on the right. And that's, the kernel loads the program just the same way, except that it also loads the runtime linker. The runtime linker and the other difference in the exec v is that it jumps into the runtime linker's entry point rather than into the program. The first thing the runtime linker does is it calls this long name, which as you can tell, if you look at it, you'll notice it includes relocate itself. That's because the runtime linker is position independent code and it needs to have its variables updated and be relocated. So it has a bit of slightly horrible code that makes sure that wherever the heck the kernel decided to put it, it can run. It's at least C. It walks at array. It does some things. It's not a big pile of assembly or something. So once that's run and the program is relocated, it starts parsing the program itself. It notices that the program depends on libc. So it loads libc and it does an initial relocation of libc, sets up all the global variables, and then sets up a table of entry points for the program to call into the symbols it needs. And we'll get to that in just a moment. So we have, let's see, we have libc linked. Yes. So libc is now loaded in. One thing about RTOD is the kernel loads it as though it were loaded by the M-map call, which maps a file in. So it attempts to pick the same location in the kernel. That seems sort of gratuitously pedantic. But that's what the kernel currently does. One thing I did notice when I was doing my initial tracing is that the location it chooses is very poorly aligned. And so it made it very hard to understand what the runtime linker was doing when I was looking at an object dump file and looking at disassembly because everything was shifted in fairly low bits. So in CherryBSD I've moved it. So it's aligned at 16 megs or something and it's much easier to do math in my head. And then libc gets mapped by M-map the normal way at the end. So then, so once libc has been initially relocated, we call in to start again. So you can see way, way, way off to the right. And this, I think, looks fairly familiar. You have main here. The stack trace is a little screwed up because I didn't get my tool fixed in time. So add exit does exit. And it's not part of the stack forever. But what you'll notice here is that there are these RTLD bind starts after things like printf, which has this funny name. So let's take a look at that. So what you see here is that printf has this long name, which is printf, and then at, at this FBSD09. That says that this is, it's looking for version 1.0 of the printf symbol. It's looking for somebody to supply that, in this case libc. That's in the symbol versioning mechanism, which is why we haven't bump, had to bump the version of libc since FBSD7, which means your programs keep working unless they do something like use libkvm or libutile, which are hard to version. So there's the symbol. And then when calls are made between modules in a dynamically linked program, they def, by default, what gets called is this, this little bit of runtime linker code rather than, so the symbol's not re, not pre-resolved. This is because libc has a gazillion symbols, and they take a fair bit of time to resolve and to look up. So it'd be, it'd be very expensive on program start if we took all several thousand symbols and started them up, you know, as you, you know, you see here, the whole runtime, you know, there's maybe room for 20 of them in that total runtime. So you wouldn't want to make, do all these resolutions. So you call this little assembly stub here, and its job is to say, oh, you're calling, you're looking for printf. So it goes in and it wanders around in the libc symbol table, does a bunch of lookups, finally finds the symbol. And then it returns and it sets up the table. So now the table includes the correct address of the symbol. So feature calls, not that anything calls anything twice in this program, would, will go direct rather than having this lookup occur. It's called lazy binding. So that's basically the same. Otherwise the program's the same. It's just that any time there's a cross module call, there's one of these lookups. Otherwise dynamic, dynamic for def is, or dynamic hello world is pretty much the same. So that is the end of my talk as I, hurry up, I'd happy to take questions. I would like to request that you give me feedback either by email or this is a link to the conference website. That's anonymous as far as I know. And the QR codes for that too. So let me know, was the talk helpful or interesting? Did it make sense? What would you like to learn more about? Maybe you wanted to learn about JMLock. Maybe there was something you wanted to learn less about. Something that was dreary and boring. This is a big, messy thing. Some of it's kind of that way. So I'd be happy to take any questions you might have. Yes? You talked about copying the arguments from the old address post and new address post. What about the copy out? So let's see. So the question is how does the argument copying from the initial process into the kernel and then back out work? So that allocate arguments, let me pull up the zoomable SPG. I'm going to turn the internet back on. Are you sending me messages, please? Okay, so exact the copy in args, it allocates some kernel storage. So it looks at the argument list and allocates some storage in the kernel and copies in all the strings using the usual copy in primitive. And then after the address space is reset, the... So after the address space is reset, there is a stack. And that's already mapped, that's just anonymous pages with the stack flag which says they grow down rather than up. It uses optimization. And then exact copy out strings is job is to take those things we've copied in and have destroyed in the parent process and copy them out. So it knows that whole layout, so it knows PS strings is at the top and it sort of works its way down just copying out through the usual copy out mechanism. So it doesn't bypass the VM and just go blank to a page? Does it's got no need to go through the VM? Yeah, that's true. So I think it could, I don't know. If double triple book feet in your argument list, you have to copy it out. You have to go out of the process to go through the VM to the pool. So you probably could in fact do that. It would make the setup more complicated. Well... Yeah, I mean this is quite a bit... Yeah, I don't think the copy out is very expensive. You don't lose very often. But I think you certainly could do it. The problem is that you would want to change the structure of copy out strings to... You would have to compute how much it needed. So you'd have to move who is responsible for creating the stack, which would be fine. But you'd have to change that and then you'd have to have something that determined how big that set of strings was going to be. And then... It's more than that though. It's a bunch of ABI-specific things. So you could... I think if you reordered everything a fair bit, you could do it. And I think part of the problem right now is there's quite a bit of plugability. So exact copy out strings is the default for the architecture, but different ABIs have different ones. So you could do it, but you'd have to refactor all of the compact layers. So be an interesting experiment to see if you got better performance or enough better performance to deal with the overhead. Anyone else have any questions? Oh, sure, I got one over here. I'll take it. What is in PS strings? Well, let's go take a look. I think I remember the name. There we are. Yeah, so it is... It's a pointer to one or more argument strings and to the environment. It's used to update... So it's primarily used, for instance, when you call set proc title to change the name of the program, you then need to point to a potentially completely new malloc set of arguments. It's a little gratuitous in that, at least for the arguments, because you also call the kernel and say, please set this, historically, at least, it was done this way so that you wouldn't have to allow the kernel... The kernel wouldn't have to accept an arbitrarily large title. In practice, there doesn't seem to be an actual limit on the sys call that's... On the sysctl that's used to set it. So in practice, I don't think it matters. So in Cherry ABI, I think I'm getting rid of this because the interface has a bunch of other problems. But it's... Yeah, so it's just pointers to these once they've changed so the debugger can find them. Part of why I say it's a bit gratuitous is that you could simply pass this same stuff to the kernel because you only really care about it if you're writing out a cord up or doing some debugging. So if the kernel knows where these things are, it's fine, it can write an elf note in the cord up and you can look it up there rather than sticking it at the top of the stack and hoping no one steps on it. You had another question? Yeah. What things in Cherry ABI do you kick off the stack? First one that comes to mind is the Cherry ABI. So what am I going to kick off the stack in Cherry ABI? In Cherry ABI, which is the alternate mode where all pointers are hardware-protected capability pointers. So what I am planning to evict everything from the stack. None of this stuff has any relation to the program stack. So I'm going to kick it all off. Right now, I'm currently writing it to that memory allocation because I guess for the reason that I said there would be a bunch of work to change how things are done, it's easier to just splat it to the stack. Then I actually restrict the stack pointer so the stack doesn't include that stuff. The pointer that's actually accessible to the program doesn't include any of those things. So I plan to kick all that stuff off basically. And eventually it'll just go somewhere else. So yeah, auxiliary args should go. And I don't really want them to be accessible from the rest of the program. So I don't want there to be a global to access them. I want you to have to get them through the runtime linker, which will be able to compartmentalize. So the runtime linker can become a trusted linker and loader. Yes? Based on this, is there anything you're going to do to make it more accurate? I thought about changing the way we found the PS strings, but Acoustic really didn't like it, so we're not doing that. What else are we doing? Not a whole lot of the AVI stuff. I guess we are, I think so, on Cherry AVI, the way we find all those auxiliary arguments and stuff is there is a structure that's the interface between the kernel and user space. So we pass a pointer, instead of passing a pointer to the top of the stack, we pass a pointer to a structure that has an argv pointer and an argc and an nv pointer. Did we end up doing that for our V8? Okay. Sorry? We need to move around. I think some of that disassociation should just be done. There's actually no reason any of this should be at the top of the stack. It's just where it is. For the most part, the big change with Cherry AVI is going to be that you don't have things next to each other that you can access one from the other. That's not valid C and is something we can prevent. So we need to actually pass proper pointers to things, rather than just knowing that things are next to each other. Then it occurs in the exit code. So let's go back. The cleanup routine does the flush in that case. Way down here. There we are. Way off at the end, the end of exit. Because you have ever done any IO on that file descriptor or attempted to do any IO on a file descriptor or perhaps a buffered file descriptor, the cleanup routine is registered and therefore it does the work. So that's why you get this silly interview questions with multiple outputs and stuff. Would you still do a flush if there were not any? It walks the descriptor array and I think flushes things that have data buffered. Do you want to just bring it in? Yes. So, yeah, it calls it, says, is there anything to do? Well, thank you all for coming.
 Good morning.  I'm going to talk to you today about Hello World, which is a deceptively simple program.  It is, as this slide shows, the first example in Kerrigan and Richie's, the C programming  language.  It's on the first page.  It's about the simplest thing you could do, or so you might think.  It turns out, actually, that talking to several people who have implemented languages in run  times that printf is usually the last part of the language you implement because it requires  all of the rest of the language.  And in fact, a slightly more complicated version of Hello World that I'll show you shortly  is essentially a complete test suite for C.  So here, again, is the first example.  So well, to start with, we have to correct it a little bit because it turns out the example  in K and R is not valid C in any modern C dialect.  So we had to add a couple of void statements.  But so that's a bit of an interesting thing.  To get all the way to the example I'm going to use today, I'm going to add a couple more  things and slightly complicate matters.  So what I've done here is I've added a static string rather than just having a printf so  that we're actually using the Verodak arguments functions of printf, which is one of those  horrible features that no language should ever have, at least not one with a register-based  calling convention.  But all modern architectures do this.  And this is one of the reasons why printf is really a test suite.  Before that, though, let's talk about the minimal version of printf, of Hello World.  This is the smallest thing you could write in C that emits the string.  It uses the right system call and just splats it out by length.  And then it calls exit, which is also a system call.  It's very simple, very straightforward.  And you might think that it would turn into a piece of assembly code much like this.  Don't worry, if you don't know assembly, this is the last you'll see in this talk.  But it is very, very simple.  So we load the one argument into the first argument register.  We look up the address of the string Hello.  We load it into the second argument register.  We load the length incorrectly into the third argument register.  We load the system call number into the system call argument register, and we trigger a system  call trap.  This goes into the kernel.  The kernel does its thing.  It reads the data.  It writes it out.  Magic happens.  And then we return.  And we do basically the same thing again, except that we load the return value of zero and  the system call number for exit.  And that's it.  This is, in fact, the compiler DLA is a pseudo instruction, so it actually does two whole  instructions when it's expanded.  But really, this is it.  So you might think that's what happens.  But in fact, the minimal C version is about a half megabyte stripped.  It turns out that this is mostly malloc and localization, despite the fact that you're  not using either of those features.  So you might wonder why that happened.  How did we get all this stuff in a program that doesn't use any memory management functionality  at all?  Well, for some hints at that, let's look at program linkage.  So here's how you might link a simple program where you have an object file and you want  to turn it into an executable.  Most programmers probably know that this link's in the C standard library.  That's expected.  But what you might not know is here's the actual link command, quite simplified, that's  generated by the compiler.  I've removed all the quoting and I've simplified the paths and I've chopped a few things out  that weren't interesting.  But you notice all these little object files.  These are linked in this order because they contain initialization code.  So here's the list of them again in a more sorted order.  So CRT1, this is the beginning of all things.  It contains the underscore underscore start function, which is the thing that actually  gets run first in any sort of conventional POSIX C program.  The name doesn't matter, but by convention, that's always the name.  And its job is to set up the environment that C expects and then call main.  In following that, there's CRTI.  CRTI's job is to ensure, is to call a number of initializers and also set up some destructors.  The things that it sets up are horrible.  It's just a genuinely terrible idea.  So there's these underscore init and underscore finet functions.  What appears in CRTI is some assembly for a function entry for each of these in a different  section.  Then any random object file that's linked later on could contain some other machine  code which is concatenated together.  Hopefully everything goes right.  And no one misaligns the stack on the way through.  No one corrupts the state of the stack in other ways and all that.  And then way at the end, CRTN adds some return from that function.  And again, hopefully everything went well.  All that's just sort of mushed together by the linker to create some functions.  It's lovely.  Then there's CRTBegin.  There's three different versions in FreeBSD.  They just differ in what kind of linkage you're doing and they're compiled slightly differently.  But they all do the same thing.  They support yet another kind of constructor.  These are C++, Ctor, Dtor constructors.  There are another set of sections.  These are at least halfway sane in that they are in fact a section which contains an array  of function pointers.  So you just call the functions.  That makes sense.  And it's fairly reasonable.  And in addition to declaring those arrays, they declare some functions to actually call  what's in them.  So the layout of the arrays is not part of the ABI in this case.  So again, CRTN terminates those.  They need to be null terminated.  So each program might contain a bunch of function pointers.  Object might contain a bunch of function pointers that get put into these arrays and then they're  terminated.  So that's fine.  And there's actually two more kinds of the constructor and destructor that we'll get  to later.  And these are where all this stuff gets pulled in.  We're going to do a walk through the code or through the execution, a little bit of  the code, mostly just looking at the stack.  And I'll talk about what various things do.  And along the way, I'll show you where we hit all these things and why stuff gets pulled  in that you wouldn't expect.  A quick note, most of the rest of my slides are going to be screen caps of Zoom mobile  SVGs.  If you look at this URL, either one, they both go to the same place.  You can see the actual SVGs.  And so you can explore on your own if you want to.  These were generated by taking an instruction trace out of QMU, doing a bunch of processing  with AUK, and then feeding that into a slightly hacked version of Brendan Gregg's Flamegraph  program so that you can see stacks.  I think it's kind of a neat visualization.  I wouldn't want to try it on a really complicated program, but as you see, it's kind of a neat  way to look at things.  So we're going to start not in printf itself, but in the kernel, at the sys execve call.  Execve's job is to take an existing process and replace all of its contents with whatever  program you want to run, in this case, Hello World.  So let's take a look here.  The first thing we get in execve is execve copy in args.  Its job is to allocate some memory and then copy in the program path and the other arguments  that were passed by reference to the execve system call.  It's not very exciting, but it is one of the first things that's involved in setting up  the exec call.  The next thing we're going to look at is current execve.  This is the actual implementation of execve, and it's used by all the different system  call avis in previous Ds.  So if you call a Linux binary, it does a bit of work to deal with Linux enus, and then  calls current execve to actually execute the process.  So the first thing that happens in current execve is it looks up the path that you've  passed.  I'm not going to go into details because I don't understand name i, but it's magic.  You give it a path, it finds it.  It's all good.  And as George has been telling me all week, names are an illusion, so we don't want to  know anyway.  Next part is a little function which checks the permissions of the file, makes sure it's  executable by the user who owns the process.  It seems like a good thing.  And as a side effect opens the file, then exec map first page does what you might expect.  It maps the first page of the program so that it can examine the header and other parts  and other things, other image activators can look at that header and say, is this mine?  Is this something I can run?  Then we get into the meat of things.  So this traces on a 64-bit MIPS machine, obviously because that's where I've been doing all my  work, and because I have nice instruction level tracing.  So on there, the first image activator we find is the one we want, which is the 64-bit  Elf execution environment, image activator.  So it's going to try and load the program, get memory setup for it, and then set up  registers so it can return into the program.  So the first function we get is exec new namespace.  Its job is to set up the memory mapping of the process in the initial state.  First thing it does is it deletes all the pages from the memory mapping because it doesn't  care about whatever the contents were from the previous process that forked.  The next thing is that it maps a stack into the address space.  This was something that actually surprised me a little.  I had expected that stacks might be placed on a per system call interface basis, but  in fact, they're always set up here.  They can move around, so 1x could put the stack in a different place, but in practice  looking at the system, it's always the same on every architecture and free BSD, which  is a little surprising.  So now we have a stack and an otherwise empty address space.  So the next thing we're going to do is we're going to load some sections into memory.  So a typical simple statically linked program, which is what we're talking about, has two  sections in the file.  There's the text section, which is the executable bit of code, so it's all the instructions  and some other non-instruction things, but it's basically all the instructions that will  be executed.  And then there's the data section.  The data section is all the static data, all the dynamic values, or all the global  variables, all the initialization values for various things.  And then there's BSS, which is all the global and static variables that default to zero.  And so those aren't in the file, that saves space, but those pages are mapped at the same  time.  So back into current exec VE.  The next step is exec copy out strings.  This is a bit of an oddity.  It's actually, despite the fact that we're on MIPS, it does a bunch of things according  to the SCO I3D6 ABI, which happens to define how every elf executable, which is the standard  file format on every UNIX anyone cares about, except macOS.  So it defines how the process is laid out and where to find things.  So for instance, where to find the argument vector, where to find the environment vector.  And I'll get to the how this is laid out.  But what's interesting is it's actually just plopped on top of the stack.  And why?  Historical reasons.  But it's an interesting thing there.  And then finally, there is exec set regs.  Exec set regs, its job is to set up the registers in the trap frame so that when the system  call returns, it says though it returns to where the start process is, with the right  arguments.  And I'll talk about what those arguments are when I get to start.  So here we are.  We've returned back from Kern exec VE.  We go to sys exec VE and we get the standard system call return path, which is out of scope  today.  And we return to user space.  So the recap, at this point, the stack is mapped into the address space.  The program is mapped into the address space.  And a bunch of strings and whatnot are set up.  And then finally, the register status set up to call the start function.  So now to talk a bit about what's on the stack.  There's a whole bunch of things.  Their order is very strict, is in part strictly defined and in part loosely defined.  The top is PS strings.  It is used by the debugger and it points and by programs like PS to find program names  that are too long.  It doesn't appear to actually be necessary in modern software, but we're stuck with  it for existing ABI's.  Another oddity, SIG code is where signal hander rollers return to so that they can jump back  into the kernel.  And for some reason, we decided to put it here long, long ago.  And then there's a bunch of strings.  There's the stack canary.  There's some arrays of things.  Those are accessed by the elf auxiliary arguments, by the environment's array or via argv.  And finally, the last thing at the bottom of the top set of the stack is the argument  count.  So this is all the things that you need to pass to main in order to run a program with  standard arguments.  One oddity here is argc in C is defined as an integer, but in the ABI it's a long so  that it's always pointer aligned.  And so that stack aligned is preserved.  So lots of weird history here.  And so I would say what I forgot to tell you in the introduction is the reason that I'm  here is that I'm looking at all these things and finding all this history is that I'm in  the process of writing a new ABI for a memory safe variant of C with hardware enforcement.  So it turns out all this history and all these odd things that you just know you can find  them at the top of the stack, for instance, that's not OK if you want memory safety.  So we have to take all these assumptions, find them, tear them apart.  So here we are.  So you end up with the stack pointer at the top of the stack.  Additionally, the first argument to start is also that stack pointer.  And it's used for things.  I go into this again, but it's used to magically find things on the stack.  So this layout is all part of the application binary interface.  And it's part of how the process works.  So now let's get into start.  One thing you might observe, I'm looking at this, is that most of the arguments have  these JEE, or most of the functions have JEE something under them.  That's because, in fact, this program spends almost all of its time in malloc.  Something like 90%, 95% of cycles in Hello World are spent in malloc.  So part of that, so most of that is because JEE malloc is a malloc designed to support  giant multi-threaded programs.  One of the optimization test cases, in the first case, was Firefox.  These days it is whatever Facebook's PHP workload looks like.  So it is a big, heavyweight implementation designed for real problems.  So it doesn't really matter.  Hello World's not a performant thing.  You don't really care how long it takes.  But it's interesting that there's all this overhead.  And in fact, for some program, if you wanted a trivial program like this to be super fast,  you would definitely want to use a simple malloc.  And actually one of the things we found in our research is we were using a trivial malloc  in some of our benchmarking.  We realized we were doing a lot better than we should be.  There should have been more overhead compared to the unmodified environment.  It turned out, oh right, we hadn't turned JEE malloc on yet.  So we made that change in our tree recently.  So let's take a look at start.  I've condensed start onto two slides by stripping out all the conditional cases for dynamic  programming and dynamic libraries and all the conditionals for profiling.  So it's just the minimal thing.  So first up, we're setting up the argument vector.  So we have AP, which as you recall is that pointer to the top of the stack.  We know that that is in fact a pointer to argc.  So we get argc out of there.  And then we say, oh, well we know the next thing.  That's argv.  And then we know that the environment is just past that plus a null pointer.  And we buy argc.  And we'll get later in a moment to an even more bizarre example of finding things on  the stack.  The next thing that we do is there's this function handle argv.  Its job is to set the environment variable.  So it sets it to the pointer to the argument array.  And it sets the program name variable.  It's kind of a minimal thing.  Next up is init tls.  So this is in fact why that example program with just write was so bloated.  It turns out that we need to set up thread local storage.  And we do this unconditionally.  We need to do it in all cases because we don't know whether or not we'll be threaded.  So in a static program, we probably could avoid it.  But in a dynamic program, we must always set up tls because we might load a library later  that uses threads.  So we need to have thread local storage set up.  So for those who don't know, thread local storage is global or static variables which  are per thread.  So each thread has its own copy of that variable.  Again unsurprisingly, most of the time here is spent in malloc.  So let's talk about what init tls does.  So the first thing it does is it finds the elfxillary arguments vector.  It does that by this cute bit of code, which is it takes the environment array and it knows  that the environment array is null terminated.  And it knows the elfxillary arguments vector is just past it.  So it walks right off the end of the array on purpose.  And then it says, oh, that's the elfxillary arguments vector.  Good.  I'll just cast it and use it.  Great.  So it uses that to find the program headers.  So one of the many things in the elfxillary arguments in addition to things like the stack  canary and the array of page sizes and whatnot is a pointer to the maps in programs program  headers.  So that's the thing that tells you what all the sections are in the program, in the program's  binary.  It uses that to find the plt tls section.  This section contains the default values for all of the tls variables that aren't zero  by default.  So you might have a tls variable that's one because you want to do something, you know,  you want a condition to be true.  That will be stored in here.  And that's the beginning of the tls section is all those things.  So that can just be splattered right over the top.  Once it has that information and it's stored it so each thread can use it in the future,  it calls this libc allocate tls which allocates space and initializes values as necessary.  And this is where JML comes in.  So it needs to allocate space and that space needs to be freeable later by calling free  because you might start another thread and kill the original one.  So you can't just leak that and you don't want to have to record that maybe I'll replace  you know, you don't want to record that this one's special in some way.  So what happens is there's a special interface into JML which doesn't use thread local storage  and which does all the initialization bits but it avoids all the things that haven't  been set up yet because as I said JML is designed for a threaded environment so of course it  needs per thread pools of storage and whatnot.  So it makes use of tls.  And finally the tls pointer is set on a per thread basis.  On MIPS this is calling a special sysarc syscall which says set the tls pointer to this value  and then when you want to use it you call the syscall again usually via an instruction  that traps, that generates a trap because it's typically unimplemented and you get the value  back.  In CherryBSD we actually have implemented an optimization so we call the syscall to set  the value but then we set a user local register which is a special register readable by the  program or readable from user space.  Yes.  I think that's the first time you mentioned the name CherryBSD.  Oh sure.  So Cherry is our processor with memory safe extensions and CherryBSD is our fork of free  BSD.  I talked about it last year so I think there's probably a talk online if you're interested.  It is basic so it adds bounds checked pointers so we can do all the things that ASLR and  control flow integrity except we can do them in forced and hardware and fast.  So yeah so we've implemented some optimizations that one we could probably merge back to free  BSD.  I think it works on the entrader light so we probably should.  So back to start again.  We've initialized TLS and now we get into a handle static in it.  I'm not going to go dive deep into the code because it is a chain of boring and hard to  read code but basically this is where all the constructors are called and where destructors  are registered.  So as I said earlier there are four different kinds because people added them at different  times and wanted different things.  So I did it in Finae were the first things historically and I believe an array was added  next so that's an array of function pointers that's the bottom one that runs last and it's  called it's a sensible interface more or less.  And then I assume that people decided oh but we need to run some things early.  Some library setup stuff that might be called by other initializers.  So we got pre an array just because why not have a consistent and sensible interface when  you can have something dumb.  Actually what's really going on here is it's hard to get all the world to change their  linker and to add new features to all the linkers in the world so therefore you have  to write something that works with the old cruddy linker you have, not the new shiny  linker you would like to have and then you encode it into the ABI as is done here you  make it part of all the standard interfaces and then you're stuck with it forever and  ever.  I think we can kill in it in Finae.  We're doing it in CherryBSD we're going to see what happens.  And there's also the C++ constructors, the Ctors section and that's weirdly in FreeBSD  I think actually due to a GNU thing called through in it and that's the only thing that  gets called through in it.  That's a trivial fix but I haven't done it.  In any case what all these do is they call a bunch of functions which set various things  up.  The thing that actually matters here is that in the process the finalized destructor which  calls Finae and which calls there's a Finae array at exit get set up.  And then finally we're ready to call main.  In FreeBSD's start that is main and whatever main returns gets passed to exit and the exit  system call gets run.  So we'll get into main now.  Okay so here it is.  Let's see and I can barely read it.  So again you know it is mostly malloc.  So and in fact although it is also mostly VF printf.  VF printf is called by printf down here and then VF printf starts doing some work.  The first thing it does is it has to get the locale which is how it can look up things  like which decimal point is used.  And we'll get to the, that is in fact the reason it's used in this case.  But get locale has a bit of interesting machinery to it.  So locales are per thread because you might want to write a server where one person connected  is in a country where the decimal separator is a comma and another one in a period.  I don't understand exactly why you would want to structure your application this way but  this is in fact something you do.  So locales are per thread.  So we look up the locale and then we call this once function which is an interesting  bit of machinery.  It's job is as you might expect to do something once.  Either once per thread or once per execution.  So you can look up a value and then you can save it.  So what it's doing here is it's trying to find the locale path.  It looks at the environment, tries to find the path.  In the end it's not set in the environment so it uses the default.  So all that work to copy a string.  But it's very general.  And that's one of the amusing things about looking through printf.  As printf is trying to solve, as you walk through all this code, it's trying to solve  the whole big problem and that's why it's such a big beast to make two system calls.  So back into VF printf.  Now let's look at underscore VF printf.  Under score VF printf we get here by passing the locale that we got before to VF printf  L which takes an explicit locale and then we call VF printf which does all the actual  work of printing a string.  So I'm going to ignore all the stuff on this side because what this is is it calls into  the file object for standard out and it looks at it and says, oh, you're a buffered file  and you don't have a buffer yet.  I guess I better allocate one.  And so that's what all this code is, is basically allocating an array.  And I don't have time to get into jmalloc.  So we won't do that.  Now this is the part that's kind of the actual meat of printing.  First, this was a bit of a surprise and I think needs to be optimized out.  Printf no matter what it's printing, what's passed to it, always spends a bit of time  here finding out what the decimal point is on entry because you might print a floating  point number so you should look that up.  Next is a fairly simple thing.  As you recall, the string that we were parsing was a percent s, space percent d slash n.  So a string, a number, and a bit of white space.  So the way vfprintf works is for each block of data up to a format code, it assembles a  buffer of things to print out and then flushes those out at the end.  So in this case, it first finds the percent s string format.  So it puts that and the string in.  In here it looks up the length of the string, sets that in the buffer.  Here it adds a little bit that says, okay, there's going to be a space and then it says,  oh, there's going to be a number and it calls this function whose name I can't quite remember  whose job it is to write out the representation of the integer string.  One thing that I found was interesting when I was reading this code and modifying it for  some other reasons is it actually works backwards.  So it starts at the end of the buffer, of the temporary buffer and writes the first digit  or the least significant digit, the next, the next, the next, which is an easy way to  do it and I thought that makes a lot of sense.  And then I stumbled across three other implementations of printf in the tree, all of which do it  the other way.  In fact, we have at least four implementations of printf in the tree.  The runtime linker has its own, libc has its own, there's one in the kernel, and there's  I think one in the boot loader and I'm pretty sure there's some more.  All right, jmalloc has one too, because printf, as you've noticed, uses malloc and jmalloc  wanted to have a printf so they could print debugging information and then, you know,  deadlock is bad.  So there's a copy of printf in there.  And then we finally get to the last bit, so there's the new line at the end and we'll  zoom in on that because that's where the actual write system call goes.  So we have this single buffer to be written out.  At the beginning, when it's scanning the buffers, it first copies them all into one place and  then it looks at them and says, it scans the mall and says, oh, is there a new line anywhere  in this output?  Because if there's a new line and it's a buffered file descriptor, then we need to actually  call write when we hit that new line.  But it finds the new line, which means that this fflush call gets made later.  fflush in turn does a bunch of things, digs down, digs down, digs down.  And finally, we write the string.  So there we are.  Now that's the basic static hello world.  I said in my abstract that I would do dynamic hello world, so I will talk a little bit about  dynamic hello world.  Unfortunately, I was assuming that I would have finished modifying the runtime linker  for our project to support our new AVI and it being a software project, I didn't get  it done in time, so I don't actually understand runtime linking yet.  But I understand the basics, so I'll give you a quick tour of it.  Oops, wait, I'm sorry.  I am not there yet.  I have one more tiny bit.  We've returned from main.  We're back into start.  We get to exit.  And exit isn't actually call the system call directly.  It's a little more complicated than that.  Inside exit is a function.  Its job is to call destructors.  So this CXA finalize is a function whose job it is to call all the destructors that were  registered through the at exit function.  So finalize is one such destructor.  So that gets called, that calls FinA, cleans up all the, does a bunch of deallocation and  clean up.  In practice, it doesn't really do much.  The next part is there is this clean up function, which could use a more descriptive name.  So what it actually is, is it's a function that's called only if a C file object was  used during the course of the program.  It's used to flush out any remaining IO.  Then finally, we call the exit system call.  So okay, now dynamic libraries.  So here's a different trace, which you might notice is you can't see where main is.  Main is this little bit on the far, far right.  So for those in the recording, it's a little bit over here.  So dynamic library, dynamic linking has quite a lot of overhead.  There's also a few changes in the process layout.  So the first, so we'll start here at the beginning.  This is the same memory layout that you saw before.  You see the program on the left, you see the stack on the right.  And that's, the kernel loads the program just the same way, except that it also loads the  runtime linker.  The runtime linker and the other difference in the exec v is that it jumps into the runtime  linker's entry point rather than into the program.  The first thing the runtime linker does is it calls this long name, which as you can  tell, if you look at it, you'll notice it includes relocate itself.  That's because the runtime linker is position independent code and it needs to have its  variables updated and be relocated.  So it has a bit of slightly horrible code that makes sure that wherever the heck the  kernel decided to put it, it can run.  It's at least C. It walks at array.  It does some things.  It's not a big pile of assembly or something.  So once that's run and the program is relocated, it starts parsing the program itself.  It notices that the program depends on libc.  So it loads libc and it does an initial relocation of libc, sets up all the global variables,  and then sets up a table of entry points for the program to call into the symbols it needs.  And we'll get to that in just a moment.  So we have, let's see, we have libc linked.  Yes.  So libc is now loaded in.  One thing about RTOD is the kernel loads it as though it were loaded by the M-map call,  which maps a file in.  So it attempts to pick the same location in the kernel.  That seems sort of gratuitously pedantic.  But that's what the kernel currently does.  One thing I did notice when I was doing my initial tracing is that the location it chooses  is very poorly aligned.  And so it made it very hard to understand what the runtime linker was doing when I was  looking at an object dump file and looking at disassembly because everything was shifted  in fairly low bits.  So in CherryBSD I've moved it.  So it's aligned at 16 megs or something and it's much easier to do math in my head.  And then libc gets mapped by M-map the normal way at the end.  So then, so once libc has been initially relocated, we call in to start again.  So you can see way, way, way off to the right.  And this, I think, looks fairly familiar.  You have main here.  The stack trace is a little screwed up because I didn't get my tool fixed in time.  So add exit does exit.  And it's not part of the stack forever.  But what you'll notice here is that there are these RTLD bind starts after things like  printf, which has this funny name.  So let's take a look at that.  So what you see here is that printf has this long name, which is printf, and then at, at  this FBSD09.  That says that this is, it's looking for version 1.0 of the printf symbol.  It's looking for somebody to supply that, in this case libc.  That's in the symbol versioning mechanism, which is why we haven't bump, had to bump  the version of libc since FBSD7, which means your programs keep working unless they do  something like use libkvm or libutile, which are hard to version.  So there's the symbol.  And then when calls are made between modules in a dynamically linked program, they def,  by default, what gets called is this, this little bit of runtime linker code rather than,  so the symbol's not re, not pre-resolved.  This is because libc has a gazillion symbols, and they take a fair bit of time to resolve  and to look up.  So it'd be, it'd be very expensive on program start if we took all several thousand symbols  and started them up, you know, as you, you know, you see here, the whole runtime, you  know, there's maybe room for 20 of them in that total runtime.  So you wouldn't want to make, do all these resolutions.  So you call this little assembly stub here, and its job is to say, oh, you're calling,  you're looking for printf.  So it goes in and it wanders around in the libc symbol table, does a bunch of lookups,  finally finds the symbol.  And then it returns and it sets up the table.  So now the table includes the correct address of the symbol.  So feature calls, not that anything calls anything twice in this program, would, will  go direct rather than having this lookup occur.  It's called lazy binding.  So that's basically the same.  Otherwise the program's the same.  It's just that any time there's a cross module call, there's one of these lookups.  Otherwise dynamic, dynamic for def is, or dynamic hello world is pretty much the same.  So that is the end of my talk as I, hurry up, I'd happy to take questions.  I would like to request that you give me feedback either by email or this is a link to the conference  website.  That's anonymous as far as I know.  And the QR codes for that too.  So let me know, was the talk helpful or interesting?  Did it make sense?  What would you like to learn more about?  Maybe you wanted to learn about JMLock.  Maybe there was something you wanted to learn less about.  Something that was dreary and boring.  This is a big, messy thing.  Some of it's kind of that way.  So I'd be happy to take any questions you might have.  Yes?  You talked about copying the arguments from the old address post and new address post.  What about the copy out?  So let's see.  So the question is how does the argument copying from the initial process into the kernel  and then back out work?  So that allocate arguments, let me pull up the zoomable SPG.  I'm going to turn the internet back on.  Are you sending me messages, please?  Okay, so exact the copy in args, it allocates some kernel storage.  So it looks at the argument list and allocates some storage in the kernel and copies in all  the strings using the usual copy in primitive.  And then after the address space is reset, the...  So after the address space is reset, there is a stack.  And that's already mapped, that's just anonymous pages with the stack flag which says they  grow down rather than up.  It uses optimization.  And then exact copy out strings is job is to take those things we've copied in and  have destroyed in the parent process and copy them out.  So it knows that whole layout, so it knows PS strings is at the top and it sort of works  its way down just copying out through the usual copy out mechanism.  So it doesn't bypass the VM and just go blank to a page?  Does it's got no need to go through the VM?  Yeah, that's true.  So I think it could, I don't know.  If double triple book feet in your argument list, you have to copy it out.  You have to go out of the process to go through the VM to the pool.  So you probably could in fact do that.  It would make the setup more complicated.  Well...  Yeah, I mean this is quite a bit...  Yeah, I don't think the copy out is very expensive.  You don't lose very often.  But I think you certainly could do it.  The problem is that you would want to change the structure of copy out strings to...  You would have to compute how much it needed.  So you'd have to move who is responsible for creating the stack, which would be fine.  But you'd have to change that and then you'd have to have something that determined how  big that set of strings was going to be.  And then...  It's more than that though.  It's a bunch of ABI-specific things.  So you could...  I think if you reordered everything a fair bit, you could do it.  And I think part of the problem right now is there's quite a bit of plugability.  So exact copy out strings is the default for the architecture, but different ABIs have  different ones.  So you could do it, but you'd have to refactor all of the compact layers.  So be an interesting experiment to see if you got better performance or enough better  performance to deal with the overhead.  Anyone else have any questions?  Oh, sure, I got one over here.  I'll take it.  What is in PS strings?  Well, let's go take a look.  I think I remember the name.  There we are.  Yeah, so it is...  It's a pointer to one or more argument strings and to the environment.  It's used to update...  So it's primarily used, for instance, when you call set proc title to change the name  of the program, you then need to point to a potentially completely new malloc set of  arguments.  It's a little gratuitous in that, at least for the arguments, because you also call the  kernel and say, please set this, historically, at least, it was done this way so that you  wouldn't have to allow the kernel...  The kernel wouldn't have to accept an arbitrarily large title.  In practice, there doesn't seem to be an actual limit on the sys call that's...  On the sysctl that's used to set it.  So in practice, I don't think it matters.  So in Cherry ABI, I think I'm getting rid of this because the interface has a bunch of  other problems.  But it's...  Yeah, so it's just pointers to these once they've changed so the debugger can find them.  Part of why I say it's a bit gratuitous is that you could simply pass this same stuff  to the kernel because you only really care about it if you're writing out a cord up  or doing some debugging.  So if the kernel knows where these things are, it's fine, it can write an elf note  in the cord up and you can look it up there rather than sticking it at the top of the  stack and hoping no one steps on it.  You had another question?  Yeah.  What things in Cherry ABI do you kick off the stack?  First one that comes to mind is the Cherry ABI.  So what am I going to kick off the stack in Cherry ABI?  In Cherry ABI, which is the alternate mode where all pointers are hardware-protected  capability pointers.  So what I am planning to evict everything from the stack.  None of this stuff has any relation to the program stack.  So I'm going to kick it all off.  Right now, I'm currently writing it to that memory allocation because I guess for the  reason that I said there would be a bunch of work to change how things are done, it's  easier to just splat it to the stack.  Then I actually restrict the stack pointer so the stack doesn't include that stuff.  The pointer that's actually accessible to the program doesn't include any of those things.  So I plan to kick all that stuff off basically.  And eventually it'll just go somewhere else.  So yeah, auxiliary args should go.  And I don't really want them to be accessible from the rest of the program.  So I don't want there to be a global to access them.  I want you to have to get them through the runtime linker, which will be able to compartmentalize.  So the runtime linker can become a trusted linker and loader.  Yes?  Based on this, is there anything you're going to do to make it more accurate?  I thought about changing the way we found the PS strings, but Acoustic really didn't  like it, so we're not doing that.  What else are we doing?  Not a whole lot of the AVI stuff.  I guess we are, I think so, on Cherry AVI, the way we find all those auxiliary arguments  and stuff is there is a structure that's the interface between the kernel and user space.  So we pass a pointer, instead of passing a pointer to the top of the stack, we pass a  pointer to a structure that has an argv pointer and an argc and an nv pointer.  Did we end up doing that for our V8?  Okay.  Sorry?  We need to move around.  I think some of that disassociation should just be done.  There's actually no reason any of this should be at the top of the stack.  It's just where it is.  For the most part, the big change with Cherry AVI is going to be that you don't have things  next to each other that you can access one from the other.  That's not valid C and is something we can prevent.  So we need to actually pass proper pointers to things, rather than just knowing that things  are next to each other.  Then it occurs in the exit code.  So let's go back.  The cleanup routine does the flush in that case.  Way down here.  There we are.  Way off at the end, the end of exit.  Because you have ever done any IO on that file descriptor or attempted to do any IO on a  file descriptor or perhaps a buffered file descriptor, the cleanup routine is registered  and therefore it does the work.  So that's why you get this silly interview questions with multiple outputs and stuff.  Would you still do a flush if there were not any?  It walks the descriptor array and I think flushes things that have data buffered.  Do you want to just bring it in?  Yes.  So, yeah, it calls it, says, is there anything to do?  Well, thank you all for coming.
Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_0_419.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_420_2027.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_2028_3491.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_3492_5255.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_5256_6083.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_6084_6167.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_6168_6275.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_6276_6539.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_6540_7187.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_7188_11999.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_12000_12983.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_12984_13703.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_13704_14219.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_14220_14735.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_14736_16247.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_16248_17219.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_17220_18731.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_18732_20123.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_20124_22283.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_22284_22571.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_22572_23051.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_23052_23099.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_23100_24323.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_24324_26819.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_26820_29231.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_29232_30683.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_30684_31163.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_31164_32507.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_32508_32687.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_32688_33311.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_33312_33371.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_33372_38435.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_38436_42551.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_42552_45311.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_45312_46199.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_46200_47051.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_47052_50807.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_50808_51803.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_51804_51935.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_51936_52055.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_52056_52187.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_52188_52247.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_52248_52955.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_52956_54623.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_54624_59867.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_59868_62723.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_62724_62927.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_62928_63371.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_63372_63755.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_63756_65495.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_65496_65627.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_65628_65711.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_65712_66083.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66084_66203.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66204_66335.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66336_66395.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66396_66491.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66492_66575.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66576_66683.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_66684_67223.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67224_67283.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67284_67415.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67416_67547.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67548_67607.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67608_67931.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_67932_68003.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_68004_72047.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_72048_72119.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_72120_72179.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_72180_72227.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_72228_72383.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_72384_80279.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80280_80339.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80340_80423.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80424_80543.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80544_80651.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80652_80711.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_80712_82067.png Everything_You_Always_Wanted_to_Know_About_Hello___tIBVQDF2YCw_82068_82086.png