Protecting FreeBSD with Secure Virtual Architecture
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Criswell, John
In this talk, I will present our research on protecting FreeBSD applications and the FreeBSD kernel from attacks. I will briefly describe the KCoFI system which protects the FreeBSD kernel from control-flow hijack attacks (such as classic buffer overflow attacks) and the Virtual Ghost system which protects applications from a compromised operating system kernel. Both KCoFI and Virtual Ghost are built using the Secure Virtual Architecture (SVA) (an LLVM-based infrastructure for enforcing security policies through compiler instrumentation and hardware techniques). In this talk, I will present our work on using the Secure Virtual Architecture (SVA) to protect FreeBSD applications and the FreeBSD kernel from security attacks. SVA is an LLVM-based infrastructure that permits us to use compiler instrumentation techniques to enforce security policies on both application and kernel code. In this talk, I will briefly describe how we used SVA to implement KCoFI: a system that enforces control-flow integrity and code segment integrity on the FreeBSD kernel to protect it from control-flow hijack attacks. I will then describe how we extended KCoFI to build Virtual Ghost. Virtual Ghost protects applications from a compromised operating system kernel. I will describe how Virtual Ghost uses compiler instrumentation to prevent the FreeBSD kernel from spying on and corrupting private application data and how it prevents the kernel from maliciously modifying application control flow (while still supporting features such as signal handlers and process creation).
 It's not changing. Oh, there we go. Okay. All right. Good afternoon, everyone. My name is John Criswell. I'm an assistant professor at the University of Rochester. And today I will be talking about protecting free BSD with the secure virtual architecture. This is work that I did when I was a PhD student at the University of Illinois at Urbana-Champaign. And the work I'm going to be talking about today was joint work with Nathan Dutton-Hahn and our advisor, Vikram Advay. Now, my dissertation work starts with a very simple question. Do you trust your operating system kernel? Now, here's the interesting thing. You say no. And yes, some people say yes, and some people say no. But in fact, in practice, we do. And the reason why is because on top of our commodity operating system kernels, such as the free BSD kernel, we run applications. And we use these applications to process sensitive information. So we may buy things online using our credit cards from websites like eBay or Amazon.com. We may file our taxes, which includes sensitive information, such as if you're an American citizen, social security number, and your address. Medical data. My doctor happens to store my medical information on a Windows machine. And certain voting machines run commodity operating system kernels or operating system kernels based on them, such as Windows CE. Now, we run these applications on our commodity operating system kernels. But maybe this is not such a good idea. And the reason why is because commodity operating system kernels are vulnerable. Now, they're vulnerable for two reasons. First, all commodity operating system kernels today are written using C and C++. And as a result, they suffer from the same security vulnerabilities that applications suffer when there are coding mistakes that are made. So for example, commodity operating system kernels have been vulnerable to buffer overflows and integer overflow attacks. Additionally, there are vulnerabilities that occur in operating system kernels due to their job as the operating system kernel. So for example, the operating system kernel provides the process abstraction, which is supposed to isolate different processes, different programs from each other. But sometimes logical bugs in the operating system kernel allow information to leak from one process to another unintentionally. So for example, through misconfiguration of the MMU, one process may be able to access the memory of another process. Finally, all commodity operating system kernels to date are dynamically extendible. You can load new drivers, load new kernel modules into the system as it is running. Now because of how the operating system kernel is structured, these new modules, when they're loaded, can modify the operating system kernel behavior in arbitrary ways. And so attackers have taken advantage of this. They've written kernel level malware, typically to hide their presence, such as hiding processes and files and network connections. But because they can use any arbitrary behavior that they want, they can do things like stealing data from applications, corrupting data within applications, even modifying application control flow. Now here's the real kicker. If the operating system kernel is exploited, then all security guarantees on your system are null and void. If your operating system kernel is compromised, you have no security on your system. And the reason why is because nearly all security policies are either enforced by the operating system kernel itself. So if the attacker controls the operating system kernel, he or she can just turn the security policy off. Or they are enforced by applications running on top of that operating system kernel. And so because monolithic operating system kernels have such a great amount of control over the system, they can just reach into the application, change its code, change its data, and therefore turn off the security policy enforcement within the application. So if the operating system goes, everything goes with it. Now there are two approaches to addressing this problem. If you don't want to abandon existing commodity operating system kernel. So if you don't want to rewrite it, if you don't want to do massive restructuring and refactoring of the kernel. The first is to automatically harden the operating system kernel from certain classes of attack. And I've looked at this in my PhD dissertation and built systems such as SVAM, which enforces strong memory safety guarantees on commodity operating system kernels, namely Linux. And then more recently, the coffee system, which enforces a lighter weight security policy called control flow integrity on the free BSD kernel. Now, while these approaches are good, one limitation that they have is they only address a particular class of attacks. So the SVAM and coffee systems, for example, address buffer overflow attacks and attacks of that nature. But they don't address things like kernel level malware that's loaded into the operating system kernel. They don't handle information leaks. They don't handle missing access control checks. So another approach that I explored in my dissertation is to just assume that the operating system kernel becomes compromised. Just raise up your hands and say, OK, we're just going to assume that the operating system kernel can be compromised. Can we run applications securely on that kernel anyway? And so we built a system called VirtualGhost, which provides data confidentiality and integrity for applications running on a potentially compromised operating system kernel. Now, today in this talk, I'm going to talk about a little bit about coffee and VirtualGhost. Most of the talk will be about VirtualGhost, because in fact, VirtualGhost is actually built on top of the coffee system. So we have control flow integrity for the free BSD kernel, and then we're going to build VirtualGhost on top of that. And so the contributions of VirtualGhost is that it protects application data confidentiality and integrity, as well as protecting other features of the application. It uses compiler techniques. And because it uses compiler techniques, we can use the same processor privilege level as the operating system kernel. We do not need to use a hypervisor based approaches, if you will, where we kick the operating system kernel up into ring one on x86, or have VMM extensions that allow us to run code below the operating system kernel. We can actually run alongside the operating system kernel. And it turns out that although we're using compiler instrumentation techniques to add instructions to the operating system kernel code, we are still faster than hypervisor based approaches. So that's a brief overview of the research work. I'll now talk about the design of VirtualGhost, and then take a small aside and talk about some of the design and implementation of coffee. I'll then talk about our experimental results. So a little bit about the implementation and the experiments that we run on coffee and VirtualGhost. And then finally, I'll conclude with some future work that we're doing at the University of Rochester. All right, what is the fundamental problem with current system design? The fundamental problem is that applications cannot protect themselves from the operating system kernel. So if I want to write an application that does not trust the operating system kernel, I will be inclined to write an application that just keeps its data encrypted as long as possible. So encrypt data, when it sends it to the file system or to the network, it may even keep data encrypted while it's stored in memory. The problem is that even if the application does this, it cannot protect itself from the operating system kernel. The reason why is the operating system kernel can access anything and everything on the system. So the operating system kernel can just reach in and read unencrypted data out of the application or modify the encryption keys to some value that the operating system likes. The operating system kernel can modify the applications code so that it just doesn't encrypt anything at all. It can even modify the application's control flow, it can stop the application, change its program counter, and then resume the application, causing it to skip over encryption and decryption operations. So no matter how hard an application tries on an existing operating system kernel, such as FreeBSD or Linux or Mac OS X, it just simply cannot protect itself. The goal of VirtualGhost is to build a system that allows applications to protect themselves. And there are three features that such a system requires. First, applications require private data and private code. So they need, they're still gonna need public memory that the operating system kernel can read and write so that they can communicate with the operating system kernel, but they're also going to need memory that the operating system kernel cannot read and cannot corrupt. Secondly, they need incorruptible control flows. They need to know that when they start execution, they start execution in their main function. And they also know that if they're interrupted by a trap or a interrupt or they execute a system call, they have to know that the operating system kernel is not gonna be able to modify their control flow maliciously while the operating system kernel is running. Third and finally, applications that protect themselves are going to need a reliable way of getting their encryption keys from some file, typically the executable image, from the executable image into that private data memory region without the operating system kernel being involved. Because if the operating system kernel is involved in that process, it can change the encryption keys, it can read the encryption key values, thereby defeating the purpose of having the application encrypt its data when it sends it to the operating system to begin with. So these are the three things that we need. How hard could it be? Well, there's two challenges. The first challenge, the more obvious challenge I think, is that modern processor design assumes that system software like the operating system kernel should be able to access all of memory. So that's challenge number one. But there's actually a more subtle challenge when you start trying to do this. The more subtle challenge is that if we want operating system kernels to provide the features that we expect them to have, they must be able to manipulate application state. It doesn't suffice to put an application over here and the operating system kernel over here and say, never the two shall meet. Because if you did that, the operating system kernel would not be able to create new processes and new threads. It would not be able to execute new programs by providing the exact family system calls. It would not be able to provide signal handler dispatch because all of these operations modify application state. They go in and make changes to the application's program counter. So instead of preventing the operating system kernel from manipulating application state, we must allow it to do so, but we must control what it does. We must ensure that it makes good state modifications, but not bad state modifications. All right, so we're gonna need some infrastructure to do this. We're going to use from our previous research of the secure virtual architecture. So in the secure virtual architecture, instead of compiling the operating system kernel down to native code, and instead of having inline assembly code written into the operating system kernel, instead of what we're going to do is we're going to compile the operating system kernel to a virtual instruction set. And we're going to design this virtual instruction set to be easy to analyze an instrument. Now, handwritten assembly code is not easy to analyze an instrument. So what we're going to do is we're going to port the operating system kernel to our virtual instruction set. We're going to have a set of instructions in that virtual instruction set called SVOS, which we can use to replace inline assembly code. The SVOS instructions basically provide features such as registering, interrupt, and trap handlers, being able to configure the MNU, being able to manipulate interrupted application or interrupted program state. In this way, we could represent an entire operating system kernel in the virtual instruction set. The operating system kernel will have no inline assembly code, no handwritten assembly code in it at all. Now, you can't actually run a virtual instruction set operating system kernel. You have to translate it to native code, like x86 or ARM or PowerPC or MIPS, in order to run it on a real processor. Now, when I give this talk, some people tend to assume that, okay, you're kind of doing something like Java. So you have a virtual instruction set, you have a native instruction set. So the translation must happen just in time, just like Java does. No. Secure Virtual Architecture is designed so that translation can happen anytime you want. It can happen ahead of time, it can happen at system boot time, system install time, run time, idle time. In our implementation, in our prototype, we implement translation ahead of time, because A, it's easier to do, and B, it's more efficient. Now, let's look at the virtual instruction set in a little bit more detail. It's comprised of two components. The first component is SVA Core. SVA Core is taken from the LVM Intermediate Representation. So it's the language that the LVM compiler uses to analyze and optimize code. It has source level types, it has explicit static single assignment form. These are things that allow us to do sophisticated compiler analysis and instrumentation on the operating system kernel code. Now, the SVA Core instruction set is what you would call regular computation. So it provides things like adding and subtracting, pointer arithmetic, reading from and writing to memory, those sorts of things. Now, the LVM IR, if you take away the inline assembly code feature, can't support an operating system kernel. You can't express the FreeBSD kernel or the Linux kernel in LVM IR. So we extended it with a new set of instructions called SVOS. These are operating system neutral instructions. We've used them with both Linux and FreeBSD. And they encapsulate the state manipulation and hardware configuration operations. So again, they provide instructions like configuring page table pages, modifying interruptive program state for signal handler dispatch, registering system call handlers and trap handlers and things like that. Now, what's interesting about that, not only do they encapsulate these operations, but because the operating system kernel has to use these instructions to interface with the hardware and to manipulate state at all, it allows us to control how the operating system kernel does these things. So we can control how the operating system kernel configures the MNU. We can control how it manipulates application state. And so what we do is when we implement these instructions, these SVOS instructions, we add runtime checks to them to help enforce the security policy that we're also enforcing with the compiler instrumentation. So one implementation of the SVOS instructions helps enforce control flow integrity for the coffee system. Another implementation implements the runtime checks that we need for virtual ghosts to protect applications from the operating system kernel. Now implementation wise, what we do is we implement the SVOS instructions as a native code runtime library. So essentially what happens is you express your operating system kernel in the virtual instruction set. You then analyze an instrumentate, convert it down to native code. The implementation of the SVOS instructions is missing from that native code. So the runtime libraries linked in provides the implementation of those instructions. You now have a complete native code kernel that you can boot on real hardware. All right, now how are we going to use secure virtual architecture to implement virtual ghosts? How are we going to provide those three features that we need? Well let's first look at private data and private code. So most operating system kernels such as FreeBSD divide the virtual address space into two partitions, the user space partition and the kernel partition. So user space memory is where the application lives, kernel space memory is where the kernel lives, and the kernel is allowed to access both user space and kernel space, whereas applications can only access user space. Virtual ghost is going to add two new partitions to the virtual address space. The first one is ghost memory. Ghost memory is memory that the application is allowed to read and write, but the operating system kernel is not allowed to read and write. So this is where the application is going to put its private data and its private code. Now what's going to happen is that implementation of the SVOS instructions, that runtime library, it has its own data structures that it uses in implementing its runtime checks. Those data structures should not be accessible to applications or the operating system kernel. So there's another region of memory called the virtual ghost VM memory or VM memory for short. This is where the SVOS data structures go. This region is not readable or writeable by applications or the operating system kernel. So essentially we provide this new, the two new regions, contiguously in memory, that the operating system kernel is not allowed to read or write. Now by this time you're probably asking me, John, how do you keep an operating system kernel from writing into ghost memory and VM memory? The trick to the secret sauce is solve for fault isolation instrumentation. So what we do when we're translating from virtual instructions set code to native code, is we look for all the load and store instructions in the operating system kernel and we add some instructions before them. What these instructions do is they check to see whether the pointer that's gonna be used in the loader store is pointing into user space memory or kernel memory. If they are, it's fine. The pointer does not need to be changed. If they are erroneously, if the pointer is erroneously pointing into ghost memory or into the virtual ghost VM memory, then a simple bit masking operation moves the pointer into kernel memory. In this way, we are guaranteed that all loads and stores are either accessing user space memory or kernel memory, but not ghost memory and not VM memory. Now, let's say your operating system kernel unfortunately has a buffer overflow in it. An attacker could potentially use that buffer overflow to change the control flow to jump over these new instructions that we've added. And that would allow the operating system kernel to access ghost memory or VM memory. And that would be bad. That would violate the virtual ghost security properties that we're trying to enforce. So what we do is we use control flow integrity. By using control flow integrity instrumentation along with software fault isolation instrumentation, we can ensure that the software fault isolation instructions are always executed. They can never be jumped over even if your kernel has a buffer overflow or other sort of memory safety error. In addition, the control flow integrity instrumentation helps protect the operating system kernel from buffer overflows and other related attacks. So we sort of get a two for one deal with virtual ghost in that we can actually protect the operating system kernel from attack and protect applications from the operating system kernel with one set of instrumentation. All right, so now we have our private data and our private code. What about secure application control flow? Well, why isn't application control flow secure today? The reason why is because on an interrupt trap or system call, the hardware transfers control flow to the operating system kernel and the operating system kernel saves the interrupted program state on the kernel stack. The kernel stack is in kernel memory, it's in kernel space, the operating system kernel can just read and write that as it likes. So in virtual ghost, what we do is when there's an interrupt trap or system call, the hardware transfers control flow to the virtual ghost runtime library, that implementation of the SVOS instructions. That runtime library then saves the interrupted program state not into kernel memory, but into the virtual ghost VM memory. Then virtual ghost transfers control flow to the operating system kernel. So now the operating system kernel can respond to the interrupt or the trap or the system call as appropriate. But now when the operating system kernel wants to modify application state, it can't do so directly. It's sitting in that virtual ghost VM memory where the operating system kernel can't touch it. So the operating system kernel wants to make a change to say the program counter or the stack pointer, it has to ask virtual ghost to do it through an SVOS instruction. So in this way, the SVOS instructions can vet changes to save program state. And if the changes are okay, then go ahead and make those changes on behalf of the operating system kernel. So in this way, we can actually control operations such as signal handler dispatch, thread and process creation and the exec family of system calls. As an example, let's look at how the exec system call is implemented on a virtual system. So you have the operating system kernel, there's some application executable that it wants to execute. There's some program that has executed the exec system call, so it's been interrupted and we are now running in the kernel code. So the kernel says, hey, virtual ghost, I have this application executable here, please set up the code segment for this executable and please change the save program state so that when I resume it, when I put it back onto the CPU, it will start up in the main function of this executable. So what virtual ghost does, it says, okay, I will go and I will set up the application code segment in the applications ghost memory. So now we have the applications code segment, it's in ghost memory, the operating system kernel cannot arbitrarily modify the applications code, but the application can use it. And then virtual ghost will locate the virtual address of the main function and change the program counter to point to it. And then it returns back to the operating system kernel. So when the operating system kernel does the return from system call, what will happen is the save program state will start executing this new application code in its main function. So what we've done is we've essentially taken the operating system out of the path, the critical path, where we rely on it to do the correct thing. Instead, virtual ghost does the critical operations of setting up the code segment and changing the program counter to point to the main function. All right, so now we have private data and private code, we have secure application control flow. The last thing we need is secure application encryption keys. So let's say you're an application developer, you've written an application, you generate a public-private key pair for that installation of the application, and then you're gonna run this on a virtual ghost system. So if you just embed the application code and the application key into the executable file, and then you send that over to a virtual ghost system, your operating system kernel can do one of two nasty things. The first thing it can do, it can simply say, well, I like the code, but I don't like that encryption key because I don't know what it is. So what I'm going to do is I'm simply going to replace it with my own public-private key pair. So when this application goes and encrypts data, I know what key it's using. Alternatively, the operating system kernel may say, well, I like that application public-private key pair, this application came with some files that are encrypted using those keys, but I wanna be able to see what's in those files. So I'm going to change the code in the application, so that it'll decrypt the data with that application key pair, and then I can see it. What we need to do is we need to tie the application code and the application key pair together, so that if they are tampered with by the operating system kernel, we can detect the tampering and refuse to run the program. So every installation of virtual ghost will have a public and private key. The virtual ghost public key will be used to encrypt the application code, the application key, and then a checksum of the combined application code and application key pair will be encrypted with a virtual ghost public key. What this allows us to do is when the operating system kernel comes along and says, hey, virtual ghost, please set up the code segment for this application, virtual ghost can verify that the code has not been modified and the key has not been modified, and then in fact, this code and this key actually go together. It's the right code and the right key paired together. If it hasn't been tampered, then what virtual ghost will do is it'll set up the code segment as we talked about a few slides ago, and then it'll decrypt the application key pair and put that into ghost memory. In this way, the operating system kernel is never involved in getting that application key pair out of the executable and into the process as ghost memory. It's not involved in that process, and so as a result, it cannot corrupt that process. Yes? If we are just sending encrypted to virtual ghost, the application code is to be paired with a hash, but if the hash is assigned with the same key pair, then can we mix and match to the text less, send data and then hold back and send previous code or something? It does the code that's sent, and then the second message, we have some other key pair that's sent both or encrypted to virtual ghost, but then when we receive some other code and key pair later, we could send it or do out of order. I'm not sure if I quite understood. Are you essentially saying if I'm an application developer and I create one version of my application and I ship it, is it possible that I can send another version of the application? Yeah, I can send, I can say, all right, let's use the old version of the application, which I have to do, I guess some of the apps will have to do that. Right, right, okay. So I think my answer to that is that if the system that the developer is using is running on virtual ghost, or if it's a system outside, so it's like an app store from Apple or something, then that should not be a problem, because the operating system kernel can't get in and modify that. It shouldn't be able to corrupt the process and create the second version of the application, correct? So although I think there is an attack though, where the operating system kernel also actually, so I think what you'd probably want to do is I think what you'd want to do is every time you update an application, I think you would actually have to give it a new application key pair, and then re-encrypt all of the files with that application key pair. So yeah, I had not thought of that, thank you. Any other questions? Yes? So I have a program, it's in my brain. I am running low on available memory, I want to swap it out. So this segment of code comes from this user space VM, goes to disk. Now this program is executing again, I wish to put it back in place. At this point, the kernel will own the buffer that has just been taken from disk, which has the application code on it. I modify that code to be something that reads from the secret memory, which I can do, because that will be executed from the context of the other process. I then swap that back in, allow it to continue executing without changing its construction point or whatever, but then I still have arbitrary code execution from the context of the process, I use that to scale the keys and I go create it. I don't see any way to protect against that without having the entire everything, it doesn't work. So let me rephrase your question, you're asking how to secure swapping work, right? So as it stands right now, you can't swap out anything in ghost memory, right? The operating system cannot swap out anything in ghost memory, because it can't read it and it can't write to it, right? So first you said, okay, let's say you swap out the code segment or the application key pair, right? Well the operating system can't do that because it's in ghost memory. Now in the paper we talk about a design for allowing secure swapping, so if the operating system kernel wants to swap out memory that happens to be mapped into ghost memory, it can do so and what virtual ghost will essentially do is essentially encrypt and digitally sign the contents and then allow the operating system kernel to have access to that. We haven't implemented that because we haven't run into a case where we need to actually swap ghost memory and things like that out. I don't need to swap ghost memory, I can swap any memory that's in that process. So, executable memory should be in ghost memory, because if it's not, yes. Because otherwise the operating system kernel can corrupt it. Okay, yep. Yes. Do you account for modifications that might happen by directly accessing? Yes, so we haven't implemented it, but that's easily solvable through IO MMUs. So you just get, so you have virtual ghost control access to the IO MMU and the operating system kernel cannot reconfigure the IO MMU, so the virtual ghost runtime, the implementation of the SVS instructions, basically configures the IO MMU so that the operating system kernel can't use DMA to get information into and out of physical memory that's mapped into the ghost memory region. What I don't really understand about this, but you can sort of, we still affect the behavioral system called the process and both names and also modified it, trap frame. You can't modify the trap frame. Because, so the trap frame, we call it interrupted program state. That's saved into the VM memory, which the operating system kernel cannot access. So, or at least it can't access it without going through virtual ghost and virtual ghost. The interface of virtual ghost combined with this runtime checks ensure that the changes are not going to violate the application's control flow integrity. So what if I would just, you know, assume that the program is both the MNAP system call, I just give it back a buffer that's not within the ghost. Excellent, excellent. So you have just reinvented the IAGO attacks. So there's an attack called the IAGO attack in which since the operating system kernel can't access the application memory anymore and it can't corrupt the control flow, it just says I'm going to return bogus values from system calls. And when I do that, maybe I can trick the application into doing something it doesn't want to do. So maybe I return, MNAP returns a pointer into ghost memory and the application just assumes that's okay. So in future work, I'm actually having a student work on that. So I will get to that in basically my last slide. But yes. Yes, just out of curiosity, you said that you created the protected ghost message by instrumenting at compile time all the old memory attack. Yes. But are you even able to catch the kernel trying to patch itself to create a new memory attack to the front end? That's because we don't allow the kernel code segment to be writable, which I'll get to in a few slides. Okay. But the second thing is the price of the bit, so if you want to use it, okay. Yeah, so in our useNIC security 2009 paper, we have a solution for the limited amount of patching that Linux actually tries to do. So, but in general, the operating system kernel is not allowed to make arbitrary changes to its native code segment. Yes. And that has to hold your previous line as well. I wouldn't be able to play that protective of the previously thought-brained thing for product that. Actually, so I haven't really thought about that. I think user space is actually fine, and we'll get to the reason why in a minute. All right. All right, I'm gonna move on, and we're gonna go to coffee break. All right, so coffee is essentially a subset of virtual ghost. So coffee divides the virtual adder space into three regions. It's got the user space region, the kernel region, and then the coffee VM memory region. So this is the virtual ghost. VM memory just renamed as coffee VM memory. Coffee provides control flow integrity so that we know where we're jumping to in the kernel code segment. It gives us code segmentary, so the operating system kernel code does not change. It gives us software fall isolation on stores. So, unlike virtual ghost, which has to maintain the confidentiality and integrity of VM memory, coffee only needs to protect the integrity of its VM memory. So it only has to do software fall isolation and documentation on stores. The SVOS checks that its SVOS runtime library does, only do control flow integrity and code segment integrity, and it doesn't have the encryption key delivery feature or other virtual ghost specific features. So basically, coffee is what gives us the control flow integrity and code segment integrity that virtual ghost requires, and then virtual ghost builds on top of that the features to protect applications from the operating system kernel. How does coffee work? Well, it could enforce the control flow integrity in multiple ways. In our prototype implementation, what we do is we use the approach from Zangton and Moorset in their CCS 2011 paper. So what we do is when we translate to native code at the beginning of every function, and after every call instruction, we insert a special no op instruction, a no op instruction that does not appear anywhere else in the kernel code segment. Then we instrument all computed jumps, so every return, every indirect function call, to first take the address that the kernel wants to jump to, and bit mask it so that it's not pointing into user space memory. So someone asked me, well, what about instructions in user space memory? Well, instructions in user space memory, their kernel code, the virtual ghost has to set up the code segment so that it's, I think virtual ghost has to set up the code segment so that it's executable. So I think virtual ghost still has to set up the code segment, but we allow arbitrary native code to be run in application space because we control what the kernel executes. So as long as the kernel is not tricked into running user space code, which is what the bit masking does, we're fine. So we bit mask the pointer, the address, to make sure that it's in the kernel code segment, and then what we do is we check to make sure that where we're jumping to, actually contains one of these no op labels. If it does, great, if not, then we use a direct branch to jump to some error handling code. Now for returns from interrupts, traps, and system calls, so in modern operating system kernels, the operating system kernel can actually interrupt itself, so the operating system kernel can experience a trap or an interrupt while it's running. So just like application state, we save that state into the VM memory where the operating system kernel cannot directly modify it. So in this case, we prevent accidental overriding through buffer overflows of changing the program counter of the operating system kernel once interrupted, or changing the privilege level of the operating system kernel once interrupted. We also have instructions that allow us to do the exception unwinding that's done on efficient implementations of copy in and copy out. So we can still use the MMU to catch faults in copy in and copy out operations, and securely unwind the control flow while being control flow integrity. Now as I said before, we also provide code segment integrity, so by controlling the access to the page table pages, and by controlling the MMU, both coffee and virtual ghost ensure that the code segment is never made writable, there are never any changes that map new data or new code into the code segment. There is an instruction that allows you to dynamically extend the code segment of the operating system kernel. So if you want to load a say a device driver, you can give virtual ghost or coffee the virtual instruction set code, it'll translate that down to native code, do the instrumentation, and then add that to the kernel's code segment. But otherwise the operating system kernel is not allowed to make arbitrary changes to its native code segment. All right, back to virtual ghost. All right, so now I'm moving on to the results. So we implemented a prototype of virtual ghost and coffee for 64-bit x86. We ported FreeBSD9 to virtual ghost, if you're wondering why it's so old, it's because we started this, I think back in 2012. So the trusted computing base is about 5,300 source lines of code, this is the size of the compiler passes that we wrote and the runtime library that we implemented. And then we modified several applications from the open SSH application suite to use ghost memory. So the SSH client, the SSH key generating program, the SSH add utility, use ghost memory for their heap. They really should be using it for globals and stack, but in our prototype implementation, we're only doing the heap. And basically what they do is they create authentication keys that are encrypted so the operating system kernel cannot read them, cannot access them, but the SSH client and the SSH add utility can use them. All right, this is now released as open source software. We released the LVM compiler extensions, the virtual ghost runtime library, and a patch to the FreeBSD9 kernel code, which gives you the port of FreeBSD9 from x86 to the secure virtual architecture virtual instruction set. This is available on my GitHub account, github.com, slash jtcriswell with two Ls. There's also a link to this from my homepage at the University of Rochester. All right, now the first experiment that we ran was we wanted to see whether we could stop a sophisticated malware attack, and specifically one that was designed with virtual ghost in mind. So what we did is we wrote a malicious kernel driver. And what this malicious kernel driver does is it tries to set up a false signal handler within the application that will try to copy data between ghost memory to traditional memory. If it can do that, then it can just read the malicious driver, it can just read the data out of traditional memory. So if it can get a mem copy instruction to act as a signal handler, then it can steal data out of ghost memory by tricking the application into copying it into traditional memory. Now this is way more sophisticated than what you need for standard FreeBSD, and standard FreeBSD there is no ghost memory, so you can just read data straight out of memory. So nevertheless, this works on native FreeBSD. It doesn't work on virtual ghost, and the reason why it doesn't work on virtual ghost is because virtual ghost is protecting the application saved program state. When the application starts running, it tells virtual ghost through a system call that goes straight into the virtual ghost runtime library, here is where all of my signal handlers are at. So when the malicious driver says, hey, virtual ghost, please change the program counter to point to this mem copy instruction, virtual ghost says, that's funny. The application didn't say that that mem copy instruction was a signal handler, so no, I'm not going to change the saved program state at all. I'm going to leave it completely unchanged. So operating system kernel, if you continue to run this application, it's gonna continue running right where it left off before the interrupt trap or system call. So that's how we stop this rather sophisticated piece of kernel malware. All right, how does virtual ghost perform in terms of execution time? So we wanted to compare virtual ghost to other approaches. Other approaches try to magically encrypt application pages when the operating system kernel tries to access them, the most recent of which is a system called intag, it uses VMM extensions in the processor. We compared our LM benchmark results to intag, and what we found is that we do quite a bit better than intag, so virtual ghost is usually about 4x to 5x overhead on system call latency, normalized to native, whereas intag is more like 7x to 9x. Also, virtual ghost has very low overhead on some key benchmarks, so for example on page faults, virtual ghost only adds 15% overhead, whereas on intag, you have 7.5x overhead. Now this is not an apples to apples comparison because intag uses Linux, we use FreeBSD, we're using different machines, we're not even using the same version of the LM benchmark suite, but it gives us a ballpark figure, and the ballpark figure says that virtual ghost is doing pretty well. Now, what about comparing virtual ghost to coffee? So what we see is this is our LM benchmark results comparing coffee to virtual ghost. Because of the additional software fall isolation instrumentation on loads, virtual ghost does incur a fair amount more overhead than coffee does. So one thing that this tells us is that the software fall isolation instrumentation actually does matter, it is actually hurting performance. So if we could get rid of it, that would be nice. All right, now that's micro benchmarks, that's latency of system calls. Most applications do not spend most of their time executing system calls, they spend most of their time doing computation. So we wanted to see what is the effect of performance on actual applications. So what we did is we took two network servers, THTPD and SSHD, and we ran performance experiments on them. We ran our experiments on an isolated one gigabit per second network, and the reason why we chose network servers is because unlike other standard benchmarks like the spec benchmark, they spend a significant amount of their time in kernel space. They spend a lot of time actually using kernel services, and therefore the overheads that we're adding to the kernel are more likely to show up. So our first experiment, we took THTPD, we used Apache Bench to transfer files between one kilobyte and one megabyte in size, configured Apache Bench to use 100 clients, 100 clients app operating parallel, doing 100,000 requests, and what we see is the performance overhead is negligible. Now I haven't shown the coffee numbers, mainly because we configured Apache Bench a little bit differently, we used I think 32 clients instead of 100,000, but the results are essentially the same, negligible overhead for THTPD. All right, that's multiple clients. What happens if what you want to do is you just want to transfer one file, transfer it over to the network as quickly as possible, and using cryptic connections so that no one changes your file or sneaks up on the contents of your file during the transfer. So what we did is we took an unmodified SSH server, so this is not using ghost memory, this is an unmodified SSH server running on a native free BSD system, the coffee system, and the virtual ghost system. We used an SCP client on another machine to transfer files between one kilobyte and one gigabyte in size, and we measured the bandwidth that we get through the verbose mode on SCP. What we find is that coffee incurs a 27% reduction in bandwidth in the worst case, whereas virtual ghost reduces the bandwidth by 45% in the worst case. So the overheads are not completely terrible, but obviously there's room for improvement. All right, now what happens if you use ghost memory? So far these experiments have basically been showing what happens to existing applications if you put them on a coffee system or a virtual ghost system. What happens if you start using ghost memory? Is there any cost to that? So we took our SSH client, which is using a wrapper library that copies data between traditional memory and ghost memory when it wants to do read and write system calls. So we took this ghosting SSH client and we ran it on a virtual ghost system and we used it to transfer files from another system between one kilobyte and one gigabyte in size, and then we also tested the original SSH client. So this is original SSH and ghost memory SSH, both running on the virtual ghost system. What we find is that there's a 5% reduction in the worst case, typically for these larger file sizes. Why is it for these larger file sizes? Well, we suspect that overhead is coming from the fact that we're copying data between traditional memory and ghost memory when we're doing reads and writes. Now here's the good news. The good news is that a lot of this copying is unnecessary because SSH is encrypting and decrypting data as it sends it to and receives it from the network. Encryption and decryption has an implicit copy operation. So what this SSH client is doing right now is when it receives encrypted data, it takes the data, the operating system kernel puts an intraditional memory, SSH copies it into ghost memory because that's what the wrapper library for the read system call does. Then it takes that encrypted data and ghost memory, decrypts it, makes another copy in ghost memory. So if we hand-tuned SSH, what we could do is the encrypted data comes into traditional memory, then it's decrypted and copied into ghost memory in one operation. So instead of having these two copies, we only have one copy. So in a nutshell, while 5% reduction isn't bad, we think we can do better. All right, future work. So one of the things that we're gonna have a student start working on soon is replacing and reducing the compiler instrumentation. So as you've seen, the compiler instrumentation, while it is better than using the VMM extensions in the way that InkTag did, it still has overhead that's not negligible, at least on the Allen Benchmark Suite and on applications like SSHD. We think we can replace the software fall isolation instrumentation using address-based identifiers or ARM domains, or perhaps only making small modifications to the MMU of a processor to provide the isolation features that we need. If we can remove the software fall isolation instrumentation, we should be able to significantly improve the performance of both coffee and virtual ghost. A second thing that we're working on is we're working on defenses against Yago attacks. So someone here, I believe, mentioned the Yago attacks. So Yago attacks, again, are attacks in which the operating system kernel returns bogus values to an application through the system call interface to try to trick it into doing something that it doesn't want to do. Our observation is that this is essentially an application trusting low integrity data. So by using standard programming language information flow techniques, we can basically check whether an application is doing computation on high integrity data and whether that computation is being influenced by low integrity data from the operating system kernel. So in this way, we should be able to build a system that formally verifies that an application is not vulnerable to these Yago attacks. We're also looking at the automated, we're also building a system that will automatically determine the efficacy of control flow integrity and code pointer integrity. So if you're following the control flow integrity literature, coarse grain control flow integrity, now there are now, sorry, there are now new attacks against coarse grain control flow integrity that allow attackers to compute, to perform malicious computation. So we now have this open question of, how good of control flow integrity do you need? So obviously, coarse grain control flow integrity where you don't distinguish between different called targets or different return targets isn't sufficient, but if you use a more accurate call graph, is that good enough? If you have perfect control flow integrity, otherwise known as code pointer integrity, is that good enough? No one knows, and currently the only way that we can solve that problem is by having four graduate students trying out attacks, trying to create new attacks against these systems. My goal, and I have some NSF money to do this now, is to try to build an infrastructure that given an application and a malicious computation that we might wanna execute can tell us, will this defense allow the malicious code, allow the malicious computation to be executed? Yes or no? And so in this way we can have a much more systematic evaluation of our defenses. Finally, I have a Google Summer of code student working on producing a tighter call graph or coffee. So if you notice, coffee is using one of these coarse grained call graphs. So now that's actually good enough for virtual ghost. It's actually better than what we need for virtual ghost. But if we want to defend operating system kernels from sophisticated memory, sophisticated buffer overflows, then we're gonna want something better than the coarse grained call graph that we're using today. And so I have a student that's working on implementing that for the coffee system. Finally, because Andrew Tannenbaum did it, I thought I would do it as well. Just tell you a little bit about what we have at the University of Rochester. So at the University of Rochester, we have obviously degrees in computer science, so we offer master's degrees in computer science and PhD degrees in computer science. We are very strong in computer architecture and operating systems and compilers. And I am now adding along with another faculty member, security expertise to our faculty. In addition, you might want to know that we have a new master's program in data science. So if you're interested in big data and being able to study big data along with some sort of application area, we now offer a one-year master's program in that. We have a small department with small class sizes. My operating systems course has about 25 students. That's both undergrads and grads. So there's a lot of personalized attention from our faculty. And if you like doing kernel programming, you get to do that in my operating systems course. So something to think about if grad school's been on your mind. So in summary, we built a system called VirtualGhost. VirtualGhost permits applications to protect themselves from a commodity operating system kernel. It uses compiler techniques, namely control flow integrity and software fault isolation. And this keeps higher processor privilege levels free. So we don't need the processor to have VMM extensions. We don't have to use them so they can be used for something else. And it turns out that by using compiler instrumentation, we are faster than current VMM based approaches. With that, I'll take questions. Thank you. Yes? What are your five screen questions? So, one of the reasons why I wanted to present here was to gauge how interested people are in this technology. And one of the reasons for that is because being a small school, I have a small research group. So I have limited developer bandwidth. You tell me? Tell me. Is this something that's interesting? Does it sound too wild and crazy? What do you think? Well, just because it's a lot of crazy doesn't mean it couldn't be. Ha ha ha ha ha. Sorry, I didn't mean to put you on the spot. When I said you, I actually meant all of you. I don't know, but I suspect that it's the sort of thing that some previous people don't know that it might be interested in, not all, but some might, at the very least, making it easy to integrate these kinds of things into a better platform for a reserve, which some of us are excited about. But also the mighty people who are interested in this sort of thing is quite busy. So again, maybe. OK. Question from the net? Yes? How much work do you support other architectures? Support other architectures? Good question. So basically what you would need to do is you would need to port the SVOS runtime library to another architecture. So you need to rewrite that. If you design, if you port it to the virtual instruction set properly, then that's probably about all you need to do. Sadly, that is not what I did in my research prototype, because I was in a hurry and kind of learning the low level parts of FreeBSD as I went. So I probably didn't actually. So basically what it is for the virtual instruction set port, I didn't create a new port, because that was more work. What I did is I took the x86 port, ripped out the x86 port parts, and put my virtual instruction set parts in. But if you do it right, then you shouldn't have to worry about that stuff. And again, it's only about 50. You also have to port the compiler parts. But again, this is only 5,300 lines of code. So it shouldn't be a lot of work. So. And the part of adoption among the DSPs is that one of the barriers to entry is that it's going to take advantage of this template that changes the way it works with programs, through the PID programs. It then makes it harder to maintain parity with other opportunities that you see with that's got some evens overhead. And if you could find a way to solve that, that would probably go a long way with what you think that's going to work more easily. Right. So to answer that, you're correct. One of the interesting things is that, so like for mobile applications, I think we can actually use automated compiler transforms to take existing applications and transform them into versions that protect themselves. For regular Unix applications, it's more difficult because we don't know which files are supposed to be shared and therefore not encrypted so that everyone can see them and which ones are supposed to be encrypted. So it is possible, I think, though, to create an API that the program just compiles to. And then on systems that support virtual ghosts, they do the whole encryption thing. And then on systems that don't use virtual ghost, the encryption is just a null operation. Yes. I think it's cool stuff. I was wondering if you could tell SGX and how it's something like it's kind of attacking the state of all this. It's attacking a similar problem, or rather SGX is starting to attack the same problems that we're attacking. So the first implementation of SGX, SGX1 was primarily designed to take small bits of applications and throw them into what they call an enclave, this isolated environment. And so to make things simple, they are maybe not simple, but to provide the security guarantees they want to provide, they did simplifying things like code and enclave can't execute system calls. So code and enclave, for example, can't receive a signal, can't have a signal handler. It can't do system calls, so it can't do read and write. Whereas in virtual ghosts, they can't. Now my understanding is that the group at Intel is starting to develop a new version of SGX called SGX2, which tries to fix some of these limitations because they're wanting to do what virtual ghosts can do, which is to protect an entire application. I'm not familiar with exactly what they've done and how far they are on that, but I know that that's what they're trying to do. Yeah, so the microsoft is attacking the internet, but they're not. Yeah, so I think you're talking about Haven from OSDI. Yeah, so one thing about Haven that they've done is I think they're trying to protect the application from the hypervisor. And what they've done for the operating system kernel, they've said, okay, we're just going to use a little OS, and so the OS is now part of the application. Oh, that works, but that's not a commodity operating system kernel, right? That's a library operating system. So... Yes? What do you do with the dynamic libraries? Ah, so we have not investigated dynamic libraries yet. I think that it would be simple enough to use cryptographic signatures so that when you ask for a dynamic library to be loaded, you can actually verify that it is the correct library to be loaded, but it's not something that I've actually thought about in detail yet. Yes, ptrace does not work. Any other questions? Is this of any value for my cloud server? Yes. So, so it's... So, so, yeah, so in a cloud server, what you're going to want to do is you're going to want to be able to isolate, first of all, in any server system, right, or any server in which you care about security, you want to isolate applications from each other, and you want to isolate them from the system software, because system software can be buggy, and in particular, the way that we build commodity operating systems, they're very large, they're very complicated, they're very privileged, right? But in addition to that, on a cloud computing system, what you'd want to do is you'd want to protect it from the hypervisor and from other virtual machines, right? Because you don't want other, you know, other people running on the same physical piece of hardware that can access your data or corrupt your data. So, yes, it's very relevant to that area. One interesting piece of future work that we could do is to look at extending the virtual instruction set to support a hypervisor, and that way protect from, you know, basically protect applications, not only from the operating system kernel, but also from the hypervisor and other virtual machines that could be running on the same hardware. So, when do the test program work? What prevents the OSO? Is there a problem with yourself doing something else that's... that you have to destroy with the program? So, in other words, when you execute a program, how do you know it's the right program? Well, okay, so, first off, the question is, is can the operating system kernel corrupt the application code? And the answer is no, because of what I showed earlier, where the application code is encrypted with the virtual ghost public key, so only virtual ghost can decrypt it, so forth and so on. Now, the second question is, let's say that you want to execute program foo. The operating system kernel says, I like program bar better. So, it executes program bar, now you want to talk to say program bar over a pipe. How do you know that it's foo... or you want to talk to foo over a pipe, how do you know it's foo and not bar? Well, that's why I said with the application key pair, that's why it's a public-private key pair. Because if you know the application's public key, or let's say you have a digitally signed certificate, where you know, okay, this is the public key of this application, if the operating system kernel starts up another application and you start talking to it, you can actually authenticate that the application you're talking to is the one that you want. So, if you know foo's public key, then you can... if you're talking to bar, bar won't be able to authenticate itself as foo. So, basically, if I set a certificate in line type SSH key gen, and the code and the kernels are compromised, they will have to somehow check that I'm right, and I want to do it by a time limit. Right. So, now in the case of SSH key gen, what's going to happen is, like, let's say the operating system kernel runs another application. That application's going to have a different public-private key pair, right? Because the operating system kernel doesn't know what your SSH key gen public-private key pair is. So, if you then run your SSH client, it's not going to be able to use the authentication keys created by the operating system's SSH key gen because it's not going to decrypt properly. Right. All right. Now, there are issues with, okay, if I'm the user and I'm actually typing on my keyboard, how do I know that the application I'm talking to is the right application? How does the application know it's talking to me, the user, as opposed to, like, some other program that the operating system is set up to masquerade as the user? That's an open research question, which I have a few ideas for, but that's definitely future work. Yes. You mentioned, like, so the applications aren't specifically key, but there's no way to improve the application because they're just only known by the... I'll hypervisor, but I... Virtual... Virtual... Yeah. When the kernel load starts up in the virtual load by the component that's within that kernel, right? Or it's running on the outside. I mean, is it being indicated at no point in time that the kernel actually knows it's a key? I mean... So, if you have... So, in a complete implementation, including one that can automatically load new kernel code, what you would have is your bootloader. You'd use trusted boot to make sure you have a bootloader, which loads virtualGhost, which loads the operating system kernel, which then can start up applications. Yeah, exactly. But I mean, it could be the case that if the hardware itself sort of would be compromised, it could just start up an operating system that behaves like it's an operating system that runs virtualGhost. In fact, it's just plain 3Bs that happens to know it's a key. If you corrupt the processor, the actual hardware, then, no, we don't defend against that. That's not in our attack model. So, yeah, you're definitely a cook then. So... How would this model protect against things like keyloggers? Keyloggers? Ah... Okay, so I think virtualGhost provides the infrastructure to protect against something like keyloggers, but we don't actually have that implemented because then what you're talking about is, okay, how do you use these features? And that becomes a whole bit more research, right? So one way to potentially do it is to have, essentially, to have a kernel driver that's trusted. So what you do is you create kernel drivers that can have ghost memory. And if you can do that, then an application, when it starts talking to a driver, can actually authenticate it. So basically, you have a few drivers that virtualGhost gives keys saying, okay, this is the driver for the keyboard, this is the driver for the screen. So applications can basically get their public keys from virtualGhost, authenticate them using those public keys, and then talk to them over an encrypted channel. That's one potential way that you could do it. But... That sounds almost like the next question. Um... Yes, but not to the same extreme, because you don't have your file systems and you don't have your networking subsystem built as isolated drivers. And more importantly, what I would like to do, so this is something that I've been thinking about, is I would like to keep these drivers in kernel space, because I don't want to have to run the user space. So, and that's an additional, you know, that's an additional challenge of, you know, can we actually do that? So I think the answer is yes. We'll be off the road on a specific address to pass on. So I know that there's a few steps in the management system, certain steps, you know, on the trust zone, I think that a few steps are attached to the mechanism, and so the question is, if they have hardware support that they can find in, and try to use it as well, do I have to... Yeah, I think it's basically a world-end, a trust-based program. And so the whole premise, if you see the whole premise of this, is that the program is only accepting trust-based inputs, and then this virtual ghost makes the program and trust that, once it has been implemented, that the data is not being modified, and it's coded, but it breaks down some of the trust-based inputs. So the thing is, is that what you... So in order to build a complete system, like an end-to-end system, you have to deal with issues such as, how do you know you're actually talking to the user through all the pipes? How do you... You know, when you rely upon another piece of software, how do you verify that it's doing what you want it to do? And I think that there are solutions to those. You know, proof-caring code might be useful, or just using public-private encryption, where applications are actually authenticating each other, as opposed to just trusting each other like they do today. But yes, it's an open problem. It's not clear exactly how to solve them or what the best way is to solve them. Yes? In your question earlier, there is always considerable interest in the previous D community on the security issue. And then this is certainly one of them. Have you been able to analyze this approach with all the other security measures that are already in the previous D family? The jails, CAPSICOM? They are addressing completely different issues. So what CAPSICOM is doing and what jails are doing, containers and all that stuff, is that they are isolating user-space applications, and they're trusting the operating system kernel to implement this correctly. So... And it's a very valuable thing to do, I should add. I used to build mandatory access controls to do this, you know, probably about 15 or 17 years ago. Okay, that's a long time ago. Anyway, so that's a very good thing to do. However, it's trusting the operating system kernel, and what I found when I worked for Argus Systems Group, which made extensions to Solaris and AIX with these mandatory access controls, is that when attackers couldn't attack applications, they attack the operating system kernel. And the operating system kernel, it violates all the security design that we know we're supposed to be doing. It's large, it's monolithic, it's overly privileged. You know, and so what my research work has focused on is trying to address this challenge, and I think that the commodity operating system kernels are not designed properly. They're susceptible to buffer overflows, they're susceptible to rootkits, so forth and so on, what can we do about it? So basically it's orthogonal problems, is what it is, I would say. So I think that's it. If you have any other questions, please feel free to come by and chat with me. Thank you.
 It's not changing.  Oh, there we go.  Okay. All right.  Good afternoon, everyone.  My name is John Criswell.  I'm an assistant professor at the University of Rochester.  And today I will be talking about protecting free BSD  with the secure virtual architecture.  This is work that I did when I was a PhD student  at the University of Illinois at Urbana-Champaign.  And the work I'm going to be talking about today  was joint work with Nathan Dutton-Hahn  and our advisor, Vikram Advay.  Now, my dissertation work starts with a very simple question.  Do you trust your operating system kernel?  Now, here's the interesting thing.  You say no.  And yes, some people say yes, and some people say no.  But in fact, in practice, we do.  And the reason why is because on top of our commodity  operating system kernels, such as the free BSD kernel,  we run applications.  And we use these applications to process sensitive information.  So we may buy things online using our credit cards  from websites like eBay or Amazon.com.  We may file our taxes, which includes sensitive information,  such as if you're an American citizen, social security  number, and your address.  Medical data.  My doctor happens to store my medical information  on a Windows machine.  And certain voting machines run commodity operating system  kernels or operating system kernels based on them,  such as Windows CE.  Now, we run these applications on our commodity operating  system kernels.  But maybe this is not such a good idea.  And the reason why is because commodity operating system  kernels are vulnerable.  Now, they're vulnerable for two reasons.  First, all commodity operating system kernels today  are written using C and C++.  And as a result, they suffer from the same security  vulnerabilities that applications suffer  when there are coding mistakes that are made.  So for example, commodity operating system kernels  have been vulnerable to buffer overflows  and integer overflow attacks.  Additionally, there are vulnerabilities  that occur in operating system kernels  due to their job as the operating system kernel.  So for example, the operating system kernel  provides the process abstraction,  which is supposed to isolate different processes, different  programs from each other.  But sometimes logical bugs in the operating system kernel  allow information to leak from one process to another  unintentionally.  So for example, through misconfiguration of the MMU,  one process may be able to access the memory  of another process.  Finally, all commodity operating system kernels to date  are dynamically extendible.  You can load new drivers, load new kernel modules  into the system as it is running.  Now because of how the operating system kernel is structured,  these new modules, when they're loaded,  can modify the operating system kernel behavior  in arbitrary ways.  And so attackers have taken advantage of this.  They've written kernel level malware, typically  to hide their presence, such as hiding processes and files  and network connections.  But because they can use any arbitrary behavior that they  want, they can do things like stealing data from applications,  corrupting data within applications,  even modifying application control flow.  Now here's the real kicker.  If the operating system kernel is exploited,  then all security guarantees on your system are null and void.  If your operating system kernel is compromised,  you have no security on your system.  And the reason why is because nearly all security policies  are either enforced by the operating system kernel itself.  So if the attacker controls the operating system kernel,  he or she can just turn the security policy off.  Or they are enforced by applications running on top  of that operating system kernel.  And so because monolithic operating system kernels  have such a great amount of control over the system,  they can just reach into the application,  change its code, change its data, and therefore turn off  the security policy enforcement within the application.  So if the operating system goes, everything goes with it.  Now there are two approaches to addressing this problem.  If you don't want to abandon existing commodity operating  system kernel.  So if you don't want to rewrite it,  if you don't want to do massive restructuring  and refactoring of the kernel.  The first is to automatically harden the operating system  kernel from certain classes of attack.  And I've looked at this in my PhD dissertation  and built systems such as SVAM, which  enforces strong memory safety guarantees  on commodity operating system kernels, namely Linux.  And then more recently, the coffee system,  which enforces a lighter weight security policy called  control flow integrity on the free BSD kernel.  Now, while these approaches are good,  one limitation that they have is they only address  a particular class of attacks.  So the SVAM and coffee systems, for example,  address buffer overflow attacks and attacks of that nature.  But they don't address things like kernel level malware that's  loaded into the operating system kernel.  They don't handle information leaks.  They don't handle missing access control checks.  So another approach that I explored in my dissertation  is to just assume that the operating system kernel  becomes compromised.  Just raise up your hands and say, OK,  we're just going to assume that the operating system kernel  can be compromised.  Can we run applications securely on that kernel anyway?  And so we built a system called VirtualGhost,  which provides data confidentiality and integrity  for applications running on a potentially compromised  operating system kernel.  Now, today in this talk, I'm going to talk about a little bit  about coffee and VirtualGhost.  Most of the talk will be about VirtualGhost,  because in fact, VirtualGhost is actually  built on top of the coffee system.  So we have control flow integrity for the free BSD kernel,  and then we're going to build VirtualGhost on top of that.  And so the contributions of VirtualGhost  is that it protects application data confidentiality  and integrity, as well as protecting other features  of the application.  It uses compiler techniques.  And because it uses compiler techniques,  we can use the same processor privilege level  as the operating system kernel.  We do not need to use a hypervisor based approaches,  if you will, where we kick the operating system kernel up  into ring one on x86, or have VMM extensions that allow us  to run code below the operating system kernel.  We can actually run alongside the operating system kernel.  And it turns out that although we're using compiler  instrumentation techniques to add instructions  to the operating system kernel code,  we are still faster than hypervisor based approaches.  So that's a brief overview of the research work.  I'll now talk about the design of VirtualGhost,  and then take a small aside and talk about some  of the design and implementation of coffee.  I'll then talk about our experimental results.  So a little bit about the implementation  and the experiments that we run on coffee and VirtualGhost.  And then finally, I'll conclude with some future work  that we're doing at the University of Rochester.  All right, what is the fundamental problem  with current system design?  The fundamental problem is that applications cannot protect  themselves from the operating system kernel.  So if I want to write an application that does not  trust the operating system kernel,  I will be inclined to write an application that just keeps  its data encrypted as long as possible.  So encrypt data, when it sends it to the file system  or to the network, it may even keep data encrypted  while it's stored in memory.  The problem is that even if the application does this,  it cannot protect itself from the operating system kernel.  The reason why is the operating system kernel can access  anything and everything on the system.  So the operating system kernel can just reach in  and read unencrypted data out of the application  or modify the encryption keys to some value  that the operating system likes.  The operating system kernel can modify the applications code  so that it just doesn't encrypt anything at all.  It can even modify the application's control flow,  it can stop the application, change its program counter,  and then resume the application,  causing it to skip over encryption  and decryption operations.  So no matter how hard an application tries  on an existing operating system kernel,  such as FreeBSD or Linux or Mac OS X,  it just simply cannot protect itself.  The goal of VirtualGhost is to build a system  that allows applications to protect themselves.  And there are three features that such a system requires.  First, applications require private data and private code.  So they need, they're still gonna need public memory  that the operating system kernel can read and write  so that they can communicate with the operating system kernel,  but they're also going to need memory  that the operating system kernel cannot read  and cannot corrupt.  Secondly, they need incorruptible control flows.  They need to know that when they start execution,  they start execution in their main function.  And they also know that if they're interrupted  by a trap or a interrupt or they execute a system call,  they have to know that the operating system kernel  is not gonna be able to modify their control flow maliciously  while the operating system kernel is running.  Third and finally, applications that protect themselves  are going to need a reliable way  of getting their encryption keys from some file,  typically the executable image,  from the executable image into that private data memory region  without the operating system kernel being involved.  Because if the operating system kernel  is involved in that process,  it can change the encryption keys,  it can read the encryption key values,  thereby defeating the purpose of having the application  encrypt its data when it sends it  to the operating system to begin with.  So these are the three things that we need.  How hard could it be?  Well, there's two challenges.  The first challenge, the more obvious challenge I think,  is that modern processor design assumes  that system software like the operating system kernel  should be able to access all of memory.  So that's challenge number one.  But there's actually a more subtle challenge  when you start trying to do this.  The more subtle challenge is that  if we want operating system kernels to provide the features  that we expect them to have,  they must be able to manipulate application state.  It doesn't suffice to put an application over here  and the operating system kernel over here  and say, never the two shall meet.  Because if you did that,  the operating system kernel would not be able  to create new processes and new threads.  It would not be able to execute new programs  by providing the exact family system calls.  It would not be able to provide signal handler dispatch  because all of these operations modify application state.  They go in and make changes  to the application's program counter.  So instead of preventing the operating system kernel  from manipulating application state,  we must allow it to do so,  but we must control what it does.  We must ensure that it makes good state modifications,  but not bad state modifications.  All right, so we're gonna need some infrastructure  to do this.  We're going to use from our previous research  of the secure virtual architecture.  So in the secure virtual architecture,  instead of compiling the operating system kernel down  to native code, and instead of having inline assembly code  written into the operating system kernel,  instead of what we're going to do  is we're going to compile the operating system kernel  to a virtual instruction set.  And we're going to design this virtual instruction set  to be easy to analyze an instrument.  Now, handwritten assembly code is not easy  to analyze an instrument.  So what we're going to do is we're going to port  the operating system kernel to our virtual instruction set.  We're going to have a set of instructions  in that virtual instruction set called SVOS,  which we can use to replace inline assembly code.  The SVOS instructions basically provide features  such as registering, interrupt, and trap handlers,  being able to configure the MNU,  being able to manipulate interrupted application  or interrupted program state.  In this way, we could represent  an entire operating system kernel  in the virtual instruction set.  The operating system kernel will have no inline assembly code,  no handwritten assembly code in it at all.  Now, you can't actually run a virtual instruction set  operating system kernel.  You have to translate it to native code,  like x86 or ARM or PowerPC or MIPS,  in order to run it on a real processor.  Now, when I give this talk,  some people tend to assume that, okay,  you're kind of doing something like Java.  So you have a virtual instruction set,  you have a native instruction set.  So the translation must happen just in time,  just like Java does.  No.  Secure Virtual Architecture is designed  so that translation can happen anytime you want.  It can happen ahead of time,  it can happen at system boot time,  system install time, run time, idle time.  In our implementation, in our prototype,  we implement translation ahead of time,  because A, it's easier to do,  and B, it's more efficient.  Now, let's look at the virtual instruction set  in a little bit more detail.  It's comprised of two components.  The first component is SVA Core.  SVA Core is taken from the LVM Intermediate Representation.  So it's the language that the LVM compiler uses  to analyze and optimize code.  It has source level types,  it has explicit static single assignment form.  These are things that allow us to do sophisticated  compiler analysis and instrumentation  on the operating system kernel code.  Now, the SVA Core instruction set  is what you would call regular computation.  So it provides things like adding and subtracting,  pointer arithmetic, reading from and writing to memory,  those sorts of things.  Now, the LVM IR, if you take away  the inline assembly code feature,  can't support an operating system kernel.  You can't express the FreeBSD kernel  or the Linux kernel in LVM IR.  So we extended it with a new set of instructions called SVOS.  These are operating system neutral instructions.  We've used them with both Linux and FreeBSD.  And they encapsulate the state manipulation  and hardware configuration operations.  So again, they provide instructions  like configuring page table pages,  modifying interruptive program state  for signal handler dispatch,  registering system call handlers and trap handlers  and things like that.  Now, what's interesting about that,  not only do they encapsulate these operations,  but because the operating system kernel  has to use these instructions to interface with the hardware  and to manipulate state at all,  it allows us to control  how the operating system kernel does these things.  So we can control how the operating system kernel  configures the MNU.  We can control how it manipulates application state.  And so what we do is when we implement  these instructions, these SVOS instructions,  we add runtime checks to them  to help enforce the security policy  that we're also enforcing with the compiler instrumentation.  So one implementation of the SVOS instructions  helps enforce control flow integrity for the coffee system.  Another implementation implements the runtime checks  that we need for virtual ghosts  to protect applications from the operating system kernel.  Now implementation wise,  what we do is we implement the SVOS instructions  as a native code runtime library.  So essentially what happens is you express  your operating system kernel in the virtual instruction set.  You then analyze an instrumentate,  convert it down to native code.  The implementation of the SVOS instructions  is missing from that native code.  So the runtime libraries linked in  provides the implementation of those instructions.  You now have a complete native code kernel  that you can boot on real hardware.  All right, now how are we going to use  secure virtual architecture to implement virtual ghosts?  How are we going to provide those three features  that we need?  Well let's first look at private data and private code.  So most operating system kernels such as FreeBSD  divide the virtual address space into two partitions,  the user space partition and the kernel partition.  So user space memory is where the application lives,  kernel space memory is where the kernel lives,  and the kernel is allowed to access  both user space and kernel space,  whereas applications can only access user space.  Virtual ghost is going to add two new partitions  to the virtual address space.  The first one is ghost memory.  Ghost memory is memory that the application  is allowed to read and write,  but the operating system kernel is not allowed  to read and write.  So this is where the application is going to put  its private data and its private code.  Now what's going to happen is that implementation  of the SVOS instructions, that runtime library,  it has its own data structures that it uses  in implementing its runtime checks.  Those data structures should not be accessible  to applications or the operating system kernel.  So there's another region of memory called  the virtual ghost VM memory or VM memory for short.  This is where the SVOS data structures go.  This region is not readable or writeable by applications  or the operating system kernel.  So essentially we provide this new,  the two new regions, contiguously in memory,  that the operating system kernel is not allowed  to read or write.  Now by this time you're probably asking me,  John, how do you keep an operating system kernel  from writing into ghost memory and VM memory?  The trick to the secret sauce is  solve for fault isolation instrumentation.  So what we do when we're translating  from virtual instructions set code to native code,  is we look for all the load and store instructions  in the operating system kernel  and we add some instructions before them.  What these instructions do is they check to see  whether the pointer that's gonna be used  in the loader store is pointing into user space memory  or kernel memory.  If they are, it's fine.  The pointer does not need to be changed.  If they are erroneously,  if the pointer is erroneously pointing into ghost memory  or into the virtual ghost VM memory,  then a simple bit masking operation  moves the pointer into kernel memory.  In this way, we are guaranteed  that all loads and stores are either accessing  user space memory or kernel memory,  but not ghost memory and not VM memory.  Now, let's say your operating system kernel  unfortunately has a buffer overflow in it.  An attacker could potentially use that buffer overflow  to change the control flow to jump over  these new instructions that we've added.  And that would allow the operating system kernel  to access ghost memory or VM memory.  And that would be bad.  That would violate the virtual ghost security properties  that we're trying to enforce.  So what we do is we use control flow integrity.  By using control flow integrity instrumentation  along with software fault isolation instrumentation,  we can ensure that the software fault isolation  instructions are always executed.  They can never be jumped over  even if your kernel has a buffer overflow  or other sort of memory safety error.  In addition, the control flow integrity instrumentation  helps protect the operating system kernel  from buffer overflows and other related attacks.  So we sort of get a two for one deal with virtual ghost  in that we can actually protect the operating system kernel  from attack and protect applications  from the operating system kernel  with one set of instrumentation.  All right, so now we have our private data  and our private code.  What about secure application control flow?  Well, why isn't application control flow secure today?  The reason why is because on an interrupt trap  or system call, the hardware transfers control flow  to the operating system kernel  and the operating system kernel saves  the interrupted program state on the kernel stack.  The kernel stack is in kernel memory, it's in kernel space,  the operating system kernel can just read and write  that as it likes.  So in virtual ghost, what we do is  when there's an interrupt trap or system call,  the hardware transfers control flow  to the virtual ghost runtime library,  that implementation of the SVOS instructions.  That runtime library then saves the interrupted program state  not into kernel memory,  but into the virtual ghost VM memory.  Then virtual ghost transfers control flow  to the operating system kernel.  So now the operating system kernel can respond  to the interrupt or the trap  or the system call as appropriate.  But now when the operating system kernel wants  to modify application state, it can't do so directly.  It's sitting in that virtual ghost VM memory  where the operating system kernel can't touch it.  So the operating system kernel wants to make a change  to say the program counter or the stack pointer,  it has to ask virtual ghost to do it  through an SVOS instruction.  So in this way, the SVOS instructions can vet changes  to save program state.  And if the changes are okay,  then go ahead and make those changes on behalf  of the operating system kernel.  So in this way, we can actually control operations  such as signal handler dispatch, thread and process creation  and the exec family of system calls.  As an example, let's look at how the exec system call  is implemented on a virtual system.  So you have the operating system kernel,  there's some application executable  that it wants to execute.  There's some program that has executed  the exec system call, so it's been interrupted  and we are now running in the kernel code.  So the kernel says, hey, virtual ghost,  I have this application executable here,  please set up the code segment for this executable  and please change the save program state  so that when I resume it, when I put it back onto the CPU,  it will start up in the main function of this executable.  So what virtual ghost does, it says, okay,  I will go and I will set up the application code segment  in the applications ghost memory.  So now we have the applications code segment,  it's in ghost memory, the operating system kernel  cannot arbitrarily modify the applications code,  but the application can use it.  And then virtual ghost will locate the virtual address  of the main function and change the program counter  to point to it.  And then it returns back to the operating system kernel.  So when the operating system kernel  does the return from system call,  what will happen is the save program state  will start executing this new application code  in its main function.  So what we've done is we've essentially taken the operating  system out of the path, the critical path,  where we rely on it to do the correct thing.  Instead, virtual ghost does the critical operations  of setting up the code segment  and changing the program counter  to point to the main function.  All right, so now we have private data and private code,  we have secure application control flow.  The last thing we need is secure application encryption keys.  So let's say you're an application developer,  you've written an application,  you generate a public-private key pair  for that installation of the application,  and then you're gonna run this on a virtual ghost system.  So if you just embed the application code  and the application key into the executable file,  and then you send that over to a virtual ghost system,  your operating system kernel can do one of two nasty things.  The first thing it can do, it can simply say,  well, I like the code, but I don't like that encryption key  because I don't know what it is.  So what I'm going to do is I'm simply going to replace it  with my own public-private key pair.  So when this application goes and encrypts data,  I know what key it's using.  Alternatively, the operating system kernel may say,  well, I like that application public-private key pair,  this application came with some files  that are encrypted using those keys,  but I wanna be able to see what's in those files.  So I'm going to change the code in the application,  so that it'll decrypt the data with that application key pair,  and then I can see it.  What we need to do is we need to tie the application code  and the application key pair together,  so that if they are tampered with  by the operating system kernel,  we can detect the tampering and refuse to run the program.  So every installation of virtual ghost  will have a public and private key.  The virtual ghost public key will be used  to encrypt the application code, the application key,  and then a checksum of the combined application code  and application key pair  will be encrypted with a virtual ghost public key.  What this allows us to do  is when the operating system kernel comes along  and says, hey, virtual ghost,  please set up the code segment for this application,  virtual ghost can verify that the code has not been modified  and the key has not been modified,  and then in fact, this code and this key actually go together.  It's the right code and the right key paired together.  If it hasn't been tampered,  then what virtual ghost will do  is it'll set up the code segment  as we talked about a few slides ago,  and then it'll decrypt the application key pair  and put that into ghost memory.  In this way, the operating system kernel  is never involved in getting that application key pair  out of the executable and into the process as ghost memory.  It's not involved in that process,  and so as a result, it cannot corrupt that process.  Yes?  If we are just sending encrypted to virtual ghost,  the application code is to be paired with a hash,  but if the hash is assigned with the same key pair,  then can we mix and match to the text less,  send data and then hold back and send  previous code or something?  It does the code that's sent,  and then the second message,  we have some other key pair that's sent both  or encrypted to virtual ghost,  but then when we receive some other code  and key pair later,  we could send it or do out of order.  I'm not sure if I quite understood.  Are you essentially saying if I'm an application developer  and I create one version of my application and I ship it,  is it possible that I can send  another version of the application?  Yeah, I can send, I can say,  all right, let's use the old version of the application,  which I have to do, I guess some of the apps  will have to do that.  Right, right, okay.  So I think my answer to that is that  if the system that the developer is using  is running on virtual ghost,  or if it's a system outside,  so it's like an app store from Apple or something,  then that should not be a problem,  because the operating system kernel can't get in  and modify that.  It shouldn't be able to corrupt the process  and create the second version of the application, correct?  So although I think there is an attack though,  where the operating system kernel also actually,  so I think what you'd probably want to do  is I think what you'd want to do is  every time you update an application,  I think you would actually have to give it  a new application key pair,  and then re-encrypt all of the files  with that application key pair.  So yeah, I had not thought of that, thank you.  Any other questions?  Yes?  So I have a program, it's in my brain.  I am running low on available memory,  I want to swap it out.  So this segment of code comes from this user space VM,  goes to disk.  Now this program is executing again,  I wish to put it back in place.  At this point, the kernel will own the buffer  that has just been taken from disk,  which has the application code on it.  I modify that code to be something that reads  from the secret memory, which I can do,  because that will be executed from the context  of the other process.  I then swap that back in, allow it to continue executing  without changing its construction point or whatever,  but then I still have arbitrary code execution  from the context of the process,  I use that to scale the keys and I go create it.  I don't see any way to protect against that  without having the entire everything,  it doesn't work.  So let me rephrase your question,  you're asking how to secure swapping work, right?  So as it stands right now,  you can't swap out anything in ghost memory, right?  The operating system cannot swap out anything  in ghost memory, because it can't read it  and it can't write to it, right?  So first you said, okay, let's say you swap out  the code segment or the application key pair, right?  Well the operating system can't do that  because it's in ghost memory.  Now in the paper we talk about a design  for allowing secure swapping,  so if the operating system kernel wants to swap out memory  that happens to be mapped into ghost memory,  it can do so and what virtual ghost will essentially do  is essentially encrypt and digitally sign the contents  and then allow the operating system kernel  to have access to that.  We haven't implemented that because we haven't run into a  case where we need to actually swap ghost memory  and things like that out.  I don't need to swap ghost memory,  I can swap any memory that's in that process.  So,  executable memory should be in ghost memory,  because if it's not, yes.  Because otherwise the operating system kernel  can corrupt it.  Okay, yep.  Yes.  Do you account for modifications  that might happen by directly accessing?  Yes, so we haven't implemented it,  but that's easily solvable through IO MMUs.  So you just get, so you have virtual ghost control access  to the IO MMU and the operating system kernel  cannot reconfigure the IO MMU,  so the virtual ghost runtime,  the implementation of the SVS instructions,  basically configures the IO MMU  so that the operating system kernel  can't use DMA to get information  into and out of physical memory that's mapped  into the ghost memory region.  What I don't really understand about this,  but you can sort of,  we still affect the  behavioral system called the process  and both names and also modified it,  trap frame.  You can't modify the trap frame.  Because, so the trap frame,  we call it interrupted program state.  That's saved into the VM memory,  which the operating system kernel cannot access.  So, or at least it can't access it  without going through virtual ghost and virtual ghost.  The interface of virtual ghost combined  with this runtime checks ensure that the changes  are not going to violate the application's  control flow integrity.  So what if I would just,  you know, assume that the program  is both the MNAP system call,  I just give it back a buffer that's not within the ghost.  Excellent, excellent.  So you have just reinvented the IAGO attacks.  So there's an attack called the IAGO attack  in which since the operating system kernel  can't access the application memory anymore  and it can't corrupt the control flow,  it just says I'm going to return bogus values  from system calls.  And when I do that,  maybe I can trick the application  into doing something it doesn't want to do.  So maybe I return,  MNAP returns a pointer into ghost memory  and the application just assumes that's okay.  So in future work,  I'm actually having a student work on that.  So I will get to that in basically my last slide.  But yes.  Yes, just out of curiosity,  you said that you created the protected ghost message  by instrumenting at compile time  all the old memory attack.  Yes.  But are you even able to catch the kernel  trying to patch itself to create a new memory attack  to the front end?  That's because we don't allow the kernel code segment  to be writable, which I'll get to in a few slides.  Okay.  But the second thing is the price of the bit,  so if you want to use it, okay.  Yeah, so in our useNIC security 2009 paper,  we have a solution for the limited amount of patching  that Linux actually tries to do.  So, but in general,  the operating system kernel is not allowed  to make arbitrary changes to its native code segment.  Yes.  And that has to hold your previous line as well.  I wouldn't be able to play that protective  of the previously thought-brained thing  for product that.  Actually, so I haven't really thought about that.  I think user space is actually fine,  and we'll get to the reason why in a minute.  All right.  All right, I'm gonna move on,  and we're gonna go to coffee break.  All right, so coffee is essentially a subset of virtual ghost.  So coffee divides the virtual adder space into three regions.  It's got the user space region, the kernel region,  and then the coffee VM memory region.  So this is the virtual ghost.  VM memory just renamed as coffee VM memory.  Coffee provides control flow integrity  so that we know where we're jumping to  in the kernel code segment.  It gives us code segmentary,  so the operating system kernel code does not change.  It gives us software fall isolation on stores.  So, unlike virtual ghost,  which has to maintain the confidentiality  and integrity of VM memory,  coffee only needs to protect the integrity of its VM memory.  So it only has to do software fall isolation  and documentation on stores.  The SVOS checks that its SVOS runtime library does,  only do control flow integrity and code segment integrity,  and it doesn't have the encryption key delivery feature  or other virtual ghost specific features.  So basically, coffee is what gives us  the control flow integrity and code segment integrity  that virtual ghost requires,  and then virtual ghost builds on top of that  the features to protect applications  from the operating system kernel.  How does coffee work?  Well, it could enforce the control flow integrity  in multiple ways.  In our prototype implementation,  what we do is we use the approach from Zangton and Moorset  in their CCS 2011 paper.  So what we do is when we translate to native code  at the beginning of every function,  and after every call instruction,  we insert a special no op instruction,  a no op instruction that does not appear anywhere else  in the kernel code segment.  Then we instrument all computed jumps,  so every return, every indirect function call,  to first take the address that the kernel wants to jump to,  and bit mask it so that it's not pointing  into user space memory.  So someone asked me, well,  what about instructions in user space memory?  Well, instructions in user space memory,  their kernel code, the virtual ghost has to set up  the code segment so that it's,  I think virtual ghost has to set up the code segment  so that it's executable.  So I think virtual ghost still has to set up  the code segment, but we allow arbitrary native code  to be run in application space  because we control what the kernel executes.  So as long as the kernel is not tricked into running  user space code, which is what the bit masking does,  we're fine.  So we bit mask the pointer, the address,  to make sure that it's in the kernel code segment,  and then what we do is we check to make sure  that where we're jumping to,  actually contains one of these no op labels.  If it does, great, if not,  then we use a direct branch to jump  to some error handling code.  Now for returns from interrupts, traps, and system calls,  so in modern operating system kernels,  the operating system kernel can actually interrupt itself,  so the operating system kernel can experience a trap  or an interrupt while it's running.  So just like application state,  we save that state into the VM memory  where the operating system kernel  cannot directly modify it.  So in this case, we prevent accidental overriding  through buffer overflows of changing the program counter  of the operating system kernel once interrupted,  or changing the privilege level  of the operating system kernel once interrupted.  We also have instructions that allow us  to do the exception unwinding that's done  on efficient implementations of copy in and copy out.  So we can still use the MMU to catch faults  in copy in and copy out operations,  and securely unwind the control flow  while being control flow integrity.  Now as I said before,  we also provide code segment integrity,  so by controlling the access to the page table pages,  and by controlling the MMU,  both coffee and virtual ghost ensure  that the code segment is never made writable,  there are never any changes that map new data  or new code into the code segment.  There is an instruction that allows you  to dynamically extend the code segment  of the operating system kernel.  So if you want to load a say a device driver,  you can give virtual ghost or coffee  the virtual instruction set code,  it'll translate that down to native code,  do the instrumentation,  and then add that to the kernel's code segment.  But otherwise the operating system kernel  is not allowed to make arbitrary changes  to its native code segment.  All right, back to virtual ghost.  All right, so now I'm moving on to the results.  So we implemented a prototype of virtual ghost  and coffee for 64-bit x86.  We ported FreeBSD9 to virtual ghost,  if you're wondering why it's so old,  it's because we started this, I think back in 2012.  So the trusted computing base is about 5,300  source lines of code,  this is the size of the compiler passes  that we wrote and the runtime library that we implemented.  And then we modified several applications  from the open SSH application suite to use ghost memory.  So the SSH client, the SSH key generating program,  the SSH add utility, use ghost memory for their heap.  They really should be using it for globals and stack,  but in our prototype implementation,  we're only doing the heap.  And basically what they do is they create authentication keys  that are encrypted so the operating system kernel  cannot read them, cannot access them,  but the SSH client and the SSH add utility can use them.  All right, this is now released as open source software.  We released the LVM compiler extensions,  the virtual ghost runtime library,  and a patch to the FreeBSD9 kernel code,  which gives you the port of FreeBSD9 from x86  to the secure virtual architecture virtual instruction set.  This is available on my GitHub account,  github.com, slash jtcriswell with two Ls.  There's also a link to this from my homepage  at the University of Rochester.  All right, now the first experiment that we ran  was we wanted to see whether we could stop  a sophisticated malware attack,  and specifically one that was designed  with virtual ghost in mind.  So what we did is we wrote a malicious kernel driver.  And what this malicious kernel driver does  is it tries to set up a false signal handler  within the application that will try to copy data  between ghost memory to traditional memory.  If it can do that, then it can just read the malicious driver,  it can just read the data out of traditional memory.  So if it can get a mem copy instruction  to act as a signal handler,  then it can steal data out of ghost memory  by tricking the application into copying it  into traditional memory.  Now this is way more sophisticated than what you need  for standard FreeBSD,  and standard FreeBSD there is no ghost memory,  so you can just read data straight out of memory.  So nevertheless, this works on native FreeBSD.  It doesn't work on virtual ghost,  and the reason why it doesn't work on virtual ghost  is because virtual ghost is protecting  the application saved program state.  When the application starts running,  it tells virtual ghost through a system call  that goes straight into the virtual ghost runtime library,  here is where all of my signal handlers are at.  So when the malicious driver says,  hey, virtual ghost, please change the program counter  to point to this mem copy instruction,  virtual ghost says, that's funny.  The application didn't say that that mem copy instruction  was a signal handler, so no,  I'm not going to change the saved program state at all.  I'm going to leave it completely unchanged.  So operating system kernel,  if you continue to run this application,  it's gonna continue running right where it left off  before the interrupt trap or system call.  So that's how we stop this rather sophisticated  piece of kernel malware.  All right, how does virtual ghost perform  in terms of execution time?  So we wanted to compare virtual ghost to other approaches.  Other approaches try to magically encrypt application pages  when the operating system kernel tries to access them,  the most recent of which is a system called intag,  it uses VMM extensions in the processor.  We compared our LM benchmark results to intag,  and what we found is that we do quite a bit better  than intag, so virtual ghost is usually about 4x  to 5x overhead on system call latency,  normalized to native, whereas intag is more like  7x to 9x.  Also, virtual ghost has very low overhead  on some key benchmarks, so for example on page faults,  virtual ghost only adds 15% overhead,  whereas on intag, you have 7.5x overhead.  Now this is not an apples to apples comparison  because intag uses Linux, we use FreeBSD,  we're using different machines,  we're not even using the same version  of the LM benchmark suite, but it gives us a ballpark figure,  and the ballpark figure says that virtual ghost  is doing pretty well.  Now, what about comparing virtual ghost to coffee?  So what we see is this is our LM benchmark results  comparing coffee to virtual ghost.  Because of the additional software fall isolation  instrumentation on loads, virtual ghost does incur  a fair amount more overhead than coffee does.  So one thing that this tells us is that the software fall  isolation instrumentation actually does matter,  it is actually hurting performance.  So if we could get rid of it, that would be nice.  All right, now that's micro benchmarks,  that's latency of system calls.  Most applications do not spend most of their time  executing system calls, they spend most of their time  doing computation.  So we wanted to see what is the effect of performance  on actual applications.  So what we did is we took two network servers,  THTPD and SSHD, and we ran performance experiments on them.  We ran our experiments on an isolated one gigabit  per second network, and the reason why we chose  network servers is because unlike other standard benchmarks  like the spec benchmark, they spend a significant amount  of their time in kernel space.  They spend a lot of time actually using kernel services,  and therefore the overheads that we're adding  to the kernel are more likely to show up.  So our first experiment, we took THTPD,  we used Apache Bench to transfer files  between one kilobyte and one megabyte in size,  configured Apache Bench to use 100 clients,  100 clients app operating parallel,  doing 100,000 requests, and what we see  is the performance overhead is negligible.  Now I haven't shown the coffee numbers,  mainly because we configured Apache Bench  a little bit differently, we used I think 32 clients  instead of 100,000, but the results are essentially  the same, negligible overhead for THTPD.  All right, that's multiple clients.  What happens if what you want to do  is you just want to transfer one file,  transfer it over to the network as quickly as possible,  and using cryptic connections so that no one changes  your file or sneaks up on the contents of your file  during the transfer.  So what we did is we took an unmodified SSH server,  so this is not using ghost memory,  this is an unmodified SSH server running on a native  free BSD system, the coffee system,  and the virtual ghost system.  We used an SCP client on another machine  to transfer files between one kilobyte  and one gigabyte in size, and we measured the bandwidth  that we get through the verbose mode on SCP.  What we find is that coffee incurs a 27% reduction  in bandwidth in the worst case, whereas virtual ghost  reduces the bandwidth by 45% in the worst case.  So the overheads are not completely terrible,  but obviously there's room for improvement.  All right, now what happens if you use ghost memory?  So far these experiments have basically been showing  what happens to existing applications  if you put them on a coffee system or a virtual ghost system.  What happens if you start using ghost memory?  Is there any cost to that?  So we took our SSH client, which is using a wrapper library  that copies data between traditional memory  and ghost memory when it wants to do read and write  system calls.  So we took this ghosting SSH client  and we ran it on a virtual ghost system  and we used it to transfer files from another system  between one kilobyte and one gigabyte in size,  and then we also tested the original SSH client.  So this is original SSH and ghost memory SSH,  both running on the virtual ghost system.  What we find is that there's a 5% reduction in the worst case,  typically for these larger file sizes.  Why is it for these larger file sizes?  Well, we suspect that overhead is coming from the fact  that we're copying data between traditional memory  and ghost memory when we're doing reads and writes.  Now here's the good news.  The good news is that a lot of this copying is unnecessary  because SSH is encrypting and decrypting data  as it sends it to and receives it from the network.  Encryption and decryption has an implicit copy operation.  So what this SSH client is doing right now  is when it receives encrypted data, it takes the data,  the operating system kernel puts an intraditional memory,  SSH copies it into ghost memory  because that's what the wrapper library  for the read system call does.  Then it takes that encrypted data and ghost memory,  decrypts it, makes another copy in ghost memory.  So if we hand-tuned SSH, what we could do  is the encrypted data comes into traditional memory,  then it's decrypted and copied into ghost memory  in one operation.  So instead of having these two copies,  we only have one copy.  So in a nutshell, while 5% reduction isn't bad,  we think we can do better.  All right, future work.  So one of the things that we're gonna have a student  start working on soon is replacing  and reducing the compiler instrumentation.  So as you've seen, the compiler instrumentation,  while it is better than using the VMM extensions  in the way that InkTag did,  it still has overhead that's not negligible,  at least on the Allen Benchmark Suite  and on applications like SSHD.  We think we can replace the software fall isolation  instrumentation using address-based identifiers  or ARM domains, or perhaps only making small modifications  to the MMU of a processor to provide the isolation features  that we need.  If we can remove the software fall isolation instrumentation,  we should be able to significantly improve the performance  of both coffee and virtual ghost.  A second thing that we're working on  is we're working on defenses against Yago attacks.  So someone here, I believe, mentioned the Yago attacks.  So Yago attacks, again, are attacks  in which the operating system kernel returns bogus values  to an application through the system call interface  to try to trick it into doing something  that it doesn't want to do.  Our observation is that this is essentially  an application trusting low integrity data.  So by using standard programming language  information flow techniques,  we can basically check whether an application  is doing computation on high integrity data  and whether that computation is being influenced  by low integrity data from the operating system kernel.  So in this way, we should be able to build a system  that formally verifies that an application  is not vulnerable to these Yago attacks.  We're also looking at the automated,  we're also building a system that will automatically  determine the efficacy of control flow integrity  and code pointer integrity.  So if you're following the control flow integrity literature,  coarse grain control flow integrity,  now there are now, sorry, there are now new attacks  against coarse grain control flow integrity  that allow attackers to compute,  to perform malicious computation.  So we now have this open question of,  how good of control flow integrity do you need?  So obviously, coarse grain control flow integrity  where you don't distinguish between different  called targets or different return targets  isn't sufficient, but if you use a more accurate call graph,  is that good enough?  If you have perfect control flow integrity,  otherwise known as code pointer integrity,  is that good enough?  No one knows, and currently the only way  that we can solve that problem is by having  four graduate students trying out attacks,  trying to create new attacks against these systems.  My goal, and I have some NSF money to do this now,  is to try to build an infrastructure  that given an application and a malicious computation  that we might wanna execute can tell us,  will this defense allow the malicious code,  allow the malicious computation to be executed?  Yes or no?  And so in this way we can have a much more  systematic evaluation of our defenses.  Finally, I have a Google Summer of code student  working on producing a tighter call graph or coffee.  So if you notice, coffee is using one  of these coarse grained call graphs.  So now that's actually good enough for virtual ghost.  It's actually better than what we need for virtual ghost.  But if we want to defend operating system kernels  from sophisticated memory, sophisticated buffer overflows,  then we're gonna want something better  than the coarse grained call graph that we're using today.  And so I have a student that's working on  implementing that for the coffee system.  Finally, because Andrew Tannenbaum did it,  I thought I would do it as well.  Just tell you a little bit about what we have  at the University of Rochester.  So at the University of Rochester,  we have obviously degrees in computer science,  so we offer master's degrees in computer science  and PhD degrees in computer science.  We are very strong in computer architecture  and operating systems and compilers.  And I am now adding along with another faculty member,  security expertise to our faculty.  In addition, you might want to know  that we have a new master's program in data science.  So if you're interested in big data  and being able to study big data  along with some sort of application area,  we now offer a one-year master's program in that.  We have a small department with small class sizes.  My operating systems course has about 25 students.  That's both undergrads and grads.  So there's a lot of personalized attention from our faculty.  And if you like doing kernel programming,  you get to do that in my operating systems course.  So something to think about  if grad school's been on your mind.  So in summary, we built a system called VirtualGhost.  VirtualGhost permits applications to protect themselves  from a commodity operating system kernel.  It uses compiler techniques,  namely control flow integrity and software fault isolation.  And this keeps higher processor privilege levels free.  So we don't need the processor to have VMM extensions.  We don't have to use them  so they can be used for something else.  And it turns out that by using compiler instrumentation,  we are faster than current VMM based approaches.  With that, I'll take questions.  Thank you.  Yes?  What are your five screen questions?  So, one of the reasons why I wanted to present here  was to gauge how interested people are in this technology.  And one of the reasons for that is because  being a small school, I have a small research group.  So I have limited developer bandwidth.  You tell me?  Tell me.  Is this something that's interesting?  Does it sound too wild and crazy?  What do you think?  Well, just because it's a lot of crazy doesn't mean it  couldn't be.  Ha ha ha ha ha.  Sorry, I didn't mean to put you on the spot.  When I said you, I actually meant all of you.  I don't know, but I suspect that it's the sort of thing  that some previous people don't know that it might be  interested in, not all, but some might, at the very least,  making it easy to integrate these kinds of things  into a better platform for a reserve, which  some of us are excited about.  But also the mighty people who are interested in this  sort of thing is quite busy.  So again, maybe.  OK.  Question from the net?  Yes?  How much work do you support other architectures?  Support other architectures?  Good question.  So basically what you would need to do  is you would need to port the SVOS runtime library  to another architecture.  So you need to rewrite that.  If you design, if you port it to the virtual instruction  set properly, then that's probably  about all you need to do.  Sadly, that is not what I did in my research prototype,  because I was in a hurry and kind of learning  the low level parts of FreeBSD as I went.  So I probably didn't actually.  So basically what it is for the virtual instruction set port,  I didn't create a new port, because that was more work.  What I did is I took the x86 port,  ripped out the x86 port parts, and put my virtual instruction  set parts in.  But if you do it right, then you shouldn't  have to worry about that stuff.  And again, it's only about 50.  You also have to port the compiler parts.  But again, this is only 5,300 lines of code.  So it shouldn't be a lot of work.  So.  And the part of adoption among the DSPs is that one of the barriers  to entry is that it's going to take advantage of this template  that changes the way it works with programs,  through the PID programs.  It then makes it harder to maintain parity with other opportunities  that you see with that's got some evens overhead.  And if you could find a way to solve that,  that would probably go a long way with what you think  that's going to work more easily.  Right.  So to answer that, you're correct.  One of the interesting things is that,  so like for mobile applications, I think we can actually  use automated compiler transforms to take existing applications  and transform them into versions that protect themselves.  For regular Unix applications, it's more difficult  because we don't know which files are supposed to be shared  and therefore not encrypted so that everyone can see them  and which ones are supposed to be encrypted.  So it is possible, I think, though, to create an API  that the program just compiles to.  And then on systems that support virtual ghosts,  they do the whole encryption thing.  And then on systems that don't use virtual ghost,  the encryption is just a null operation.  Yes.  I think it's cool stuff.  I was wondering if you could tell SGX and how it's something like  it's kind of attacking the state of all this.  It's attacking a similar problem,  or rather SGX is starting to attack the same problems  that we're attacking.  So the first implementation of SGX,  SGX1 was primarily designed to take small bits of applications  and throw them into what they call an enclave,  this isolated environment.  And so to make things simple, they are maybe not simple,  but to provide the security guarantees they want to provide,  they did simplifying things like code and enclave  can't execute system calls.  So code and enclave, for example, can't receive a signal,  can't have a signal handler.  It can't do system calls, so it can't do read and write.  Whereas in virtual ghosts, they can't.  Now my understanding is that the group at Intel  is starting to develop a new version of SGX called SGX2,  which tries to fix some of these limitations  because they're wanting to do what virtual ghosts can do,  which is to protect an entire application.  I'm not familiar with exactly what they've done  and how far they are on that,  but I know that that's what they're trying to do.  Yeah, so the microsoft is attacking the internet,  but they're not.  Yeah, so I think you're talking about Haven from OSDI.  Yeah, so one thing about Haven that they've done  is I think they're trying to protect the application  from the hypervisor.  And what they've done for the operating system kernel,  they've said, okay, we're just going to use a little OS,  and so the OS is now part of the application.  Oh, that works, but that's not a commodity operating system kernel, right?  That's a library operating system.  So...  Yes?  What do you do with the dynamic libraries?  Ah, so we have not investigated dynamic libraries yet.  I think that it would be simple enough to use cryptographic signatures  so that when you ask for a dynamic library to be loaded,  you can actually verify that it is the correct library to be loaded,  but it's not something that I've actually thought about in detail yet.  Yes, ptrace does not work.  Any other questions?  Is this of any value for my cloud server?  Yes.  So, so it's...  So, so, yeah, so in a cloud server,  what you're going to want to do is you're going to want to be able to isolate,  first of all, in any server system, right,  or any server in which you care about security,  you want to isolate applications from each other,  and you want to isolate them from the system software,  because system software can be buggy,  and in particular, the way that we build commodity operating systems,  they're very large, they're very complicated, they're very privileged, right?  But in addition to that, on a cloud computing system,  what you'd want to do is you'd want to protect it from the hypervisor  and from other virtual machines, right?  Because you don't want other, you know, other people running on the same physical piece of hardware  that can access your data or corrupt your data.  So, yes, it's very relevant to that area.  One interesting piece of future work that we could do  is to look at extending the virtual instruction set to support a hypervisor,  and that way protect from, you know, basically protect applications,  not only from the operating system kernel,  but also from the hypervisor and other virtual machines  that could be running on the same hardware.  So, when do the test program work?  What prevents the OSO?  Is there a problem with yourself doing something else that's...  that you have to destroy with the program?  So, in other words, when you execute a program,  how do you know it's the right program?  Well, okay, so, first off, the question is,  is can the operating system kernel corrupt the application code?  And the answer is no, because of what I showed earlier,  where the application code is encrypted with the virtual ghost public key,  so only virtual ghost can decrypt it, so forth and so on.  Now, the second question is, let's say that you want to execute program foo.  The operating system kernel says, I like program bar better.  So, it executes program bar,  now you want to talk to say program bar over a pipe.  How do you know that it's foo...  or you want to talk to foo over a pipe, how do you know it's foo and not bar?  Well, that's why I said with the application key pair,  that's why it's a public-private key pair.  Because if you know the application's public key,  or let's say you have a digitally signed certificate,  where you know, okay, this is the public key of this application,  if the operating system kernel starts up another application  and you start talking to it, you can actually authenticate  that the application you're talking to is the one that you want.  So, if you know foo's public key,  then you can... if you're talking to bar,  bar won't be able to authenticate itself as foo.  So, basically, if I set a certificate in line type SSH key gen,  and the code and the kernels are compromised,  they will have to somehow check that I'm right,  and I want to do it by a time limit.  Right.  So, now in the case of SSH key gen,  what's going to happen is, like, let's say the operating system kernel  runs another application.  That application's going to have a different public-private key pair, right?  Because the operating system kernel doesn't know what your SSH key gen public-private key pair is.  So, if you then run your SSH client,  it's not going to be able to use the authentication keys  created by the operating system's SSH key gen  because it's not going to decrypt properly.  Right.  All right. Now, there are issues with,  okay, if I'm the user and I'm actually typing on my keyboard,  how do I know that the application I'm talking to is the right application?  How does the application know it's talking to me, the user,  as opposed to, like, some other program  that the operating system is set up to masquerade as the user?  That's an open research question,  which I have a few ideas for,  but that's definitely future work.  Yes.  You mentioned, like, so the applications aren't specifically key,  but there's no way to improve the application  because they're just only known by the...  I'll hypervisor, but I...  Virtual...  Virtual...  Yeah.  When the kernel load starts up in the virtual load  by the component that's within that kernel, right?  Or it's running on the outside.  I mean, is it being indicated  at no point in time that the kernel actually knows it's a key?  I mean...  So, if you have...  So, in a complete implementation, including one  that can automatically load new kernel code,  what you would have is your bootloader.  You'd use trusted boot to make sure you have a bootloader,  which loads virtualGhost, which loads the operating system kernel,  which then can start up applications.  Yeah, exactly. But I mean, it could be the case that if the hardware itself  sort of would be compromised,  it could just start up an operating system that behaves like  it's an operating system that runs virtualGhost.  In fact, it's just plain 3Bs that happens to know it's a key.  If you corrupt the processor, the actual hardware,  then, no, we don't defend against that.  That's not in our attack model.  So, yeah, you're definitely a cook then.  So...  How would this model protect against things like keyloggers?  Keyloggers?  Ah...  Okay, so I think virtualGhost provides the infrastructure  to protect against something like keyloggers,  but we don't actually have that implemented  because then what you're talking about is,  okay, how do you use these features?  And that becomes a whole bit more research, right?  So one way to potentially do it is to have,  essentially, to have a kernel driver that's trusted.  So what you do is you create kernel drivers that can have ghost memory.  And if you can do that, then an application,  when it starts talking to a driver, can actually authenticate it.  So basically, you have a few drivers that virtualGhost  gives keys saying, okay, this is the driver for the keyboard,  this is the driver for the screen.  So applications can basically get their public keys from virtualGhost,  authenticate them using those public keys,  and then talk to them over an encrypted channel.  That's one potential way that you could do it.  But...  That sounds almost like the next question.  Um...  Yes, but not to the same extreme,  because you don't have your file systems  and you don't have your networking subsystem  built as isolated drivers.  And more importantly, what I would like to do,  so this is something that I've been thinking about,  is I would like to keep these drivers in kernel space,  because I don't want to have to run the user space.  So, and that's an additional, you know,  that's an additional challenge of, you know,  can we actually do that?  So I think the answer is yes.  We'll be off the road on a specific address to pass on.  So I know that there's a few steps in the management system,  certain steps, you know, on the trust zone,  I think that a few steps are attached to the mechanism,  and so the question is,  if they have hardware support that they can find in,  and try to use it as well,  do I have to...  Yeah, I think it's basically a world-end,  a trust-based program.  And so the whole premise,  if you see the whole premise of this,  is that the program is only accepting trust-based inputs,  and then this virtual ghost  makes the program and trust that,  once it has been implemented,  that the data is not being modified,  and it's coded,  but it breaks down some of the trust-based inputs.  So the thing is, is that what you...  So in order to build a complete system,  like an end-to-end system,  you have to deal with issues such as,  how do you know you're actually talking to the user through all the pipes?  How do you...  You know, when you rely upon another piece of software,  how do you verify that it's doing what you want it to do?  And I think that there are solutions to those.  You know, proof-caring code might be useful,  or just using public-private encryption,  where applications are actually authenticating each other,  as opposed to just trusting each other like they do today.  But yes, it's an open problem.  It's not clear exactly how to solve them  or what the best way is to solve them.  Yes?  In your question earlier, there is always considerable interest in the previous D community on the security issue.  And then this is certainly one of them.  Have you been able to analyze this approach  with all the other security measures  that are already in the previous D family?  The jails, CAPSICOM?  They are addressing completely different issues.  So what CAPSICOM is doing and what jails are doing,  containers and all that stuff,  is that they are isolating user-space applications,  and they're trusting the operating system kernel to implement this correctly.  So... And it's a very valuable thing to do, I should add.  I used to build mandatory access controls to do this,  you know, probably about 15 or 17 years ago.  Okay, that's a long time ago.  Anyway, so that's a very good thing to do.  However, it's trusting the operating system kernel,  and what I found when I worked for Argus Systems Group,  which made extensions to Solaris and AIX with these mandatory access controls,  is that when attackers couldn't attack applications,  they attack the operating system kernel.  And the operating system kernel, it violates all the security design  that we know we're supposed to be doing.  It's large, it's monolithic, it's overly privileged.  You know, and so what my research work has focused on  is trying to address this challenge,  and I think that the commodity operating system kernels are not designed properly.  They're susceptible to buffer overflows, they're susceptible to rootkits,  so forth and so on, what can we do about it?  So basically it's orthogonal problems, is what it is, I would say.  So I think that's it.  If you have any other questions, please feel free to come by and chat with me.  Thank you.
Protecting_FreeBSD_with_Secure_Virtual_Architecture_0_107.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_108_359.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_360_467.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_468_527.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_528_707.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_708_755.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_756_827.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_828_983.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_984_1031.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1032_1319.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1320_1367.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1368_1415.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1416_1463.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1464_1511.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1512_1607.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1608_1979.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_1980_2387.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_2388_5039.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5040_5279.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5280_5339.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5340_5483.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5484_5639.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5640_5723.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5724_5771.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5772_5855.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5856_5951.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_5952_6011.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6012_6059.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6060_6203.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6204_6251.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6252_6335.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6336_6383.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6384_6515.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_6516_7763.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_7764_8423.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_8424_8903.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_8904_10019.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10020_10163.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10164_10355.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10356_10439.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10440_10499.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10500_10595.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10596_10643.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_10644_12491.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_12492_14567.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_14568_14615.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_14616_14663.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_14664_14747.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_14748_14903.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_14904_15083.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_15084_16427.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_16428_16967.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_16968_19463.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_19464_23399.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_23400_23447.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_23448_23843.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_23844_23903.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_23904_23975.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_23976_24251.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24252_24347.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24348_24419.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24420_24503.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24504_24623.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24624_24671.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24672_24743.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24744_24851.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24852_24899.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24900_24947.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24948_24995.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_24996_25223.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_25224_25571.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_25572_25643.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_25644_25703.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_25704_28607.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_28608_31019.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_31020_33203.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_33204_34919.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_34920_47351.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_47352_47411.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_47412_49091.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_49092_51179.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_51180_52463.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_52464_53531.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_53532_53591.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_53592_53687.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_53688_55115.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_55116_55799.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_55800_58367.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_58368_60095.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_60096_60863.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_60864_62699.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_62700_63995.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_63996_66767.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_66768_66815.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_66816_71831.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_71832_73367.png Protecting_FreeBSD_with_Secure_Virtual_Architecture_73368_99016.png