ToroV, a kernel in user-space, or sort of
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Vara, Matias
This talk presents ToroV, a novel open-source technology that combines virtualization and containerization to enable the execution of users’ applications in a safer and improved manner. In ToroV, applications run as Virtual Machines without the need of an OS, unikernel nor device-model. ToroV combines a minimalist Virtual Machine Monitor and a virtualized guest program communicating through POSIX APIs. When the guest application requires to open or write a file, it just invokes the VMM using hypercalls. The VMM intercepts those hypercalls, processes the request, and returns to the guest. The sysadmin defines the ACL (Access Control List) of authorized hypercall per virtualized guest application. This allows the user to control the host’s surface that is exposed to the guest. In this talk, we propose to present the ToroV architecture together with several ongoing experiments. For example, the minimalist VMM allows us to boot up a VM in KVM in less than 6 ms. Also, we show how we debug guest applications by simply using GDB and the KVM API for debugging. During the talk, we discuss the main differences with unikernels and containers and how ToroV gets the best of both worlds. Also, we present the main differences with gVisor, which is a similar project from Google.
 Hello everyone and thanks for joining me. I'm Matias Vara and I'm going to talk about TotoV, which is a currently new space or sort of. Before to start, I would like to thank all the people that made possible FOS10 this year. I would like first to present myself. I enjoy working on a patent system and on mutualization. I have worked at Citrix, TTTech and Huawei and you can find some of my projects at my GitHub webpage. Before go deeply into the presentation, I would like to picture the use case that motivates this project. I want to talk a bit about microservice and serverless applications. Some way to decompose monolithic application is by using service in which each service provides a functionality. For example, in this picture, I implement the application at left as three independent microservice. Instead, the future is to implement this microservice as serverless applications. It is the cloud provider who deals with the deployment. This presentation talks about how we can improve the deployment of serverless applications. In such a context, performance and isolation are key. There are basically two mechanisms to deploy serverless applications, either by using containers or by using virtual machines. These mechanisms should be able to limit host resources, prevent interference between applications and with the host, be efficient in terms of CPU and memory. In general, these mechanisms are chosen based on a tradeoff between the performance and security. By relying on these two parameters, let's analyze these solutions. When using containers, the applications run as a process on top of a general proposed operating system. These processes have a limited view of the host resources. This is implemented by relying on C groups and 9 space, which are features of the host operating system. The application still communicates with the host kernel by using C-scales. This deployment is very efficient, however, the host kernel is exposed so the attack surface is large. For example, a bug in the kernel host could be used to compromise all the containers. In this sense, containers do not offer a complete isolation. Instead, virtual machines provide a stronger isolation. To create virtual machines, we require a hypervisor that would handle the virtualization features of the hardware. In this case, applications run in a virtual machine on top of a guest operating system. In this case, only the kernel in the virtual machine can be compromised. However, a virtual machine requires a larger footprint because guest consume resources like memory, vcpu, and on-disk image. Also, the time to be up and running is longer than by using containers. This issues end up limiting the number of instances that a server can host. Also, this deployment requires a device model to provide hardware to the virtual machine, like for example, KMU. This represents a new component that is exposed to the guest that can be also compromised, together with the hypervisor. The device model can be reduced by using approaches like Firecracker or KMU micro-vm, in which only a few virtual devices are exposed to the guest. To reduce the complexity of a general proposed OS and the number of resources that a VM requires for a dedicated task, Sol approaches proposed the use of a unikernel to host microservice. A unikernel is when you compile the kernel with the user application. Currently, there are different unikernels which are used in different scenarios like Toto, OSv, MirageOS, Unicraft, Nano VMs, and so on. In this context, we wonder if we can just offload the guest kernel in the host. This is called a kernel-using space because it shows a process in the host serving requests from to the guest kernel. But what I call a kernel-using space? Well, it is a way to offload guest kernel to the host. In this design, the VM does not need a device model. Conversely, instead of exposing a whole device model, we expose some high-level service to the guest. In the following slide, I'm going to present, first, user-mod Linux and SheVisor and show the difference with TotoV. In user-mod Linux, we have basically two processes in the host, one process that holds the application and another that intercepts the syscalls. The interception mechanism is based on ptrace. The process that holds the syscalls emulates the whole Linux operating system. So basically, we have split the user space and kernel space into two processes. In approaches like SheVisor or in TotoV, the application runs as a VM on top of a simple kernel and still communicates with the guest kernel by using syscalls. However, this kernel may decide to forward some syscalls to a process that is running on the host. This is triggered by a VM exit from the guest operating system. In the host, this process emulates the service that a kernel will offer like file system or networking. Such service will be provided by using a minimal interaction with the host kernel. The reason to do this is to reduce the tax-for-face as much as possible of the host kernel. This is the case of SheVisor, which is a project from Google. In SheVisor, the serverless application is running in the context of a VM in RINs3 and communicates with the guest OS by using syscalls. These syscalls are trapped and then forwarded to the host process, naming as entry. This component does part of the syscall and delegates file system syscall to a service named gopher. These two components interact with the host kernel with a reduced number of syscalls, with short the host kernel attack surface. This approach increments the cost of a syscall, so applications that heavily use syscalls may perform not so well. Well, in this context, what is exactly then ToroV? Well, ToroV is a minimalistic kernel in user space in which syscalls from the guest are forwarded to the host. It allows the user to configure what syscalls are allowed per application. It provides a modified syscdleave that the user application must be compiled within. It explodes apoxie API based on hypercalls to the guest. It runs in the host as a container-eased process to reduce host attack surface. And it allows the user to debug this application by simply using gdv. In ToroV, the application runs in the context of a virtual machine, and when an application requires a service from the host, it calls the corresponding syscalls with trigger a hypercall. This hypercall is trapped by a component named virtual machine monitor, which processes the syscalls. For each application, the user set up what are the syscalls that are allowed. This is configured by using a JSON. For example, the JSON that you can see corresponds with the configuration for the hello world example. In this case, only the IOCTL and bright syscalls are allowed. The user can also configure if the process that holds the virtual machine monitor runs in its own PID namespace and root directory. So to build an application that runs in ToroV, we base on a modified glipc or fpcrtl, depending on the language that you are using. The application compiles into a ELF64, and from this file, we generate a flat binary, which is going to be loaded into the VM. When the application starts, the VM is in 64-bit load mode in Rinsuio. So in this sense, the virtual machine monitor is just a wrapper for the Kavian API. The following pictures shows the memory layout when the application starts. This is built by the virtual machine monitor. As we read the picture from left to right, we have first the page directory table, then the application binary, the heap, and the stack. Currently, we are limited to 2 megabytes for the application binary and less than 2 megabytes for the heap. In the current version, the heap is preallocated. This means that CSCalls like nmap shows increment pointer and returns. The amount of memory that we allocate is 2 megabytes per application. This memory is accessed from the virtual machine monitor. So when a CSCALL has to be forwarded to the host, the virtual machine monitor just modifies the address of the parameters to point to the corresponding address in the host process. Let's see this in an example. So let's see what happens when we launch the Heal or Work example by using S-Troys. In the first line, you have the command line that you can use to get the same output. This is a partial output in which I just highlight the most important things. You can see in the first line in green, the virtual machine monitor creates the vCPU and then creates a child process by using CLOM. This is the process that will intercept the CSCALLs from the guest. This process will have a limited view of the system by setting the corresponding flag in the CLOM CSCALLs. Then the child process changes the root directory and executes the vCPU. The first interception corresponds with the bright CSCALL, which we can see in red in the RAX register. Then the virtual machine monitor triggers the bright CSCALL but in the host with the corresponding parameters. These are some numbers for the Heal or Work application. It consumes about 1.5 megabytes of memory and it takes about 7 milliseconds to execute. This is an average. The cost of the bright CSCALL is about 10 times slower than in the host. This is the different steps to perform the CSCALL from the guest application to the host. As we can see, Toto V is still a proof of concept. There is a lot of work to do. These are some of the tasks that I have in mind for the future. If someone is interested in helping me, feel free to contact me. For example, I am currently working on porting Glypsy and I would like to also put other languages like GO or RAS to Toto V so we could generate vinyas that instead of using a CSCALL do a hypercall and interact with the host. I also research about comparing Toto V with ZCOM. There is also the possibility to run vinyas without recompiling it but by replacing the CSCALL instruction with some instruction that could trigger a VM-Essit like the out-destruction as I am doing now. Also the idea is to maybe pour the whole project to Rust. To try to work on the CSCALL's bottleneck which is where we spend most of the time, there is the idea that I could replace the asynchronous mechanism of the CSCALL for something that is asynchronous like for example a virtual device for CSCALL. So in that case the guest application will send packets by using build queues in which have all the information for performing the CSCALL in the host instead of executing an instruction that blocks the guest. And also regarding how it is implemented in the host, I would like to try to be more modular so maybe to support different binaries coming from different operating systems, we would like to handle in different ways the same CSCALL. So to do that maybe we can split the virtual machine monitor in components that handle differently the CSCALL, this is still a proof of comes idea. If you want to see more interesting examples, here you have a couple of them. I think one of the most interesting examples I have is the echo server in which you can see the use of the socket CSCALLs from the guest application. Also in that example I show how I use GDV to debug the application and the reason why I can do that is because ToroV includes a GDV staff so any application that you run on ToroV you could debug it by using a GDV client. Well, that's all folks, thank you very much for listening me and I will be online for questions. Thank you again very much. I have a question in which Leo asks why Pascal and I don't have a good answer for that. I just like the language but one of the future work is to replace it with Rust. Another question from Leo too said do you think it will scale at some point? Well, I'm not sure and if you compare with SheVisor at least some papers say that if you have a factor of 100 still 10 times. So I think depending on your use case you can tolerate that but I'm not sure yet. What Stefan is asking is if I have implemented something like the Linux VDSO where the CSCALL is actually handed inside the guest as a performance optimization, no I didn't. The reason why I didn't do that is because I wanted to have a minimal guest.
 Hello everyone and thanks for joining me.  I'm Matias Vara and I'm going to talk about TotoV, which is a currently new space or sort  of.  Before to start, I would like to thank all the people that made possible FOS10 this year.  I would like first to present myself.  I enjoy working on a patent system and on mutualization.  I have worked at Citrix, TTTech and Huawei and you can find some of my projects at my  GitHub webpage.  Before go deeply into the presentation, I would like to picture the use case that motivates  this project.  I want to talk a bit about microservice and serverless applications.  Some way to decompose monolithic application is by using service in which each service  provides a functionality.  For example, in this picture, I implement the application at left as three independent  microservice.  Instead, the future is to implement this microservice as serverless applications.  It is the cloud provider who deals with the deployment.  This presentation talks about how we can improve the deployment of serverless applications.  In such a context, performance and isolation are key.  There are basically two mechanisms to deploy serverless applications, either by using containers  or by using virtual machines.  These mechanisms should be able to limit host resources, prevent interference between  applications and with the host, be efficient in terms of CPU and memory.  In general, these mechanisms are chosen based on a tradeoff between the performance and  security.  By relying on these two parameters, let's analyze these solutions.  When using containers, the applications run as a process on top of a general proposed  operating system.  These processes have a limited view of the host resources.  This is implemented by relying on C groups and 9 space, which are features of the host  operating system.  The application still communicates with the host kernel by using C-scales.  This deployment is very efficient, however, the host kernel is exposed so the attack  surface is large.  For example, a bug in the kernel host could be used to compromise all the containers.  In this sense, containers do not offer a complete isolation.  Instead, virtual machines provide a stronger isolation.  To create virtual machines, we require a hypervisor that would handle the virtualization  features of the hardware.  In this case, applications run in a virtual machine on top of a guest operating system.  In this case, only the kernel in the virtual machine can be compromised.  However, a virtual machine requires a larger footprint because guest consume resources  like memory, vcpu, and on-disk image.  Also, the time to be up and running is longer than by using containers.  This issues end up limiting the number of instances that a server can host.  Also, this deployment requires a device model to provide hardware to the virtual machine,  like for example, KMU.  This represents a new component that is exposed to the guest that can be also compromised,  together with the hypervisor.  The device model can be reduced by using approaches like Firecracker or KMU micro-vm, in which  only a few virtual devices are exposed to the guest.  To reduce the complexity of a general proposed OS and the number of resources that a VM requires  for a dedicated task, Sol approaches proposed the use of a unikernel to host microservice.  A unikernel is when you compile the kernel with the user application.  Currently, there are different unikernels which are used in different scenarios like  Toto, OSv, MirageOS, Unicraft, Nano VMs, and so on.  In this context, we wonder if we can just offload the guest kernel in the host.  This is called a kernel-using space because it shows a process in the host serving requests  from to the guest kernel.  But what I call a kernel-using space?  Well, it is a way to offload guest kernel to the host.  In this design, the VM does not need a device model.  Conversely, instead of exposing a whole device model, we expose some high-level service to  the guest.  In the following slide, I'm going to present, first, user-mod Linux and SheVisor and show  the difference with TotoV.  In user-mod Linux, we have basically two processes in the host, one process that holds  the application and another that intercepts the syscalls.  The interception mechanism is based on ptrace.  The process that holds the syscalls emulates the whole Linux operating system.  So basically, we have split the user space and kernel space into two processes.  In approaches like SheVisor or in TotoV, the application runs as a VM on top of a simple  kernel and still communicates with the guest kernel by using syscalls.  However, this kernel may decide to forward some syscalls to a process that is running  on the host.  This is triggered by a VM exit from the guest operating system.  In the host, this process emulates the service that a kernel will offer like file system  or networking.  Such service will be provided by using a minimal interaction with the host kernel.  The reason to do this is to reduce the tax-for-face as much as possible of the host kernel.  This is the case of SheVisor, which is a project from Google.  In SheVisor, the serverless application is running in the context of a VM in RINs3 and  communicates with the guest OS by using syscalls.  These syscalls are trapped and then forwarded to the host process, naming as entry.  This component does part of the syscall and delegates file system syscall to a service  named gopher.  These two components interact with the host kernel with a reduced number of syscalls,  with short the host kernel attack surface.  This approach increments the cost of a syscall, so applications that heavily use syscalls  may perform not so well.  Well, in this context, what is exactly then ToroV?  Well, ToroV is a minimalistic kernel in user space in which syscalls from the guest are  forwarded to the host.  It allows the user to configure what syscalls are allowed per application.  It provides a modified syscdleave that the user application must be compiled within.  It explodes apoxie API based on hypercalls to the guest.  It runs in the host as a container-eased process to reduce host attack surface.  And it allows the user to debug this application by simply using gdv.  In ToroV, the application runs in the context of a virtual machine, and when an application  requires a service from the host, it calls the corresponding syscalls with trigger a  hypercall.  This hypercall is trapped by a component named virtual machine monitor, which processes the  syscalls.  For each application, the user set up what are the syscalls that are allowed.  This is configured by using a JSON.  For example, the JSON that you can see corresponds with the configuration for the hello world  example.  In this case, only the IOCTL and bright syscalls are allowed.  The user can also configure if the process that holds the virtual machine monitor runs  in its own PID namespace and root directory.  So to build an application that runs in ToroV, we base on a modified glipc or fpcrtl, depending  on the language that you are using.  The application compiles into a ELF64, and from this file, we generate a flat binary,  which is going to be loaded into the VM.  When the application starts, the VM is in 64-bit load mode in Rinsuio.  So in this sense, the virtual machine monitor is just a wrapper for the Kavian API.  The following pictures shows the memory layout when the application starts.  This is built by the virtual machine monitor.  As we read the picture from left to right, we have first the page directory table, then  the application binary, the heap, and the stack.  Currently, we are limited to 2 megabytes for the application binary and less than 2 megabytes  for the heap.  In the current version, the heap is preallocated.  This means that CSCalls like nmap shows increment pointer and returns.  The amount of memory that we allocate is 2 megabytes per application.  This memory is accessed from the virtual machine monitor.  So when a CSCALL has to be forwarded to the host, the virtual machine monitor just modifies  the address of the parameters to point to the corresponding address in the host process.  Let's see this in an example.  So let's see what happens when we launch the Heal or Work example by using S-Troys.  In the first line, you have the command line that you can use to get the same output.  This is a partial output in which I just highlight the most important things.  You can see in the first line in green, the virtual machine monitor creates the vCPU and  then creates a child process by using CLOM.  This is the process that will intercept the CSCALLs from the guest.  This process will have a limited view of the system by setting the corresponding flag in  the CLOM CSCALLs.  Then the child process changes the root directory and executes the vCPU.  The first interception corresponds with the bright CSCALL, which we can see in red in  the RAX register.  Then the virtual machine monitor triggers the bright CSCALL but in the host with the corresponding  parameters.  These are some numbers for the Heal or Work application.  It consumes about 1.5 megabytes of memory and it takes about 7 milliseconds to execute.  This is an average.  The cost of the bright CSCALL is about 10 times slower than in the host.  This is the different steps to perform the CSCALL from the guest application to the host.  As we can see, Toto V is still a proof of concept.  There is a lot of work to do.  These are some of the tasks that I have in mind for the future.  If someone is interested in helping me, feel free to contact me.  For example, I am currently working on porting Glypsy and I would like to also put other  languages like GO or RAS to Toto V so we could generate vinyas that instead of using a CSCALL  do a hypercall and interact with the host.  I also research about comparing Toto V with ZCOM.  There is also the possibility to run vinyas without recompiling it but by replacing the  CSCALL instruction with some instruction that could trigger a VM-Essit like the out-destruction  as I am doing now.  Also the idea is to maybe pour the whole project to Rust.  To try to work on the CSCALL's bottleneck which is where we spend most of the time, there  is the idea that I could replace the asynchronous mechanism of the CSCALL for something that  is asynchronous like for example a virtual device for CSCALL.  So in that case the guest application will send packets by using build queues in which  have all the information for performing the CSCALL in the host instead of executing an  instruction that blocks the guest.  And also regarding how it is implemented in the host, I would like to try to be more  modular so maybe to support different binaries coming from different operating systems, we  would like to handle in different ways the same CSCALL.  So to do that maybe we can split the virtual machine monitor in components that handle  differently the CSCALL, this is still a proof of comes idea.  If you want to see more interesting examples, here you have a couple of them.  I think one of the most interesting examples I have is the echo server in which you can  see the use of the socket CSCALLs from the guest application.  Also in that example I show how I use GDV to debug the application and the reason why  I can do that is because ToroV includes a GDV staff so any application that you run on  ToroV you could debug it by using a GDV client.  Well, that's all folks, thank you very much for listening me and I will be online for  questions.  Thank you again very much.  I have a question in which Leo asks why Pascal and I don't have a good answer for that.  I just like the language but one of the future work is to replace it with Rust.  Another question from Leo too said do you think it will scale at some point?  Well, I'm not sure and if you compare with SheVisor at least some papers say that if  you have a factor of 100 still 10 times.  So I think depending on your use case you can tolerate that but I'm not sure yet.  What Stefan is asking is if I have implemented something like the Linux VDSO where the CSCALL  is actually handed inside the guest as a performance optimization, no I didn't.  The reason why I didn't do that is because I wanted to have a minimal guest.
Vai_Torov_0_47.png Vai_Torov_48_299.png Vai_Torov_300_491.png Vai_Torov_492_1019.png Vai_Torov_1020_1787.png Vai_Torov_1788_2507.png Vai_Torov_2508_3527.png Vai_Torov_3528_4871.png Vai_Torov_4872_7463.png Vai_Torov_7464_8375.png Vai_Torov_8376_8891.png Vai_Torov_8892_9815.png Vai_Torov_9816_10583.png Vai_Torov_10584_13403.png Vai_Torov_13404_14639.png Vai_Torov_14640_16571.png Vai_Torov_16572_17735.png Vai_Torov_17736_20111.png Vai_Torov_20112_22319.png Vai_Torov_22320_23315.png Vai_Torov_23316_27287.png Vai_Torov_27288_28631.png Vai_Torov_28632_28823.png Vai_Torov_28824_28919.png Vai_Torov_28920_29039.png Vai_Torov_29040_29087.png Vai_Torov_29088_29447.png Vai_Torov_29448_29639.png Vai_Torov_29640_30311.png Vai_Torov_30312_30383.png Vai_Torov_30384_30767.png Vai_Torov_30768_30770.png