KubeVirt: privilege dropping one capability at a time
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Barroso, Miguel Duarte
KubeVirt's architecture is composed of two main components: virt-handler, a trusted DaemonSet, running in each node, which operates as the virtualization agent, and virt-launcher, an untrusted Kubernetes pod encapsulating a single libvirt + qemu process. To reduce the attack surface of the overall solution, the untrusted virt-launcher component should run with as little linux capabilities as possible. The goal of this talk is to explain the journey to get there, and the steps taken to drop CAP NET ADMIN, and CAP NET RAW from the untrusted component. This talk will encompass changes in KubeVirt and Libvirt, and requires some general prior information about networking (dhcp / L2 networking).
 you Hello, good morning. My name is Miguel Duarte Borroz. I am a senior software developer working for Red Hat on the networking team of our downstream distribution of Kubevert. For those of you that are not aware, Kubevert is a virtualization plugin for Kubernetes that allows the user to run a virtual machine within a Kubernetes spot. I'm here today to present a privilege dropping one capability at a time on the virtualization and infrastructure as a service dev room at FOSDEM 2021. In this talk, we will explain what Linux capabilities are, how they improve security, and especially how we have refactored Kubevert's code to reduce the overall attack surface, thus effectively making our virtualization solution more secure. You can find my email address, along with my GitHub profile link in the bottom left. You can contact me via those. Without further ado, let us proceed to the agenda. We will begin with an introduction about Linux capabilities, where I will explain what they are and how the kernel uses them. I will furthermore perform a small demo about those, which will hopefully make it all easier to understand. Once we have common understanding about Linux capabilities, I will briefly explain the Kubevert architecture, focusing on how security is segmented in its process model and how networking is configured for virtual machines running in it. With these pieces, knowledge about capabilities and minimal knowledge about networking configuration within Kubevert, I will then proceed into explaining our motivation for the work behind this presentation, along with its problem statement, and ultimately the goals of the development work predating this talk. We will then explain the considerations we've had and how the code was refactored to allow dropping the capabilities. We will then conclude the talk with a list of next steps, since there are a few nits left behind yet to be addressed. Let's finally begin the presentation by moving into the introduction section. When evaluating permission checks in Linux, there are two categories of processes, privileged, owned by super user, effective user ID zero, and unprivileged, literally everyone else. A privileged process bypasses all kernel permission checks, whereas an unprivileged one is subject to full permission checking based on the caller's credentials, effective user ID and group ID, and also further group membership. This sounds and is a one-size-fits-all solution. Take the ping command, for example. A while back, it had to be granted full privileged access so it could bind to a raw socket. An attacker, if he could exploit the ping command and succeeded in compromising it, would have unlimited access to the system. Such capabilities were introduced in kernel 2.2. The privileged kernel calls were split into groups of related functionality that can be independently enabled or disabled on a per thread basis. If you need to bind to a raw socket, you shouldn't need to be super user for that. After this refactor, you would use the net rock capability instead. If you need to create or configure a bridge or a Veth pair or any other virtual interface type, you would require the net admin capability instead of being super user. With this improved model, if an attacker manages to successfully compromise a piece of the system, he does just that. Compromises a piece of the system instead of being granted unlimited access to it. On this talk, I'll focus on the two network related capabilities that Qvert is using. Capability net raw and capability net admin. I will now perform a short demo where we will see what groups of privileged calls are controlled by each of the aforementioned capabilities. You can find the detailed guides in the repository mentioned in this slide. Let's begin by demoing the group of calls that require the net admin capability. For this, we will require two shells. In the one above, we will run a Docker container based on the CentOS 8 image that does feature the net admin capability. We will execute the bash command in it because we will be interacting directly with the container. In the shell below, on the other hand, we will execute the exact same thing with the difference that we will not load the net admin capability into the container. We will now create a bridge on the container that features the net admin capability. As we can see, creating the bridge was possible and we have our Br0 bridge within the container. We will now replicate the experiment on the container that does not feature the net admin capability. And we can see that we got an error back. The operation is not permitted. We are missing the required permissions, in this case net admin capability to perform this operation. The same would happen for every other type of virtual interface. For instance, if you created a tap device, the results would have been the same. For VethPair, exactly the same. You would need this capability in order to create those. Let us now try to configure an existing interface instead. For instance, we will try and configure the MAC address of the Ethernet zero. As we can see, it succeeded on the container that features the capability. Let us try in the one that does not have it. And as we can see, it failed. This means that configuring an existing interface also requires this capability. The same would have happened if we updated the DMT instead or any other sort of parameter. In this part of the demo, we will showcase how the net rocket capability impacts network operations. For it, we will create two containers. The first will feature the net rocket capability and the other will not have it. Then we will try to create a raw socket. For this, we will use a custom image that you can build from the repository previously mentioned in the presentation. It features a simple goal-line program that attempts to create and configure different types of sockets. For our first experiment, we must ensure that the net raw capability is not present on the container. For that, we use the cap drop option of Docker. We will run the capabilities demo image, which is generated from the repository that I have linked on the presentation. Let us now check the CLI parameters on the image. Cap raw, okay, I have made a typo here. It is net raw instead of cap raw. And the command that we will be using is the raw socket. This will request a raw socket and we will bind it to an address that features the Ethernet zero zone. And as we can see, since we do not have the net raw capability since we were dropping it, we get an operation not permitted error. Let us now replicate this experiment, but instead of dropping the capability and ensuring that it is present by using cap add. As we can see, it succeeded. We managed to create a raw socket on the interface, ethernet bound to the Ethernet zero zone. You can check the code for this on the repository as well. Let us now move to the second net raw example where we do something different. Instead of opening a raw socket, we open a datagram socket and set the bind to device socket option on the datagram socket. For that, we will use the same image as before this time with the bind to device command. It also receives a parameter. We will still use the ethernet zero interface as our guinea pig. And in the scenario, we are adding the net raw capability. So I expect this to work. As we can see, we succeeded in creating a datagram socket. We managed to bind it to the device ethernet zero via the bind to device socket option. And we bound it to the 547 port. Let us now do the exact same thing, but this time ensuring the capability is not present. As we see, we got an error back. We cannot bind to the interface without the cap net raw or root permissions. You can see the code for this short demo in the repository I've linked before. And in fact, I encourage you to do so. I will finalize this section with a few examples of attacks that would be possible by exploiting these capabilities and we'll follow up with a demo of one of them. With net admin, you could leak the HTTP replies from the pod or container to the host that's effectively reconfiguring the host's networks. With it, you could install a new default route in the system or a new DNS server. This would result in taking over address or DNS resolution. With net raw, you've already granted the ability to craft raw packets like ARP and DNS, which even malicious user ample room to play with network related attacks. An attacker could spoof the identity of a default gateway, thus replacing it with a gateway under his or her control. The same can be applied to DNS, which means that net raw can also be exploited to take over address or DNS resolution from the host. In order to demo this attack, we will need first to create a Veth pair in the host's network namespace. It will be called sneaky Veth. Its type will be Veth, obviously, and its peer name will be sneaky Veth-BR, so we can easily identify which end of the Veth is to be connected to the Docker zero bridge. We will then connect the sneaky Veth-BR end of the Veth pair to the Docker zero bridge in the host's namespace and also set both sides of the Veth pair to the up state. As we can see, we have both the sides of the Veth pair, one connected to the Docker zero bridge and both of them in the up state. One thing left to do, let's take a brief look at our routing table, particularly at the default routes. There's a single one installed, and let's check connectivity to a well-known address, let's say redhat.com. It works, we are connected to the internet. Let us now start a T-Mock session and start our DHCP server. Its name will be DHCP server, let's not invent much here. It will, it does require the net admin capability as we've been discussing throughout this time. We need to pull it from the correct registry because we are again using our custom built image that you can find available on the repo. And let's call the capabilities demo binary with the start DHCP server argument, bind that the DHCP server to the Ethernet zero interface. Let's advertise this slider, the 10.10.10.2 slash 24. The IP server will be in the same subnet, but with the dot one address. And here in the router, this will be what we will advertise as a default route. Let's just put some garbage in and specify a weirdly enough MTU. We're now pulling the container image and we have started the DHCP server. Now let us look at our interfaces again and start an instance of DH client on our sneaky Vef, end of the Vef pair. Okay, we can see that we got a reply. We got these logs on the DHCP server shell. We were offered an IP address. We are now bound to the dot two IP address, which is the slider that we advertise. We can now see the IP address that we were offered from the server. And we can also see the MTU that we advertised. Let's take a look at default routes. And we can see that we have now two new default routes, the one that I called Garovitch before. And let's take a look. Let's ping the same well-known address and we see that our networking is broken. Basically, because we have configured a default route with a lower metric. Now that we have a basic understanding about Linux capabilities, let us focus on the Qvert architecture. Qvert is a virtualization plugin for Kubernetes and follows its declarative philosophy. When a set of specifications are posted to the Kubernetes API server, they are then transformed into an internal object of a specific type. After that, a specific controller that has previously registered itself as that type's handler will push all the buttons required to make that request happen. For instance, once a new pod object is seen internally, the respective controllers perform the necessary actions to bring the pod alive and match the required state. Qvert delivers three things to provide a new functionality. The first, additional virtualization-related types known by custom resource definitions in Kubernetes lingo that will extend the existing API. The second, additional controllers for cluster-wide logic associated with these new types. And finally, additional demons for no specific logic associated with the new types. An example task is mounting the KVM device required by QM into the pods that will hold the user's VM. The Qvert process model is simple. There is a virtualization controller that specifies the pod where the VM will run. A process called Virt handler, which operates as Qvert's agent in each of the nodes. And finally, a virtual launcher process that will encapsulate the libvert and QM processes that will run the user's virtual machine. On this talk, I will be focusing on Virt handler, again, Qvert's agent, and in Virt launcher, which encapsulates the user's VM. Virt handler is a trusted component of Qvert. It runs on a privileged demon set and is responsible for creating and configuring any required infrastructure for the pod encapsulating the VM. Virt launcher process, on the other hand, is an untrusted component of Qvert, since it wraps the libvert process that will run third party workloads. As a result, it must be executed with as little privileges as required. Throughout the reminder of the presentation, I will be referring to these two processes by trusted and untrusted components. Keep in mind that the former refers to Virt handler, while the latter refers to Virt launcher. Configuring networking for a virtual machine is, in its essence, composed of two different operations. To perform those, Qvert introduced the bind mechanism concept. Bind mechanism is something that binds the pod interface to the virtual machine interface. Its first task is how to extend the networking that Kubernetes delivers for the pods into the virtual machine that will be spawned. Let's see how that is done. I am a firm believer that it is easier to understand something when given an example. So let's imagine we want to create a virtual machine having two different interfaces. When the pod is created, Kubernetes agent in the node invokes the corresponding CNI plugins, which are responsible for creating and configuring a network interface in the newly created pod. If we recall our example, we requested two interfaces. It is part of CNI's responsibility to create these interfaces in our pods. The bind mechanisms will then kick in and create all sorts of auxiliary networking infrastructure to extend networking from the pod interface into what will be our virtual machine. It is important to refer that there is a different bind mechanism for each interface type. The reference bind mechanism shown in our previous example will create an input bridge for each interface along with a tap device for the VM, both of which they will configure accordingly. In the example, both the tap device and the pod interface is connected to this input bridge, which effectively extends the pod networking into the future virtual machine. Let's not get into too much detail in this. The main message here is that each binding mechanism does different things, but all of them will create and configure different types of networking infrastructure, which, as we've seen before, is all privileged operations. You must remember that it is actually libvert that instantiates the virtual machine, an operation that is specified in a domain XML document. Hence, one of the steps involved in the networking configuration for the virtual machine is translating from the kubvert virtual machine specification to a libvert DOM XML document. This short example shows parts of both, focusing on the network-related entities. In the left, you can see that we are requesting a VM with two interfaces, one of type bridge, another type master rate. The second part of the bind mechanism will perform this translation from the kubvert specification into a domain XML document, which will finally be fed to libvert. Libvert, in turn, will transform that XML document into a sequence of parameters for QM, the architectural element responsible for running the VM. There are two main takeaways from this architecture section. The first is that kubvert has two types of components, a trusted component, which is kubvert's agent in each of the nodes, and an untrusted component that will execute the user's VMs and as such run third-party code. The other takeaway is that network configuration occurs in two different stages, one that requires privileged operations, since it is creating and configuring network infrastructure, and another stage that only features unprivileged operations, because it is mainly translating from kubvert spec into libvert's DOM XML. Now that we have a clear picture of kubvert's process architecture and how networking is configured for the virtual machines, we can describe what is our motivation, problem, and goals. The motivation should be pretty clear by now. We want to reduce the overall attack surface of the solution, in a way that if a malicious user manages to compromise the untrusted component of the architecture, the potential damage to the system is not severe. With this in mind, the problem can be clearly defined. An untrusted component of the architecture is executing privileged operations, and as such the whole component is being granted more capabilities than what it should have. Finally, our goals, despite somewhat complex, are straightforward. We want to remove both the net admin and the net rock capabilities from the untrusted component of the architecture. In order to allow the aforementioned capabilities to be dropped, we have first to split the network configuration, making sure that all the privileged operations are executed in the trusted component. Let's see how that was possible. Originally, the prepare pod networking stage was executed on the untrusted component. It would create the input bridge, configure it, and connect it to the pod interface. All of these are privileged operations, and as such should now be relocated to the trusted component of the architecture. After that, on a later stage, KubeVertWood, translate the KubeVertVM specification into a libvert.xml, which would be used to instantiate a running virtual machine. The solution is to move these privileged operations to the trusted component, but execute them in the untrusted component's network namespace. To do that, we need to use the untrusted process, PID, to load its network namespace. This activated rectangle in the right represents just that. Despite the fact that the prepare pod networking calls are now executed in the trusted component, it happens in the untrusted process network namespace. The second stage of VM network configuration will still be executed in the untrusted process, which is perfectly fine, since it does not require any particular set of capabilities. You can find in this slide both the KubeVert issue and the pull request where this work was implemented. In this section of the presentation, I will explain how we have refactored KubeVert's code in order to remove the NetRoc capability from the untrusted component of the process architecture. If you remember the demos I've shown previously in the introduction section, you would remember I showed two tasks that required the NetRoc capability. The first, open a raw socket. The second, setting the SO bind to device socket option. While our codebase did not use raw sockets, our DHCP server implementation that dynamically configures VM IP addresses was using this option. As such, we had to find an alternative way of filtering data from the center face and limit writing to it. You can find the pull request that implemented this feature in KubeVert referenced in the slide. To have a proper understanding of the performary factor, we should first understand what was the initial status quo. Originally, our in-pot DHCP server was creating a datagram socket and it would afterwards set the SO bind to device socket option in it, which would then drop all the received traffic from interfaces other than the one requested. In this short example, let us consider we want to bind to interface number two. Finally, we would then bind the socket to an address. In this case, it would be port 67, which is DHCP servers listening port. For this net-raw related refactor, the idea is to manually filter incoming traffic by the interface index. In order for that happen, the in-pot DHCP server creates a datagram socket. It would then directly bind the socket to DHCP server port, which is still port 67. This step would leave the socket listening to all the interfaces in the pot, and outgoing interface selection would happen based on the kernel routing table. This is far from what we require. To preserve the whole behavior, we need to listen only to a particular interface and to send the DHCP replies via that very same interface. To enable that, we first need to ask the kernel to decorate the incoming packets with the interface index from which they were received on. Afterwards, whenever we receive a packet view socket, we can compare the interface from which the packet came from with the interface we are interested on. All traffic originating from an interface that we are not interested in will be silently dropped at the application level. Finally, we reply back to the sender, indicating the interface we want to send the packet to via Linux control messages. In this next section, I will focus on the work done to enable the net admin capability to be removed from the untrusted component pod specification. Let's begin by enumerating the calls that are executed to configure the networks for our virtual machines that do require the net admin capability. Those are creating and configuring the tap device, creating the input bridge, connect the tap device to the input bridge. Depending on the bind mechanism, connect the pod interface to the input bridge and finally configure the input bridge. I am breaking down the input bridge configuration part in two different tasks for a simple reason. Part of this was already performed when we split the network configuration to occur in two different stages, configuring the MAC and MTU of the bridge was already addressed and explained. However, we had left out the small configuration detail. We had yet to move the sampling, the offloading of the transmit checksum from the untrusted components to the trusted components. This operation uses an IOCDL that requires the net admin capability. As such, and once that was done, we were missing two tasks. The first, create and configure the tap device on behalf of libvert and the second, connect the tap device to the input bridge, which obviously requires the tap device to have been created previously. You must remember that it was originally libvert, which is running in the untrusted components that was creating the tap device and that creating a virtual interface is an operation that does require the net admin capability. Again, the solution was to move the tap device creation from the untrusted components to the trusted one and then indicate libvert that it should use a pre-existing tap device instead of creating a new one. For this last step to happen, we required libvert to learn a new trick. It had to consume preconfigured tap devices without touching their configuration. Furthermore, our untrusted component domain XML generation mechanism, the part of the bind mechanism that translates from Qvert language to libvert language, had to be updated to use the new API libvert kindly provided for us. Before this change, Qvert translated the user provided specification to an interface type bridge, which would task it with creating a new tap device and attach the tap device to the bridge specified in the target element. libvert already had an ethernet type through which it would create and configure a new tap device, which would in turn allow the virtual machine to have an emulated network device. Upon learning about a requirement, the libvert team introduced a new managed parameter to the target element. If a non-managed already existing target is requested, no further configuration is performed, which allows the preconfigured tap device to be imported by libvert without having the net admin capability. There is a tiny nuance to this. This capability is not needed only if the tap creator and the tap consumer user and group IDs match. This gives Qvert the ability to consume a preconfigured tap device, but for that to happen, the tap device must be created and configured beforehand. In this step, we have had akin to what I have shown previously for the bridge, to create a tap device on the trusted component on behalf of the untrusted component. For it, we had to use the untrusted process PID to tap into the untrusted component network namespace, so the tap device is available for when libvert attempts to consume it. It was somewhat trickier than when creating the bridge since SU lin is frowned upon our plan. Then, despite creating the tap on the untrusted component network namespace, it is created by the trusted component. And as such, the tap device internal structure, which is the ton socket, is inheriting the SU linx context from its creator. In this case, it received the super privileged type context. When libvert, which runs using the SU linx context of the untrusted component, attempts to consume the existing tap, SU linx prevents it since the context of the resource, the ton socket, and the context of the process accessing the resource do not match. To appease SU linx, we have had to create the tap device with the untrusted component SU linx context, which is something the super privileged pod has access to. You can find below in this short code snippet how the privileged pod accesses the SU linx context of another process. Now that Qvert's trusted component is able to successfully create the tap device in the network namespace of the untrusted component using the correct SU linx context, all that was left to do was to update the translation mechanism between the Qvert specification language, the Qvert Stomach XML to use the new API that libvert has provided for us. Removing the net admin capability from the untrusted component pod is pending a single leftover. When libvert is instructed to import a pre-configured, pre-existing tap device via the manageuplesno parameter in the target element, but also instructed to configure the guest's MTU via the MTU element, it should not configure the tap site and perform only MTU configuration of the emulated device present within the virtual machine. Remember that updating the MTU of virtual interface does require the net admin capability. This is tracked in the bug mentioned below and was patched upstream mid-January. Qvert will update this version of libvert as soon as possible. Let's finalize this presentation with a list of next steps to be done. As I've just mentioned, we need to wait for the bug that I've mentioned previously about the MTU on libvert to be released, like the fix is already merged upstream and we just need to wait for it to be available on an upstream distribution. Once we have that, we'll update the libvert version with this new patched version and with that we can finally remove the cabinet admin from the verb launcher, which is the untrusted component pot template by reverting this commit right here. And this is all. Thank you very much for your time. I hope you had some fun and learned something and I'm now available for some questions. Thank you again. you you you you you you you you you you you you you you you you you you
 you  Hello, good morning. My name is Miguel Duarte Borroz. I am a senior software developer working  for Red Hat on the networking team of our downstream distribution of Kubevert. For those of you that  are not aware, Kubevert is a virtualization plugin for Kubernetes that allows the user to run a  virtual machine within a Kubernetes spot. I'm here today to present a privilege dropping one  capability at a time on the virtualization and infrastructure as a service dev room at FOSDEM 2021.  In this talk, we will explain what Linux capabilities are, how they improve security,  and especially how we have refactored Kubevert's code to reduce the overall attack surface,  thus effectively making our virtualization solution more secure. You can find my email address,  along with my GitHub profile link in the bottom left. You can contact me via those.  Without further ado, let us proceed to the agenda.  We will begin with an introduction about Linux capabilities, where I will explain what they are  and how the kernel uses them. I will furthermore perform a small demo about those, which will  hopefully make it all easier to understand. Once we have common understanding about Linux  capabilities, I will briefly explain the Kubevert architecture, focusing on how security is  segmented in its process model and how networking is configured for virtual machines running in it.  With these pieces, knowledge about capabilities and minimal knowledge about networking configuration  within Kubevert, I will then proceed into explaining our motivation for the work behind  this presentation, along with its problem statement, and ultimately the goals of the  development work predating this talk. We will then explain the considerations we've had and how the  code was refactored to allow dropping the capabilities. We will then conclude the talk with a list of  next steps, since there are a few nits left behind yet to be addressed.  Let's finally begin the presentation by moving into the introduction section.  When evaluating permission checks in Linux, there are two categories of processes,  privileged, owned by super user, effective user ID zero, and unprivileged, literally everyone else.  A privileged process bypasses all kernel permission checks, whereas an unprivileged one  is subject to full permission checking based on the caller's credentials, effective user ID and  group ID, and also further group membership. This sounds and is a one-size-fits-all solution.  Take the ping command, for example. A while back, it had to be granted full privileged access so  it could bind to a raw socket. An attacker, if he could exploit the ping command and succeeded  in compromising it, would have unlimited access to the system. Such capabilities were introduced  in kernel 2.2. The privileged kernel calls were split into groups of related functionality that  can be independently enabled or disabled on a per thread basis. If you need to bind to a raw  socket, you shouldn't need to be super user for that. After this refactor, you would use the  net rock capability instead. If you need to create or configure a bridge or a Veth pair or any other  virtual interface type, you would require the net admin capability instead of being super user.  With this improved model, if an attacker manages to successfully compromise a piece of the system,  he does just that. Compromises a piece of the system instead of being granted unlimited access to it.  On this talk, I'll focus on the two network related capabilities that Qvert is using.  Capability net raw and capability net admin.  I will now perform a short demo where we will see what groups of privileged calls are controlled by  each of the aforementioned capabilities. You can find the detailed guides in the repository mentioned  in this slide. Let's begin by demoing the group of calls that require the net admin capability.  For this, we will require two shells. In the one above, we will run a Docker container based  on the CentOS 8 image that does feature the net admin capability. We will execute the bash command in  it because we will be interacting directly with the container. In the shell below, on the other  hand, we will execute the exact same thing with the difference that we will not load the net admin  capability into the container. We will now create a bridge on the container that features the net  admin capability. As we can see, creating the bridge was possible and we have our Br0 bridge  within the container. We will now replicate the experiment on the container that does not feature  the net admin capability. And we can see that we got an error back. The operation is not permitted.  We are missing the required permissions, in this case net admin capability to perform this operation.  The same would happen for every other type of virtual interface. For instance, if you created a  tap device, the results would have been the same. For VethPair, exactly the same. You would need  this capability in order to create those. Let us now try to configure an existing interface instead.  For instance, we will try and configure the MAC address of the Ethernet zero.  As we can see, it succeeded on the container that features the capability. Let us try in the one that does not have it.  And as we can see, it failed. This means that configuring an existing interface also requires this capability.  The same would have happened if we updated the DMT instead or any other sort of parameter.  In this part of the demo, we will showcase how the net rocket capability impacts network operations.  For it, we will create two containers. The first will feature the net rocket capability and the  other will not have it. Then we will try to create a raw socket. For this, we will use a custom image  that you can build from the repository previously mentioned in the presentation. It features a  simple goal-line program that attempts to create and configure different types of sockets.  For our first experiment, we must ensure that the net raw capability is not present on the  container. For that, we use the cap drop option of Docker. We will run the capabilities demo image,  which is generated from the repository that I have linked on the presentation. Let us now check the  CLI parameters on the image. Cap raw, okay, I have made a typo here. It is net raw instead of cap raw.  And the command that we will be using is the raw socket. This will request a raw socket and we will  bind it to an address that features the Ethernet zero zone. And as we can see, since we do not have  the net raw capability since we were dropping it, we get an operation not permitted error. Let us  now replicate this experiment, but instead of dropping the capability and ensuring that it is  present by using cap add. As we can see, it succeeded. We managed to create a raw socket on the  interface, ethernet bound to the Ethernet zero zone. You can check the code for this on the  repository as well. Let us now move to the second net raw example where we do something  different. Instead of opening a raw socket, we open a datagram socket and set the bind to device  socket option on the datagram socket. For that, we will use the same image as before this time  with the bind to device command. It also receives a parameter. We will still use the ethernet zero  interface as our guinea pig. And in the scenario, we are adding the net raw capability. So I expect  this to work. As we can see, we succeeded in creating a datagram socket. We managed to bind it to the  device ethernet zero via the bind to device socket option. And we bound it to the 547 port.  Let us now do the exact same thing, but this time ensuring the capability is not present.  As we see, we got an error back. We cannot bind to the interface without the cap net raw or root  permissions. You can see the code for this short demo in the repository I've linked before. And  in fact, I encourage you to do so. I will finalize this section with a few examples of attacks that  would be possible by exploiting these capabilities and we'll follow up with a demo of one of them.  With net admin, you could leak the HTTP replies from the pod or container to the host that's  effectively reconfiguring the host's networks. With it, you could install a new default route in  the system or a new DNS server. This would result in taking over address or DNS resolution.  With net raw, you've already granted the ability to craft raw packets like ARP and DNS,  which even malicious user ample room to play with network related attacks. An attacker could  spoof the identity of a default gateway, thus replacing it with a gateway under his or her  control. The same can be applied to DNS, which means that net raw can also be exploited to take  over address or DNS resolution from the host. In order to demo this attack, we will need first to  create a Veth pair in the host's network namespace. It will be called sneaky Veth. Its type will be  Veth, obviously, and its peer name will be sneaky Veth-BR, so we can easily identify which end of  the Veth is to be connected to the Docker zero bridge.  We will then connect the sneaky Veth-BR end of the Veth pair to the Docker zero bridge in the  host's namespace and also set both sides of the Veth pair to the up state.  As we can see, we have both the sides of the Veth pair, one connected to the Docker zero bridge  and both of them in the up state. One thing left to do, let's take a brief look at our routing table,  particularly at the default routes. There's a single one installed, and let's check connectivity to  a well-known address, let's say redhat.com. It works, we are connected to the internet.  Let us now start a T-Mock session and start our DHCP server.  Its name will be DHCP server, let's not invent much here. It will, it does require the net admin  capability as we've been discussing throughout this time. We need to pull it from the correct  registry because we are again using our custom built image that you can find available on the  repo. And let's call the capabilities demo binary with the start DHCP server argument,  bind that the DHCP server to the Ethernet zero interface. Let's advertise this slider, the 10.10.10.2  slash 24. The IP server will be in the same subnet, but with the dot one address. And here in the  router, this will be what we will advertise as a default route. Let's just put some garbage in and  specify a weirdly enough MTU. We're now pulling the container image and we have started the DHCP  server. Now let us look at our interfaces again and start an instance of DH client on our sneaky  Vef, end of the Vef pair. Okay, we can see that we got a reply. We got these logs on the DHCP  server shell. We were offered an IP address. We are now bound to the dot two IP address, which is  the slider that we advertise. We can now see the IP address that we were offered from the server.  And we can also see the MTU that we advertised. Let's take a look at default routes. And we  can see that we have now two new default routes, the one that I called Garovitch before. And let's  take a look. Let's ping the same well-known address and we see that our networking is broken.  Basically, because we have configured a default route with a lower metric.  Now that we have a basic understanding about Linux capabilities, let us focus on the Qvert  architecture. Qvert is a virtualization plugin for Kubernetes and follows its declarative philosophy.  When a set of specifications are posted to the Kubernetes API server, they are then transformed  into an internal object of a specific type. After that, a specific controller that has  previously registered itself as that type's handler will push all the buttons required to  make that request happen. For instance, once a new pod object is seen internally,  the respective controllers perform the necessary actions to bring the pod alive  and match the required state. Qvert delivers three things to provide a new functionality.  The first, additional virtualization-related types known by custom resource definitions  in Kubernetes lingo that will extend the existing API. The second, additional controllers for  cluster-wide logic associated with these new types. And finally, additional demons for no  specific logic associated with the new types. An example task is mounting the KVM device  required by QM into the pods that will hold the user's VM. The Qvert process model is simple.  There is a virtualization controller that specifies the pod where the VM will run. A process called  Virt handler, which operates as Qvert's agent in each of the nodes. And finally, a virtual  launcher process that will encapsulate the libvert and QM processes that will run the user's  virtual machine. On this talk, I will be focusing on Virt handler, again, Qvert's agent, and in  Virt launcher, which encapsulates the user's VM. Virt handler is a trusted component of Qvert.  It runs on a privileged demon set and is responsible for creating and configuring any  required infrastructure for the pod encapsulating the VM. Virt launcher process, on the other hand,  is an untrusted component of Qvert, since it wraps the libvert process that will run third party  workloads. As a result, it must be executed with as little privileges as required.  Throughout the reminder of the presentation, I will be referring to these two processes  by trusted and untrusted components. Keep in mind that the former refers to Virt handler,  while the latter refers to Virt launcher. Configuring networking for a virtual machine  is, in its essence, composed of two different operations. To perform those,  Qvert introduced the bind mechanism concept. Bind mechanism is something that binds the pod  interface to the virtual machine interface. Its first task is how to extend the networking  that Kubernetes delivers for the pods into the virtual machine that will be spawned.  Let's see how that is done.  I am a firm believer that it is easier to understand something when given an example.  So let's imagine we want to create a virtual machine having two different interfaces.  When the pod is created, Kubernetes agent in the node invokes the corresponding CNI plugins,  which are responsible for creating and configuring a network interface in the newly created pod.  If we recall our example, we requested two interfaces. It is part of CNI's responsibility  to create these interfaces in our pods. The bind mechanisms will then kick in and create  all sorts of auxiliary networking infrastructure to extend networking from the pod interface  into what will be our virtual machine. It is important to refer that there is a different  bind mechanism for each interface type. The reference bind mechanism shown in our previous  example will create an input bridge for each interface along with a tap device for the VM,  both of which they will configure accordingly. In the example, both the tap device and the  pod interface is connected to this input bridge, which effectively extends the pod networking  into the future virtual machine. Let's not get into too much detail in this. The main message  here is that each binding mechanism does different things, but all of them will create and configure  different types of networking infrastructure, which, as we've seen before, is all privileged  operations. You must remember that it is actually libvert that instantiates the virtual machine,  an operation that is specified in a domain XML document. Hence, one of the steps involved in  the networking configuration for the virtual machine is translating from the kubvert virtual  machine specification to a libvert DOM XML document. This short example shows parts of both,  focusing on the network-related entities. In the left, you can see that we are requesting a VM  with two interfaces, one of type bridge, another type master rate. The second part of the bind  mechanism will perform this translation from the kubvert specification into a domain XML document,  which will finally be fed to libvert. Libvert, in turn, will transform that XML document into a  sequence of parameters for QM, the architectural element responsible for running the VM.  There are two main takeaways from this architecture section. The first is that kubvert has two types  of components, a trusted component, which is kubvert's agent in each of the nodes, and an  untrusted component that will execute the user's VMs and as such run third-party code.  The other takeaway is that network configuration occurs in two different stages, one that requires  privileged operations, since it is creating and configuring network infrastructure,  and another stage that only features unprivileged operations, because it is mainly translating  from kubvert spec into libvert's DOM XML. Now that we have a clear picture of kubvert's process  architecture and how networking is configured for the virtual machines, we can describe what is our  motivation, problem, and goals. The motivation should be pretty clear by now. We want to reduce  the overall attack surface of the solution, in a way that if a malicious user manages to compromise  the untrusted component of the architecture, the potential damage to the system is not severe.  With this in mind, the problem can be clearly defined. An untrusted component of the architecture  is executing privileged operations, and as such the whole component is being granted more  capabilities than what it should have. Finally, our goals, despite somewhat complex, are straightforward.  We want to remove both the net admin and the net rock capabilities from the untrusted component  of the architecture. In order to allow the aforementioned capabilities to be dropped,  we have first to split the network configuration, making sure that all the privileged operations  are executed in the trusted component. Let's see how that was possible. Originally, the  prepare pod networking stage was executed on the untrusted component. It would create the  input bridge, configure it, and connect it to the pod interface. All of these are privileged  operations, and as such should now be relocated to the trusted component of the architecture.  After that, on a later stage, KubeVertWood, translate the KubeVertVM specification  into a libvert.xml, which would be used to instantiate a running virtual machine.  The solution is to move these privileged operations to the trusted component,  but execute them in the untrusted component's network namespace. To do that, we need to use  the untrusted process, PID, to load its network namespace. This activated rectangle in the right  represents just that. Despite the fact that the prepare pod networking calls are now executed  in the trusted component, it happens in the untrusted process network namespace.  The second stage of VM network configuration will still be executed in the untrusted process,  which is perfectly fine, since it does not require any particular set of capabilities.  You can find in this slide both the KubeVert issue and the pull request where this work was  implemented. In this section of the presentation, I will explain how we have refactored KubeVert's  code in order to remove the NetRoc capability from the untrusted component of the process  architecture. If you remember the demos I've shown previously in the introduction section,  you would remember I showed two tasks that required the NetRoc capability.  The first, open a raw socket. The second, setting the SO bind to device socket option.  While our codebase did not use raw sockets, our DHCP server implementation that dynamically  configures VM IP addresses was using this option. As such, we had to find an alternative way of  filtering data from the center face and limit writing to it. You can find the pull request  that implemented this feature in KubeVert referenced in the slide. To have a proper  understanding of the performary factor, we should first understand what was the initial status quo.  Originally, our in-pot DHCP server was creating a datagram socket and it would afterwards set  the SO bind to device socket option in it, which would then drop all the received traffic from  interfaces other than the one requested. In this short example, let us consider we want to bind to  interface number two. Finally, we would then bind the socket to an address. In this case,  it would be port 67, which is DHCP servers listening port. For this net-raw related refactor,  the idea is to manually filter incoming traffic by the interface index. In order for that happen,  the in-pot DHCP server creates a datagram socket. It would then directly bind the socket to DHCP  server port, which is still port 67. This step would leave the socket listening to all the  interfaces in the pot, and outgoing interface selection would happen based on the kernel  routing table. This is far from what we require. To preserve the whole behavior, we need to listen  only to a particular interface and to send the DHCP replies via that very same interface.  To enable that, we first need to ask the kernel to decorate the incoming packets with the interface  index from which they were received on. Afterwards, whenever we receive a packet view  socket, we can compare the interface from which the packet came from with the interface we are  interested on. All traffic originating from an interface that we are not interested in  will be silently dropped at the application level. Finally, we reply back to the sender,  indicating the interface we want to send the packet to via Linux control messages.  In this next section, I will focus on the work done to enable the net admin capability to be  removed from the untrusted component pod specification. Let's begin by enumerating the  calls that are executed to configure the networks for our virtual machines that do require the net  admin capability. Those are creating and configuring the tap device, creating the input bridge,  connect the tap device to the input bridge. Depending on the bind mechanism, connect the  pod interface to the input bridge and finally configure the input bridge. I am breaking down the  input bridge configuration part in two different tasks for a simple reason. Part of this was already  performed when we split the network configuration to occur in two different stages, configuring the  MAC and MTU of the bridge was already addressed and explained. However, we had left out the small  configuration detail. We had yet to move the sampling, the offloading of the transmit checksum  from the untrusted components to the trusted components. This operation uses an IOCDL that  requires the net admin capability. As such, and once that was done, we were missing two tasks.  The first, create and configure the tap device on behalf of libvert and the second, connect the tap  device to the input bridge, which obviously requires the tap device to have been created previously.  You must remember that it was originally libvert, which is running in the untrusted components  that was creating the tap device and that creating a virtual interface is an operation that does  require the net admin capability. Again, the solution was to move the tap device creation  from the untrusted components to the trusted one and then indicate libvert that it should use a  pre-existing tap device instead of creating a new one. For this last step to happen, we required  libvert to learn a new trick. It had to consume preconfigured tap devices without touching their  configuration. Furthermore, our untrusted component domain XML generation mechanism,  the part of the bind mechanism that translates from Qvert language to libvert language,  had to be updated to use the new API libvert kindly provided for us.  Before this change, Qvert translated the user provided specification to an interface type bridge,  which would task it with creating a new tap device and attach the tap device to the bridge  specified in the target element. libvert already had an ethernet type through which it would create  and configure a new tap device, which would in turn allow the virtual machine to have an  emulated network device. Upon learning about a requirement, the libvert team introduced a new  managed parameter to the target element. If a non-managed already existing target is requested,  no further configuration is performed, which allows the preconfigured tap device to be imported by  libvert without having the net admin capability. There is a tiny nuance to this. This capability is  not needed only if the tap creator and the tap consumer user and group IDs match. This gives  Qvert the ability to consume a preconfigured tap device, but for that to happen, the tap device  must be created and configured beforehand. In this step, we have had akin to what I have shown  previously for the bridge, to create a tap device on the trusted component on behalf of the untrusted  component. For it, we had to use the untrusted process PID to tap into the untrusted component  network namespace, so the tap device is available for when libvert attempts to consume it.  It was somewhat trickier than when creating the bridge since SU lin is frowned upon our plan.  Then, despite creating the tap on the untrusted component network namespace, it is created by  the trusted component. And as such, the tap device internal structure, which is the ton socket,  is inheriting the SU linx context from its creator. In this case, it received the super  privileged type context. When libvert, which runs using the SU linx context of the untrusted component,  attempts to consume the existing tap, SU linx prevents it since the context of the resource,  the ton socket, and the context of the process accessing the resource do not match.  To appease SU linx, we have had to create the tap device with the untrusted component SU linx  context, which is something the super privileged pod has access to. You can find below in this  short code snippet how the privileged pod accesses the SU linx context of another process.  Now that Qvert's trusted component is able to successfully create the tap device in the  network namespace of the untrusted component using the correct SU linx context, all that was left to  do was to update the translation mechanism between the Qvert specification language,  the Qvert Stomach XML to use the new API that libvert has provided for us.  Removing the net admin capability from the untrusted component pod is pending a single  leftover. When libvert is instructed to import a pre-configured, pre-existing tap device via  the manageuplesno parameter in the target element, but also instructed to configure the guest's MTU  via the MTU element, it should not configure the tap site and perform only MTU configuration  of the emulated device present within the virtual machine. Remember that updating the  MTU of virtual interface does require the net admin capability. This is tracked in the bug mentioned  below and was patched upstream mid-January. Qvert will update this version of libvert as soon as  possible. Let's finalize this presentation with a list of next steps to be done.  As I've just mentioned, we need to wait for the bug that I've mentioned previously about the MTU  on libvert to be released, like the fix is already merged upstream and we just need to wait for it  to be available on an upstream distribution. Once we have that, we'll update the libvert version with  this new patched version and with that we can finally remove the cabinet admin from the  verb launcher, which is the untrusted component pot template by reverting this commit right here.  And this is all. Thank you very much for your time. I hope you had some fun and learned something  and I'm now available for some questions. Thank you again.  you  you  you  you  you  you  you  you  you  you  you  you  you  you  you  you  you  you
vai_kubevirt_privilege_0_935.png vai_kubevirt_privilege_936_2747.png vai_kubevirt_privilege_2748_2831.png vai_kubevirt_privilege_2832_4475.png vai_kubevirt_privilege_4476_7595.png vai_kubevirt_privilege_7596_7643.png vai_kubevirt_privilege_7644_7955.png vai_kubevirt_privilege_7956_8123.png vai_kubevirt_privilege_8124_8219.png vai_kubevirt_privilege_8220_8771.png vai_kubevirt_privilege_8772_8855.png vai_kubevirt_privilege_8856_9023.png vai_kubevirt_privilege_9024_11051.png vai_kubevirt_privilege_11052_11603.png vai_kubevirt_privilege_11604_11819.png vai_kubevirt_privilege_11820_11867.png vai_kubevirt_privilege_11868_12083.png vai_kubevirt_privilege_12084_12143.png vai_kubevirt_privilege_12144_12371.png vai_kubevirt_privilege_12372_12587.png vai_kubevirt_privilege_12588_12959.png vai_kubevirt_privilege_12960_13043.png vai_kubevirt_privilege_13044_13091.png vai_kubevirt_privilege_13092_13247.png vai_kubevirt_privilege_13248_13319.png vai_kubevirt_privilege_13320_13811.png vai_kubevirt_privilege_13812_14051.png vai_kubevirt_privilege_14052_14363.png vai_kubevirt_privilege_14364_14687.png vai_kubevirt_privilege_14688_15179.png vai_kubevirt_privilege_15180_15359.png vai_kubevirt_privilege_15360_16331.png vai_kubevirt_privilege_16332_16379.png vai_kubevirt_privilege_16380_17843.png vai_kubevirt_privilege_17844_17903.png vai_kubevirt_privilege_17904_19811.png vai_kubevirt_privilege_19812_20435.png vai_kubevirt_privilege_20436_20663.png vai_kubevirt_privilege_20664_21239.png vai_kubevirt_privilege_21240_21287.png vai_kubevirt_privilege_21288_21491.png vai_kubevirt_privilege_21492_22247.png vai_kubevirt_privilege_22248_22703.png vai_kubevirt_privilege_22704_22847.png vai_kubevirt_privilege_22848_22895.png vai_kubevirt_privilege_22896_24215.png vai_kubevirt_privilege_24216_24431.png vai_kubevirt_privilege_24432_31679.png vai_kubevirt_privilege_31680_32831.png vai_kubevirt_privilege_32832_34247.png vai_kubevirt_privilege_34248_34571.png vai_kubevirt_privilege_34572_35651.png vai_kubevirt_privilege_35652_36095.png vai_kubevirt_privilege_36096_37043.png vai_kubevirt_privilege_37044_38363.png vai_kubevirt_privilege_38364_38687.png vai_kubevirt_privilege_38688_39767.png vai_kubevirt_privilege_39768_42887.png vai_kubevirt_privilege_42888_43139.png vai_kubevirt_privilege_43140_45359.png vai_kubevirt_privilege_45360_46787.png vai_kubevirt_privilege_46788_48383.png vai_kubevirt_privilege_48384_51023.png vai_kubevirt_privilege_51024_51659.png vai_kubevirt_privilege_51660_52907.png vai_kubevirt_privilege_52908_67619.png vai_kubevirt_privilege_67620_67738.png