eBPF, I thought we were friends!
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Fournier, Guillaume
Afchain, Sylvain
Baubeau, Sylvain
Since its first appearance in Kernel 3.18, eBPF (Extended Berkley Packet Filter) has progressively become a key technology for observability in the Linux kernel. Initially dedicated to network monitoring, eBPF can now be used to monitor and trace any kind of kernel space activity. Over the past few years, many vendors have started using eBPF to speed up their services or introduce innovative features. Cilium, Calico, Cloudflare, Netflix and Facebook are leading the charge, showing off new complex networking use cases on a monthly basis. On the security side of things, Google recently contributed the Kernel Runtime Security Instrumentation which opens the door to writing Linux Security Modules with eBPF. In other words, eBPF is the new kid in town and a growing number of companies are running services with eBPF access in production. This leads us to a simple question: how bad can things get if one of those services were to be compromised ? This talk will cover how we leveraged eBPF to implement a full blown rootkit with all the features you would expect: various obfuscation techniques, command and control with remote and persistent access, data theft and exfiltration techniques, Runtime Application Self-Protection evasion techniques, and finally two original container breakout techniques. Simply put, our goal is to demonstrate that rogue kernel modules might have finally found a worthy opponent. We will also detail how to detect such attacks and protect your infrastructure from them, while safely enjoying the exciting capabilities that eBPF has to offer. REFERENCES: Bibliography and documentation links cited in the submission: 1. Russian GRU 85th GTsSS deploys previously undisclosed drovorub malware, NSA / FBI, August 2020-1/-1/0/CSA_DROVORUB_RUSSIAN_GRU_MALWARE_AUG_2020.PDF 2. Kprobe-based Event Tracing, 3. Linux Kernel tracepoints, 4. “bpf_probe_write_user” bpf helper,#L1472 5. Uprobe-based Event Tracing, 6. Cilium’s XDP documentation,#xdp Previous eBPF related talks & projects that helped us build the rootkit: 7. Evil eBPF In-Depth: Practical Abuses of an In-Kernel Bytecode Runtime, Jeff Dileo, DEF CON 27,-27/dc-27-speakers.html#Dileo 8. Process level network security monitoring and enforcement with eBPF, Guillaume Fournier,_level_network_security_monitoring_and_enforcement_with_ebpf/ 9. Runtime Security with eBPF, Sylvain Afchain, Sylvain Baubeau, Guillaume Fournier,_security_with_ebpf/ 10. Monitoring and protecting SSH sessions with eBPF, Guillaume Fournier,_and_protecting_ssh_sessions_with_ebpf/
 Hi, and welcome to our talk. My name is Guillaume Fournier. I'm a security engineer at Datalog, and today, C.LiveChan and I are going to present the rootkits that we implemented using eBPF. If you don't know what eBPF is, don't worry, we are going to present this technology and tell you everything you need to know in order to understand the talk. So let's start with a few words about us. We are the cloud workload security team. We usually use eBPF for good, and our goal is to detect threats at front time. Everything we do is added to the data.org agent, which is an open source project, so feel free to check it out if you are interested. That being said, for DevCon, we decided to use everything we knew about eBPF to build the ultimate rootkits. So as I said before, we are going to start the talk with a brief introduction to eBPF. Then Sylvain will take it over to talk about how we implemented obfuscation and persistent access in our rootkits. After that, I will come back to present the command and control feature along with some data exfiltration examples. And then I will talk about the network discovery and RAS bypass features of the rootkits. And finally, Sylvain will present a few detection and mitigation strategies that you can follow to detect rootkits, such as ours. All right, so let's start with eBPF. eBPF stands for Extended Berkeley Packet Filter. It is a set of technologies that can run sandboxed programs in the Linux kernel without changing the kernel source code or having to load kernel modules. It was initially designed for network packet processing, but many new use cases were progressively added. So for example, you can now use eBPF to do kernel performance tracing along with network security and, you know, runtime security in general. So how does it work? So eBPF is simply a two-step process. First, you have to load your eBPF programs in the Linux kernel, and then you need to tell the kernel how to trigger your programs. So let's have a look at the first step. eBPF programs are reach in C. So it's not exactly C. It's more like a subset of C because of many restrictions that eBPF has to follow. But I'm going to talk about this later. So once you have your C program, you can use LLVM to generate eBPF bytecode, which you can then load into the kernel using the bpfc school. eBPF programs are really made of two different things, eBPF maps and the actual program. So there are a lot of different types of eBPF maps, but all you need to know is that they are the only way to persist data generated by your eBPF programs. Similarly, there are a lot of different program types, and each program type has its own use case. However, regardless of the program type, each program has to go through the same following two phases. So the first one is the verifier step. So I will talk about this later, but for now, just know that this ensures that your program is valid. And second, you have the, I mean, second, your eBPF bytecode will be converted into machine code by a just-in-time compiler. And when those two phases succeed, your program is ready to be executed. Step two is attaching eBPF programs. So in other words, this is when you tell the kernel how to trigger your program. So there are many different program types, and I can't present them all, but I'm just going to talk about four of them. So for example, you have, you can use a k-probe to trigger an eBPF program whenever a specific symbol in the kernel is called. Trace points are similar to k-probes, but the hook points on which they can be attached have to be declared manually by the kernel developers. Those two programs require another syscall in order to be attached, and this syscall is the perf event open syscall. So the other two program types I wanted to talk about are TC classifiers, so it's SCAD-CLS, and XDP programs. So those program types can be used to do packet processing, so whenever some network traffic is detected at the host level or at a specific network interface level. Those two require a net-click command to be attached. And the only thing to remember here is that each program type has its own setup, and thus, Mac require a different level of access. Another very important fact about eBPF is that eBPF maps can be shared between different programs regardless of their program types. All right, so the eBPF verifier. So the verifier is used to ensure that eBPF programs will finish and won't crash. To do so, it's really just a list of rules that the verifier checks, and your program has to comply with those rules. So for example, your program has to finish. It cannot be an infinite loop. So your program has to be a directed acyclic graph. You can have unreachable code, you can have unchecked differences, your stack size is limited, and your overall program size is also limited. And finally, one of the most infamous features of the verifier is its very cryptic outputs. So basically, if your program doesn't pass the verifier step, you will have a huge log of everything that the verifier looked into, and eventually some kind of error telling you what happens. But yeah, basically you are in for a very painful debugging session. Last but not least, eBPF comes with a list of helpers that will help you access data or execute operations that you wouldn't be able to write natively. So for example, you have context helpers, you have map helpers, a lot of things that you wouldn't be able to write in C and that you would need external instrumentation to do. In short, you have about 160 helpers, and most of the heavy lifting of your eBPF programs will be based on those helpers. So that concludes this introduction to eBPF, and I will hand it over to you Silvan, so that you can kick off the presentation of the breakkits. Thank you Guillaume. Before we get into the details, let's see why eBPF is an interesting technology to write a rootkit. First, the safety guarantee brought by eBPF means that a bug in our rootkit cannot crash the host. An error in the execution will not cause any log message to be emitted. The user's effort has no way to know that something actually went wrong and no sees notice the presence of the rootkit. As we saw earlier, the eBPF by code is converted to native code, and the number of instructions is limited, which limits by extension the performance impact that our rootkit can have on the machine that could otherwise be detected by the user. On the commercial side, eBPF is used by an increasing number of vendors, in various use cases, network monitoring security, for instance. With eBPF becoming widespread, the sense of one product being abused to load malicious programs also increases. The safety guarantee we just talked about should not give the security administrators the false feeling of security. There is a lot of activity around eBPF, and each new version of the Linux kernel comes with a new set of eBPF helpers, bringing new capabilities. As we wanted our rootkit to run on a widely used distribution Linux such as Red Hat Enterprise Linux, or the latest Ubuntu LTS, we used a limited number of helpers. Using recent helpers or features like KLSI would have probably made the development of the rootkit easier. One of the primary tasks of a rootkit is to hide itself. What does it mean in our case? eBPF programs are bound to a running process. If this process gets killed, all the attached eBPF programs will be unloaded. For that reason, it is essential that we both hide our program and protect it from being killed. The eBPF programs and maps used by the rootkit should also be hidden, and we should forbid other programs to gain access to them through their file descriptors. Let's see our rootkit in action. Let's start the rootkit. It gives us its PRG. Then we can try a ps command in order to see if we can detect it from the output. We can try using its procfs entry and nothing. We can even try using a sub file or even a relative pass. We still have the same issue, no such file directory. Finally, we can try to send a signal to see what happened, and we get no such process error. The obfuscation capabilities of our rootkit mainly rely on the use of two eBPF helpers. The eBPF program right user helper allows our eBPF program to write into memory of the process that issues a syscall. This can be used, for instance, to alter the data that is returned by a syscall. It's also possible to alter the syscall arguments. There is one caveat with this eC helper. The memory to be modified has to be mapped into the kernel aliaspace. Otherwise, a minor or major page fault will be triggered, causing the bpf property user call to fail. The other bpf helper is to use the bpf override return. This one allows you to change the return value of a syscall and has an interesting property. If this helper is used at this syscall exit, it will simply change the return value of the executed syscall. But if we use it at the entry of the syscall, the execution of the syscall will be completely skipped. It is important to note that this helper can only be used at the entry of the syscall or at the exit. So let's see how the obfuscation of a file actually works. At startup, the rootkit will populate a map with the pass of its pre-proc.pid folder. Now, the user space issues a fragmented syscall, such as state. This syscall usually comes in two forms. One that accepts the pass to the file as a string. Another one that accepts the file descriptor for the file that the user space program must have previously resrived using an open syscall. So let's consider the former. To properly identify the targeted file, the rootkit needs to do an accurate resolution of the pass, as the pass-specific file could be a relative pass. At the entry of the syscall, there is no in-us context to do the resolution. So we need to go deeper in the kernel in our case in dbfs code. So we have the resolution, but at this point, we cannot block the syscall, as we are outside of the allowed hookpoints for the ebpf override return helper. So the only thing that we can do is to change the return value so that the user space believes that the syscall failed. We also need to scrub the content of the switcher that could have been filled by the kernel. Now let's consider the later, the version that accepts the file descriptor. We do the same pass resolution as before, but instead of just pretending to the user space that the syscall failed, we store the file descriptor that the kernel allocated into an ebpf map. If the same process issues the syscall with the same file descriptor, we can, at the syscall entry, return an error, and block the syscall. In this situation, the user has no way to know that the file descriptor exists, and as we control the read the syscall, we can also hide all the references to the file descriptor in procfs. Blocking the syscall that accepts the PID as an argument is trivial using bpf override return, same for loading kernel modules. Now let's see a demo, let's demo the obfuscation of our bpf programs and maps. So we still have the rootkit started, and if we list the maps and the programs thanks to the bpf tool command line, we can't see anything related to the rootkit. Now if we start a binary loading some kprobs and maps, and we list again the programs, we can't see the programs related to the binary, but still nothing related to the rootkit. Then we can even try checking the kprobs and still nothing related to the rootkit. The idea is to hook into the ebpf syscall itself. Two things are required here. We want to avoid the user space to iterate through the IDs of our entities using the bpf.proget.next.id command, and we need to prevent the user space from getting a file descriptor to the program or map from a DID using bpf.proget.fd by ID, basically to prevent ID gassing. Once these operations are endowed by the rootkit, the user space is not able to view or modify our programs and maps. bpf.progrite.user is a crucial part of our rootkit, but it comes with a pitfall. When an ebpf program that makes use of it is loaded, the kernel prints a pretty scary but legitimate message in the kernel ring buffer. So let's see how the rootkit handles this. First, let's check what is the current status of the kernel ring buffer using the command line dmissage. So nothing special here. And we can even start a cat on the device used by the ring buffer. Then we start the rootkit. So we can see that the cat terminated with a pretty legit looking message. We can recheck the kernel ring buffer thanks to the dmissage command line, and we can only see legit looking messages. But a bit more. And then we can stop the rootkit and we can run again the dmissage command line in order to see what was overwritten by the rootkit. So let's see how it works. In this diagram, John Alde is waiting for a new message and is blocked on a readcscall. Two bpf programs are loaded secondarily by the rootkit. The first one, which doesn't use the bpfprogrite user, will hook on a readcscall and will make it return 0 and will write a legit looking message. This will guarantee that the warning message is related to the bpfprogrite user won't be read. Then the second program using bpfprogriteuser is loaded. At this point, the readcscall can be unblocked and we can override the content of the warning messages with legit looking messages. Another important task for the rootkit is to set up a persistent access across reboots, for instance. The rootkit can copy itself to a dedicated place and hide its binary file with the same mechanism that we already saw. For the persistent access to the system, we can use a generic method close to what we described in the obfuscation part. We can replace the content of some critical files read by the root demands such as cronzy or even sshd. Let's see an example targeting sshd and using the read overwrite approach. So the approach here is to append an ssh key to the authorize key files. Only sshd should be impacted, meaning the file will remain the same for the user point of view. And we want to have it available through the command and controls. So let's see this in action. So let's check the authorize keys content first. So we can see that only one key is present. So let's start the connection. And it seems that a password is required. So now we are going to start the rootkit. And we are going to specify that we want to inject an ssh key to the authorize keys file but only for ssh. So we can try a connection again. And it seems to be successful. And now we can check what is the content of the authorize key from the user point of view and nothing changed apparently. Persistent access to an application database can also be set up using an other type of evpf program. Uprobs evpf programs attached to user space function. In addition to being safer and easier to use than ptrace, they offer a valuable advantage. The kernel will automatically set up for us the hooks on every instance of the program. Let's see a uprob demonstration using proso-grace-sql. So first let's try to connect to proso-grace-sql using the word bonsoir as password. This one seems to be the good one. Then trying hello. And this one is redacted. Now we start the rootkit. And we'll get the opposite result. Now the valid password is hello. So the idea here is to hook on the md5 crypt verify function of evpf-grace-sql that checks whether the user provided the right md5 for its four passwords and the challenge sent by the server. Overwriting the expected hash contained in shadowpass with a known value makes the comparison succeed and gives persistent access to the database to the attacker. Now I will hand over to Guillaume that will show you the command and control capabilities of the rootkit. Thank you Sylla. Let's talk about the command and control feature of the rootkit. So what exactly do we want to do? We want to be able to send commands to the rootkit to extract data and to get remote access to the infected hosts. Unfortunately there are a few evpf-related challenges that we need to face in order to implement those features. First you can't initiate a connection with evpf. Second you can't open a port. However evpf can hijack an existing connection. So in order to show up this feature we have set up a very simple infrastructure on AWS. A web app was installed on an ec2 instance and we used a classic load balancer to redirect HTTPS traffic to our instance over HTTP. In other words the tier estimation is done at the load balancer level and HTTPS requests are sent to our instance unencrypted. So our goal is to implement cnc by hijacking the network traffic to our web app. First we need to figure out which evpf-program types we're going to use in order to implement this feature. Although evpf provides a lot of options to choose from we decided to go with two evpf-program types xdp-programs and tc-classifier-programs. So both those programs are usually used to do deep back-end inspection use cases and while xdp only works for ingress, tc works on both ingress and egress traffic. Another difference between the two program types is that xdp-programs can be offloaded to the network interface controller which essentially means that your program will be run before the packet enters any subsystem into the network stack. On the other hand tc-programs have to be attached to a network interface but like much later in the network stack which means that they are triggered later in the kernel. With both programs you can drop, allow and modify your packets and with an xdp-program you can also retransmit a packet. This option is actually super interesting for us because it means that you can essentially receive an answer to a packet even before it reaches the network stack which in other words means that you can do this even before it reaches any kind of network firewall or monitoring on the host. Skipping the network stack also explains why xdp-programs are mainly used for DDoS mitigation and tc-programs are usually used to monitor and secure network access at the pod or container level. So what you need to remember about this slide is that first xdp-programs can be used to hide network traffic from the kernel entirely and tc-programs can be used to excretory data on its way out. Okay first let's see how we used xdp-programs to receive commands with the rootkit. So we implemented a client for the rootkit and this client communicates with the rootkit by sending simple https requests and with a custom route in custom user agent. So after going through the load balancer the request eventually reaches the host and triggers our xdp-programs. Then our program parsed the request, the http-route and understand that this request is not meant for the web app but is meant for us. So after reading the user agent the rootkit executes the request that can end and moves on to the final step. So this final step is probably the most important one. It overrides the entire request with a simple health check request and we do this for two different reasons. First we don't want the malicious request to reach the web app or any kind of user space monitoring tool that might be grinding and that might detect the unusual traffic. And second we want the client to receive an answer in order to know if the request was successful. So as I said before we could also have dropped the the packet entirely but since we're using TCP the load balancer would have retransmitted the packet over and over again until the request times out and this would have generated noise and increase our chances of getting discovered. That said if you have I mean if you were working with a UDP server this would be a totally valid strategy. So let's have a look at how we can send progress credentials remotely. All right so on the left of the screen you can see two different shells. Those shells are connected to the remote infected host on the AWS and on the right this is my local shell and this is the attacker machine. Okay so let's start with trying to log into the progress database using the normal password and again the rootkit is not running yet. So as you can see the Bonsoir password works fine and then let's start the rootkit and restart to log in again and as expected and as you've seen before during a Sylvain's demo it doesn't work. So you have to change into hello and this time it will work. Here you go. Okay so we're going to try to do the same thing but instead of hard coding the new password with the rootkit we're going to define remotely through cnc what the new password should be. So as you can see we have a custom client that will make a request to HTTPS, an HTTPS request to defconn.demoted.doc and then we will provide both the rule and the secret to override the normal secret with. So the request that we'll go through is a very simple one with a custom route and the user agent will contain the new password that will be used at runtime. So as expected we get the 200 okay from the health check which essentially means that we know that the new password now is defconn and not hello anymore. So as you can see hello doesn't work but if I'm changing it to defconn here we go this time it draws work. Okay so this is how we send the command to the rootkits. Now let's see how we can exfiltrate data. So to exfiltrate data the client has to send an initial request to specify what kind of data and what kind of resource we want to exfiltrate. So the xdp part of this process is basically the same as before but this time the xdp program stores the network flow that made the request along with the request resource the requested resource sorry in an ebpf map and the reason why we do so is because when the web app answers the health check we want to be able to detect the packets that are meant to be sent back to the client. So when the HTTP answers answer reaches the tcegress classifier our ebpf program looks up the network flow and overrides the answer with the requested data. Now the question is what kind of data can you exfiltrate with the rootkit right and the answer is well pretty much anything that is accessible to ebpf and the reason for that is as I said before multiple program types can share data through ebpf maps regardless of you know what those programs are supposed to do. So basically you can exfiltrate things like file content environment variables, database dumps, in-marry data if you start looking at the stacks of the programs. Anyway you can pretty much exfiltrate whatever you want. So let's have a look at a simple demo that is we can exfiltrate postgres credentials along with the file content of etcpassword. All right so and again the two shells on the left are the ones connected to the infected host in on the ws and on the right this is my local shell. So the first request that I make here is to list the I mean to do is progress list which basically means place list all the credentials that you have detected so far since the rootkit has started. And as you can see the answer was so the hashtag answer was overridden with the content of a map that we used to store the passwords that we've collected at runtime. And again remember that with postgres you don't need the clear password to login you just need the hash password that is stored in the database. Here you go so now we're going to try to do the same thing to dump the content of etcpassword. So to do so this is a two-step process. The first thing you want to do is tell the rootkit to start looking for this specific file and as soon as a user space process tries to open the file and read the content of the file our rootkit will actually copy the data as it is sent to the user space application and save it into the nubbf map so that it can be retrieved later. So this first request will tell the the the rootkit to start looking for etcpassword. And now let's go back to the host and you know do trigger some kind of pseudo operation so that we can actually I mean so that a user space process tries to open the file. Here you go. And then this time instead of saying add we're going to say get and this will dump the content of the etcpassword. Here you go. All right so the cool thing about this technique is that it applies to any unencrypted network protocols so for example we also implemented it for dns which means that you can actually use it to do dns poofing so the only difference between the normal way of doing this and the dns poofing is that instead of using a tc program to override the answer of the request you will actually switch to see an xdp programs because dns requests are made from the host instead of received by the host. All right so let's move on to our network discovery feature. So I know everybody knows what it is but I have to say it anyway network discovery is the ability to discover machines and services on the network so that you know where you want to go next in the infrastructure and also discovering services is a super important step when you are trying to pivot between hosts because it will tell you what kind of attacks you might want to try. So the wordkit has two different network discovery features. One of them is passive the other one is active and you can control both of them through command and control. So I'm going to get into more details later but basically the only difference between the two is the kind of network scanning you're looking for and also the level of traffic that you are willing to generate on the network. So first let's have a look at the passive option. So the passive option is simply a basic network monitoring tool so it will do pretty much the same thing as any other ebpf based network monitoring tool which means that it will listen for any ingress or egress traffic and then generate a graph from all the collected network flows. It will also show you the amount of data that was sent per network flow and to implement this feature we used our TC and XDP programs. So the TC programs were used to monitor the egress traffic and the XDP programs were used to monitor the ingress traffic. So for this version of the wordkit we are limited to IPv4 and TCP UDP packets that said support for IPv6 and all the other protocols could have been added easily. So the reason why the passive option is pretty cool is that it will not generate any traffic on the network. In other words it is basically impossible to detect that someone is tapping into your network and more specifically the network that is you know that reaches this specific infected host. However this doesn't work for services that do not communicate with the infected host and so in other words the graph will definitely not be complete and that's also why we implemented the active method. So the active method is a simple ARP scanner along with a SIN scanner. So we implemented it using only our XDP programs which means that only I mean that the entire process is done without involving the kernel stack and although this will be a slower process you can use this method to discover hosts and services that are reachable by the infected host but that are not communicating usually with the infected host. And again the rootkit client will generate a nice network graph for you once the scan is complete. So on a technical level this feature of the rootkit is actually quite interesting because as I said before eBPF cannot create a connection from scratch. So in other words we had to figure out a way to generate hundreds of SIN requests while dealing with this limitation of eBPF. So let's see how we solved this problem. So in order to send a SIN request you first need to know the MAC address of the IP that you want to scan. To do so we used the same trick that we've been using so far which is to override the request from the rootkit client. So when our XDP program receives a scan request for a specific IP and a specific port range it will override the entire request with an ARP request for the target IP. And then instead of returning XDP pass which is what we've done so far and also which would send the packets to the network stack our eBPF program returns XDP TX. So what XDP TX does is send the packet out to the network interface controller it came in from. In other words our HTTP packet was transformed into an ARP request and was sent back to and broadcasted back to the entire local network. So eventually the target IP will answer the ARP request and we will be able to store the MAC address of this specific IP in an eBPF map. However during this entire process the TCP packet that was used to send the HTTP request was never acknowledged by the kernel and that is simply because it never made its way to the kernel in the first place which means that the load balancer or the client itself will eventually try to retransmit the packet and when this packet is retransmitted and when it eventually reaches our XDP program we will do the exact same thing but this time instead of because we know the MAC address instead of overriding the request with an ARP request we're going to override the request with a SYN request and more specifically a SYN request with the first port of the provided port range the target IP and and the MAC address of the target IP and assuming that the remote IP or the remote host doesn't have any kind of protection against SYN requests and sorry SYN scanning it will answer either resets or SYN plus AC to this first request so a reset would mean that the port is open and SYN plus AC would mean would indicate that there might be a service running running on the host and this is where the basically the network loop happens and the reason why we were able to generate hundreds of packets without having to I mean while dealing with the limitation of eBPF that is the inability to create packets so whenever we get an answer from a SYN request we override the received packet with another SYN request on the next port and we also switch the IPs switch the MAC addresses and send it back again to the target IP and we do so in a loop until the we go through the entire port range so eventually the clients will try one last time to retransmit the the initial HTTP requests because once again during the network loop we never answered the second retransmit so eventually when this third retransmit reaches our XDP program we will override the request with the usual health check requests so that the 200 okay answer will make its way back to the client after the request was handled by the whatapp in user space all right so let's see it in action so on the right of the screen this is a shell to the infected host on the dws and at the bottom here it's another one and at the top this is my local shell on my machine so the first thing you want to do is to start the rootkit then second is to start dumping the logs of the rootkit so in eBPF you can actually generate logs using a trace pipe obviously you would not want to do this in a real use case for a rootkit but this is a great way of visualizing the the scan as it goes through um yeah so that's why i'm doing this and that's why we'll see what the rootkit does at runtime and then let's make the scan requests so what i'm saying here is please scan the ip 10.0 the 2.3 from port 790 and for the next 20 ports after this one so the first thing you can see is is the request is immediately changed into an arp request and we already got the answer for this arp request so next up when we get a retransmit we will change this into a scene request there you go so the scene request went through and then you can see the loop that happened and we you know like increased port one by one until we reached the final port requested by the the port branch and then now we are waiting for the third retransmit and this retransmit will be the one that we override with the health check requests which means that we will eventually get here go the 200k and in other words you know the answer from the user space web app all right so now what you want to do is retrieve the output of the scan and x-ray the all the network flows that were detected at runtime and here goes so you would say network discovery get and eventually um so it actually requires a lot of different requests because there is a lot of data to x-ray but eventually you will get the entire list of network flows that were captured by by the work kit here you go so you have the all the different individual flows and then more importantly you will have a graph generated for you so this one is the passive sorry active graph so as you can see in a range there you can see the arp request and replies from I mean between those different hosts and then in gray those are the scene requests and the reset answers and in red is the only scene plus ac answer from the remote host all right and then you have also the passive graph which is the one that we saw before okay so now let's move on to our rasp bypass so rasp stands for runtime application self protection self protection um so in a few words a rasp is a new generation of security tools that uses um runtime instrumentation to detect and block application level attacks and more importantly it leverages its insight into the application in order to make more intelligent decisions so simply put it is some kind of advanced input monitoring tool um that can detect malicious parameters and can understand if a malicious input will successfully exploit a weakness um from one of your apps so the textbook example of a rasp is usually a SQL injection so the rasp would implement multiple functions uh instruments sorry multiple functions in the libraries that you use and such as for example the htp server library or the sql library and it will check at runtime that the user controlled parameters in your queries are properly sanitized um if not the rasp will stop the query before it reaches the database and redirect the client to an error page or some kind of error message in other words a rasp and relies on the assumption that the the application runtime has not been compromised which is exactly what we can do with the bpf so just a little disclaimer before I move forward I want to stress the fact that we are playing outside of the boundaries of what a rasp can protect you from and more importantly this bypass does not apply to one specific rasp but to all of them because this is one of the core principles of how a rasp works so let's have a look at how a rasp products um a go web app from a sql injection so let's say that you have a web app with a simple products page and a get parameter um to specify the the the category of the products that you want to see chances are your web app uses the default go database sql interface um so this is a generic interface that you can use to query your database without having to worry about the underlying driver and the database type that you're using more importantly for us since it is such a generic interface this is usually where the the rasp tools um instrument your code um because simply it's much easier to hook at this layer rather than having to hook onto all the underlying drivers so when your request is handled by the web server the query will be formatted with with the provided category parameter and eventually the web app will call the query context function of this sql interface um this is when the rasp checks the the the query and makes sure that everything is normal and if it is the execution will resume its normal flow and the underlying query um driver will be called so in our example we use sqlite so the sqlite driver is called eventually the query makes its way to the database and the answer is sent back to the client however if the rasp detects um that's you know something is wrong or detects some kind of sql injection it will block the query and redirect the client to an error page all right so now let's see what we did to bypass this protection well the answer is actually pretty simple we added a u probe on both the database sql interface and the sqlite driver interface um what this allows us to do is call one of our bpf programs right before the rasp checks the sql query and trigger another one right before the um the the sql query is executed by the database itself and thanks to the bpf prog write user helper um we can override the input parameters of the hooked functions so that the the rasp sees a benign query um and the the the database executes our sql injection and the cool thing about this is that we can even do it conditionally um which means that we can um you know bypass the the rasp only if one specific secret password was added to the beginning of the query perfect so let's move on to the demo so as you can see we have a very simple web app um so it's a shoe shoe's retailer um so you have a lot of different products and um you can filter by category so let's try to do a simple injection um using the get parameter so the the injection will simply be a union select star from user and because the rasp is not running right now um the sql injection should work here you go and if you scroll down this time you will see the users and along with the the passwords perfect so now let's restart the web app with the rasp which is what i've just done and try this again so let's go to the shop and then override the um category parameter perfect and this time the rasp blocked the request because um it detected that um you know someone tried to do a sql injection and the sql injection would actually have succeeded great so now let's start the rootkits by providing the path to the web app and then try to refresh the page so again this should also be blocked by the rootkits by the the rasp because we haven't provided the the secret password for the bypass to work and the secret password is of course defcon and when we say defcon the entire process that i described before will be triggered and as you can see the rasp did not detected so that's all for our rasp bypass i hope you had fun um just before i hand it over to sylvan for the detection and mitigation strategies i wanted to say that unfortunately we won't have time to talk about the container breakouts that are implemented into the rootkits however they have been presented during our black hat talk this year so if you are interested feel free to check it out that being said sylvan take it away so let's talk about detection and mitigation how could we detect and protect ourselves from this type of rootkit we could do this at different levels first if a vendor provided you ebpf programs you should go through another audit an audit and an assessment phase of their programs some changes are bad the code has to be gpl it probably uses some internal kernel symbols so you can ask for it what should we be looking for the program types that are used but also the ebpfl pros used the communication so maps between programs may indicate a punctual risk in the case of that the vendor program is compromised we developed a tool to assist in this auditing phase by inspecting the health files containing the ebpf programs it is able to list the used entities programs and maps and complete a graph of the interactions between them the tool was run on our rootkit with the following result we can identify on the graph that the xdp program has storing information into maps that are also used by some kprobs which correspond to the common uncontrolled capabilities of the rootkits it is also possible to mitigate at runtime the loading of such programs by monitoring calls to the bpfc's call and logging the usage of it it will even it will even be possible to protect the bpfc's call itself by either restricting the call to it to only some trusted processes have the programs inspected before loading and rejected if they contain suspicious patterns or make use of some dangerous helpers we could also compute and validate the signature of the programs before loading them an initiative exists to add this verification logic to the channel itself using tls everywhere for natural traffic also helps mitigating the risk of a rogue ebpf program that intersects natural data now if we were not able to block the loading of such a rootkit how difficult would it be to detect its presence even if it's possible though very challenging to write an almost perfect ebpf rootkit we should concentrate on the action that the rootkit would have to block and lie about the result of such actions for instance our rootkit disables the loading of the kernel modules because such a module would have the ability to list the ebpf programs and the active kprobs now let's imagine that we insert a module that executes a specific action only known to us the blocking of the module by the rootkit would then be easy to detect monitoring the natural traffic traffic adding infrastructure levels could help detecting hijacked connection or strange package transmission our rootkit being far from complete and far from perfect it should be relatively easy to detect it that being said we hope it will bring to light the potential and the risk of such an ebpf based rootkit while presenting some interesting technique the code of both the rootkit and the monitor is available at this addresses please have a look thanks for your attention and have a great conference
 Hi, and welcome to our talk. My name is Guillaume Fournier. I'm a security engineer at Datalog,  and today, C.LiveChan and I are going to present the rootkits that we implemented using eBPF.  If you don't know what eBPF is, don't worry, we are going to present this technology and  tell you everything you need to know in order to understand the talk.  So let's start with a few words about us. We are the cloud workload security team. We usually  use eBPF for good, and our goal is to detect threats at front time. Everything we do is added  to the data.org agent, which is an open source project, so feel free to check it out if you are  interested. That being said, for DevCon, we decided to use everything we knew about eBPF to build  the ultimate rootkits. So as I said before, we are going to start the talk with a brief introduction  to eBPF. Then Sylvain will take it over to talk about how we implemented obfuscation and persistent  access in our rootkits. After that, I will come back to present the command and control feature  along with some data exfiltration examples. And then I will talk about the network discovery  and RAS bypass features of the rootkits. And finally, Sylvain will present a few detection  and mitigation strategies that you can follow to detect rootkits, such as ours.  All right, so let's start with eBPF. eBPF stands for Extended Berkeley Packet Filter.  It is a set of technologies that can run sandboxed programs in the Linux kernel without changing the  kernel source code or having to load kernel modules. It was initially designed for network packet  processing, but many new use cases were progressively added. So for example, you can now use eBPF to do  kernel performance tracing along with network security and, you know, runtime security in general.  So how does it work? So eBPF is simply a two-step process. First, you have to load your eBPF programs  in the Linux kernel, and then you need to tell the kernel how to trigger your programs.  So let's have a look at the first step. eBPF programs are reach in C. So it's not exactly C.  It's more like a subset of C because of many restrictions that eBPF has to follow. But I'm  going to talk about this later. So once you have your C program, you can use LLVM to generate eBPF  bytecode, which you can then load into the kernel using the bpfc school. eBPF programs are really  made of two different things, eBPF maps and the actual program. So there are a lot of different  types of eBPF maps, but all you need to know is that they are the only way to persist data  generated by your eBPF programs. Similarly, there are a lot of different program types,  and each program type has its own use case. However, regardless of the program type,  each program has to go through the same following two phases. So the first one is the verifier step.  So I will talk about this later, but for now, just know that this ensures that your program is  valid. And second, you have the, I mean, second, your eBPF bytecode will be converted into machine  code by a just-in-time compiler. And when those two phases succeed, your program is ready to be  executed. Step two is attaching eBPF programs. So in other words, this is when you tell the kernel  how to trigger your program. So there are many different program types, and I can't present  them all, but I'm just going to talk about four of them. So for example, you have, you can use a  k-probe to trigger an eBPF program whenever a specific symbol in the kernel is called. Trace  points are similar to k-probes, but the hook points on which they can be attached have to be declared  manually by the kernel developers. Those two programs require another syscall in order to  be attached, and this syscall is the perf event open syscall. So the other two program types I  wanted to talk about are TC classifiers, so it's SCAD-CLS, and XDP programs. So those program types  can be used to do packet processing, so whenever some network traffic is detected at the host level  or at a specific network interface level. Those two require a net-click command to be attached.  And the only thing to remember here is that each program type has its own setup,  and thus, Mac require a different level of access. Another very important fact about eBPF is that  eBPF maps can be shared between different programs regardless of their program types.  All right, so the eBPF verifier. So the verifier is used to ensure that eBPF programs will finish  and won't crash. To do so, it's really just a list of rules that the verifier checks,  and your program has to comply with those rules. So for example, your program has to finish. It  cannot be an infinite loop. So your program has to be a directed acyclic graph. You can have  unreachable code, you can have unchecked differences, your stack size is limited, and your overall  program size is also limited. And finally, one of the most infamous features of the verifier is  its very cryptic outputs. So basically, if your program doesn't pass the verifier step, you will  have a huge log of everything that the verifier looked into, and eventually some kind of error  telling you what happens. But yeah, basically you are in for a very painful debugging session.  Last but not least, eBPF comes with a list of helpers that will help you access data  or execute operations that you wouldn't be able to write natively. So for example, you have context  helpers, you have map helpers, a lot of things that you wouldn't be able to write in C and that you  would need external instrumentation to do. In short, you have about 160 helpers, and most of the  heavy lifting of your eBPF programs will be based on those helpers. So that concludes this  introduction to eBPF, and I will hand it over to you Silvan, so that you can kick off the presentation  of the breakkits. Thank you Guillaume. Before we get into the details, let's see why eBPF is an  interesting technology to write a rootkit. First, the safety guarantee brought by eBPF  means that a bug in our rootkit cannot crash the host. An error in the execution will not cause any  log message to be emitted. The user's effort has no way to know that something actually went wrong  and no sees notice the presence of the rootkit. As we saw earlier, the eBPF by code is converted  to native code, and the number of instructions is limited, which limits by extension the performance  impact that our rootkit can have on the machine that could otherwise be detected by the user.  On the commercial side, eBPF is used by an increasing number of vendors, in various use cases,  network monitoring security, for instance. With eBPF becoming widespread, the sense of one product  being abused to load malicious programs also increases. The safety guarantee we just talked  about should not give the security administrators the false feeling of security.  There is a lot of activity around eBPF, and each new version of the Linux kernel comes with a new  set of eBPF helpers, bringing new capabilities. As we wanted our rootkit to run on a widely used  distribution Linux such as Red Hat Enterprise Linux, or the latest Ubuntu LTS, we used a limited  number of helpers. Using recent helpers or features like KLSI would have probably made the development  of the rootkit easier. One of the primary tasks of a rootkit is to hide itself. What does it mean  in our case? eBPF programs are bound to a running process. If this process gets killed, all the  attached eBPF programs will be unloaded. For that reason, it is essential that we both hide our  program and protect it from being killed. The eBPF programs and maps used by the rootkit  should also be hidden, and we should forbid other programs to gain access to them through their  file descriptors. Let's see our rootkit in action. Let's start the rootkit. It gives us its PRG.  Then we can try a ps command in order to see if we can detect it from the output. We can try  using its procfs entry and nothing. We can even try using a sub file or even a relative pass.  We still have the same issue, no such file directory. Finally, we can try to send a signal  to see what happened, and we get no such process error.  The obfuscation capabilities of our rootkit mainly rely on the use of two eBPF helpers.  The eBPF program right user helper allows our eBPF program to write into memory of the process  that issues a syscall. This can be used, for instance, to alter the data that is returned  by a syscall. It's also possible to alter the syscall arguments. There is one caveat with this  eC helper. The memory to be modified has to be mapped into the kernel aliaspace.  Otherwise, a minor or major page fault will be triggered, causing the bpf property user call  to fail. The other bpf helper is to use the bpf override return. This one allows you to  change the return value of a syscall and has an interesting property. If this helper is used  at this syscall exit, it will simply change the return value of the executed syscall. But if we  use it at the entry of the syscall, the execution of the syscall will be completely skipped.  It is important to note that this helper can only be used at the entry of the syscall or at the exit.  So let's see how the obfuscation of a file actually works. At startup, the rootkit will  populate a map with the pass of its pre-proc.pid folder. Now, the user space issues a fragmented  syscall, such as state. This syscall usually comes in two forms. One that accepts the pass to the  file as a string. Another one that accepts the file descriptor for the file that the user space  program must have previously resrived using an open syscall. So let's consider the former.  To properly identify the targeted file, the rootkit needs to do an accurate resolution of the pass,  as the pass-specific file could be a relative pass. At the entry of the syscall, there is  no in-us context to do the resolution. So we need to go deeper in the kernel in our case in dbfs code.  So we have the resolution, but at this point, we cannot block the syscall,  as we are outside of the allowed hookpoints for the ebpf override return helper. So the only  thing that we can do is to change the return value so that the user space believes that the syscall  failed. We also need to scrub the content of the switcher that could have been filled by the kernel.  Now let's consider the later, the version that accepts the file descriptor. We do the same pass  resolution as before, but instead of just pretending to the user space that the syscall failed,  we store the file descriptor that the kernel allocated into an ebpf map. If the same process  issues the syscall with the same file descriptor, we can, at the syscall entry, return an error,  and block the syscall. In this situation, the user has no way to know that the file descriptor exists,  and as we control the read the syscall, we can also hide all the references to the file descriptor  in procfs.  Blocking the syscall that accepts the PID as an argument is trivial using bpf override return,  same for loading kernel modules.  Now let's see a demo, let's demo the obfuscation of our bpf programs and maps.  So we still have the rootkit started, and if we list the maps and the programs  thanks to the bpf tool command line, we can't see anything related to the rootkit.  Now if we start a binary loading some kprobs and maps, and we list again the programs,  we can't see the programs related to the binary, but still nothing related to the rootkit.  Then we can even try checking the kprobs and still nothing related to the rootkit.  The idea is to hook into the ebpf syscall itself. Two things are required here. We want to avoid  the user space to iterate through the IDs of our entities using the bpf.proget.next.id command,  and we need to prevent the user space from getting a file descriptor to the program or map  from a DID using bpf.proget.fd by ID, basically to prevent ID gassing.  Once these operations are endowed by the rootkit, the user space is not able to view or modify  our programs and maps. bpf.progrite.user is a crucial part of our rootkit,  but it comes with a pitfall. When an ebpf program that makes use of it is loaded,  the kernel prints a pretty scary but legitimate message in the kernel ring buffer.  So let's see how the rootkit handles this.  First, let's check what is the current status of the kernel ring buffer using the command line  dmissage. So nothing special here. And we can even start a cat on the device used by the ring buffer.  Then we start the rootkit. So we can see that the cat terminated with a pretty legit looking message.  We can recheck the kernel ring buffer thanks to the dmissage command line,  and we can only see legit looking messages.  But a bit more. And then we can stop the rootkit and we can run again the dmissage command line  in order to see what was overwritten by the rootkit.  So let's see how it works. In this diagram,  John Alde is waiting for a new message and is blocked on a readcscall. Two bpf programs are  loaded secondarily by the rootkit. The first one, which doesn't use the bpfprogrite user,  will hook on a readcscall and will make it return 0 and will write a legit looking message.  This will guarantee that the warning message is related to the bpfprogrite user won't be read.  Then the second program using bpfprogriteuser is loaded. At this point, the readcscall can be  unblocked and we can override the content of the warning messages with legit looking messages.  Another important task for the rootkit is to set up a persistent access across reboots, for instance.  The rootkit can copy itself to a dedicated place and hide its binary file with the same mechanism  that we already saw. For the persistent access to the system, we can use a generic method close  to what we described in the obfuscation part. We can replace the content of some critical files  read by the root demands such as cronzy or even sshd. Let's see an example targeting sshd  and using the read overwrite approach. So the approach here is to append an ssh key to the  authorize key files. Only sshd should be impacted, meaning the file will remain the same for the  user point of view. And we want to have it available through the command and controls.  So let's see this in action.  So let's check the authorize keys content first.  So we can see that only one key is present. So let's start the connection. And it seems that  a password is required. So now we are going to start the rootkit.  And we are going to specify that we want to inject an ssh key to the authorize keys  file but only for ssh. So we can try a connection again.  And it seems to be successful. And now we can check what is the content of the  authorize key from the user point of view and nothing changed apparently.  Persistent access to an application database can also be set up using an other type of  evpf program. Uprobs evpf programs attached to user space function. In addition to being safer  and easier to use than ptrace, they offer a valuable advantage. The kernel will automatically  set up for us the hooks on every instance of the program. Let's see a uprob demonstration using  proso-grace-sql. So first let's try to connect to proso-grace-sql using the word bonsoir as  password. This one seems to be the good one. Then trying hello. And this one is redacted.  Now we start the rootkit. And we'll get the opposite result. Now the valid password is hello.  So the idea here is to hook on the md5 crypt verify function of evpf-grace-sql that checks  whether the user provided the right md5 for its four passwords and the challenge sent by the server.  Overwriting the expected hash contained in shadowpass with a known value makes the comparison  succeed and gives persistent access to the database to the attacker. Now I will hand over  to Guillaume that will show you the command and control capabilities of the rootkit.  Thank you Sylla. Let's talk about the command and control feature of the rootkit. So what exactly  do we want to do? We want to be able to send commands to the rootkit to extract data and to  get remote access to the infected hosts. Unfortunately there are a few evpf-related challenges that we  need to face in order to implement those features. First you can't initiate a connection with evpf.  Second you can't open a port. However evpf can hijack an existing connection. So in order to  show up this feature we have set up a very simple infrastructure on AWS. A web app was installed on  an ec2 instance and we used a classic load balancer to redirect HTTPS traffic to our instance over  HTTP. In other words the tier estimation is done at the load balancer level and HTTPS requests  are sent to our instance unencrypted. So our goal is to implement cnc by hijacking the network  traffic to our web app. First we need to figure out which evpf-program types we're going to use  in order to implement this feature. Although evpf provides a lot of options to choose from  we decided to go with two evpf-program types xdp-programs and tc-classifier-programs.  So both those programs are usually used to do deep back-end inspection use cases and while xdp  only works for ingress, tc works on both ingress and egress traffic. Another difference between  the two program types is that xdp-programs can be offloaded to the network interface controller  which essentially means that your program will be run before the packet enters any subsystem  into the network stack. On the other hand tc-programs have to be attached to a network  interface but like much later in the network stack which means that they are triggered later in the  kernel. With both programs you can drop, allow and modify your packets and with an xdp-program  you can also retransmit a packet. This option is actually super interesting for us because it  means that you can essentially receive an answer to a packet even before it reaches the network  stack which in other words means that you can do this even before it reaches any kind of network  firewall or monitoring on the host. Skipping the network stack also explains why xdp-programs are  mainly used for DDoS mitigation and tc-programs are usually used to monitor and secure network  access at the pod or container level. So what you need to remember about this slide is that first  xdp-programs can be used to hide network traffic from the kernel entirely and tc-programs can be  used to excretory data on its way out. Okay first let's see how we used xdp-programs to receive  commands with the rootkit. So we implemented a client for the rootkit and this client communicates  with the rootkit by sending simple https requests and with a custom route in custom user agent.  So after going through the load balancer the request eventually reaches the host and triggers our  xdp-programs. Then our program parsed the request, the http-route and understand that this request  is not meant for the web app but is meant for us. So after reading the user agent the rootkit  executes the request that can end and moves on to the final step. So this final step is probably  the most important one. It overrides the entire request with a simple health check request  and we do this for two different reasons. First we don't want the malicious request to reach the  web app or any kind of user space monitoring tool that might be grinding and that might detect  the unusual traffic. And second we want the client to receive an answer in order to know if the request  was successful. So as I said before we could also have dropped the the packet entirely but  since we're using TCP the load balancer would have retransmitted the packet over and over again  until the request times out and this would have generated noise and increase our chances of getting  discovered. That said if you have I mean if you were working with a UDP server this would be a  totally valid strategy. So let's have a look at how we can send progress credentials remotely.  All right so on the left of the screen you can see two different shells. Those shells are connected  to the remote infected host on the AWS and on the right this is my local shell and this is the  attacker machine. Okay so let's start with trying to log into the progress database using the normal  password and again the rootkit is not running yet. So as you can see the Bonsoir password works fine  and then let's start the rootkit and restart to log in again and as expected and as you've seen  before during a Sylvain's demo it doesn't work. So you have to change into hello and this time it  will work. Here you go. Okay so we're going to try to do the same thing but instead of hard coding  the new password with the rootkit we're going to define remotely through cnc what the new password  should be. So as you can see we have a custom client that will make a request to HTTPS, an HTTPS  request to defconn.demoted.doc and then we will provide both the rule and the secret to override  the normal secret with. So the request that we'll go through is a very simple one with a custom route  and the user agent will contain the new password that will be used at runtime. So as expected we  get the 200 okay from the health check which essentially means that we know that the new  password now is defconn and not hello anymore. So as you can see hello doesn't work but if I'm  changing it to defconn here we go this time it draws work. Okay so this is how we send the  command to the rootkits. Now let's see how we can exfiltrate data. So to exfiltrate data the  client has to send an initial request to specify what kind of data and what kind of resource we  want to exfiltrate. So the xdp part of this process is basically the same as before but this time  the xdp program stores the network flow that made the request along with the request resource  the requested resource sorry in an ebpf map and the reason why we do so is because when the web  app answers the health check we want to be able to detect the packets that are meant to be sent  back to the client. So when the HTTP answers answer reaches the tcegress classifier our ebpf  program looks up the network flow and overrides the answer with the requested data. Now the question  is what kind of data can you exfiltrate with the rootkit right and the answer is well pretty much  anything that is accessible to ebpf and the reason for that is as I said before multiple  program types can share data through ebpf maps regardless of you know what those programs are  supposed to do. So basically you can exfiltrate things like file content environment variables,  database dumps, in-marry data if you start looking at the stacks of the programs. Anyway you can  pretty much exfiltrate whatever you want. So let's have a look at a simple demo that is we can  exfiltrate postgres credentials along with the file content of etcpassword.  All right so and again the two shells on the left are the ones connected to the infected host  in on the ws and on the right this is my local shell. So the first request that I make here is  to list the I mean to do is progress list which basically means place list all the credentials  that you have detected so far since the rootkit has started. And as you can see the answer was  so the hashtag answer was overridden with the content of a map that we used to store the  passwords that we've collected at runtime. And again remember that with postgres you don't need  the clear password to login you just need the hash password that is stored in the database.  Here you go so now we're going to try to do the same thing to dump the content of etcpassword.  So to do so this is a two-step process. The first thing you want to do is tell the rootkit to  start looking for this specific file and as soon as a user space process tries to open the file  and read the content of the file our rootkit will actually copy the data as it is sent to the  user space application and save it into the nubbf map so that it can be retrieved later.  So this first request will tell the the the rootkit to start looking for etcpassword.  And now let's go back to the host and you know do trigger some kind of pseudo operation so that  we can actually I mean so that a user space process tries to open the file. Here you go.  And then this time instead of saying add we're going to say get and this will dump the content  of the etcpassword. Here you go. All right so the cool thing about this technique is that it applies  to any unencrypted network protocols so for example we also implemented it for dns which means that  you can actually use it to do dns poofing so the only difference between the normal way of doing  this and the dns poofing is that instead of using a tc program to override the answer of the request  you will actually switch to see an xdp programs because dns requests are made from the host instead  of received by the host. All right so let's move on to our network discovery feature. So I know  everybody knows what it is but I have to say it anyway network discovery is the ability to discover  machines and services on the network so that you know where you want to go next in the infrastructure  and also discovering services is a super important step when you are trying to pivot  between hosts because it will tell you what kind of attacks you might want to try. So the  wordkit has two different network discovery features. One of them is passive the other one  is active and you can control both of them through command and control. So I'm going to get into  more details later but basically the only difference between the two is the kind of network scanning  you're looking for and also the level of traffic that you are willing to generate on the network.  So first let's have a look at the passive option. So the passive option is simply a basic  network monitoring tool so it will do pretty much the same thing as any other ebpf based  network monitoring tool which means that it will listen for any ingress or egress traffic and then  generate a graph from all the collected network flows. It will also show you the amount of data  that was sent per network flow and to implement this feature we used our TC and XDP programs. So  the TC programs were used to monitor the egress traffic and the XDP programs were used to monitor  the ingress traffic. So for this version of the wordkit we are limited to IPv4 and TCP UDP packets  that said support for IPv6 and all the other protocols could have been added easily.  So the reason why the passive option is pretty cool is that it will not generate any traffic  on the network. In other words it is basically impossible to detect that someone is tapping  into your network and more specifically the network that is you know that reaches this specific  infected host. However this doesn't work for services that do not communicate with the infected  host and so in other words the graph will definitely not be complete and that's also why we implemented  the active method. So the active method is a simple ARP scanner along with a SIN scanner. So we  implemented it using only our XDP programs which means that only I mean that the entire process  is done without involving the kernel stack and although this will be a slower process you can use  this method to discover hosts and services that are reachable by the infected host but that are  not communicating usually with the infected host. And again the rootkit client will generate a nice  network graph for you once the scan is complete. So on a technical level this feature of the rootkit  is actually quite interesting because as I said before eBPF cannot create a connection from scratch.  So in other words we had to figure out a way to generate hundreds of SIN requests while dealing  with this limitation of eBPF. So let's see how we solved this problem.  So in order to send a SIN request you first need to know the MAC address of the IP that you want to  scan. To do so we used the same trick that we've been using so far which is to override the request  from the rootkit client. So when our XDP program receives a scan request for a specific IP and a  specific port range it will override the entire request with an ARP request for the target IP.  And then instead of returning XDP pass which is what we've done so far and also which would send  the packets to the network stack our eBPF program returns XDP TX. So what XDP TX does is send the  packet out to the network interface controller it came in from. In other words our HTTP packet was  transformed into an ARP request and was sent back to and broadcasted back to the entire local network.  So eventually the target IP will answer the ARP request and we will be able to store the MAC  address of this specific IP in an eBPF map. However during this entire process the TCP packet that  was used to send the HTTP request was never acknowledged by the kernel and that is simply  because it never made its way to the kernel in the first place which means that the load balancer  or the client itself will eventually try to retransmit the packet and when this packet is  retransmitted and when it eventually reaches our XDP program we will do the exact same thing  but this time instead of because we know the MAC address instead of overriding the request with  an ARP request we're going to override the request with a SYN request and more specifically a SYN request  with the first port of the provided port range the target IP and and the MAC address of the target IP  and assuming that the remote IP or the remote host doesn't have any kind of protection against  SYN requests and sorry SYN scanning it will answer either resets or SYN plus AC to this first request  so a reset would mean that the port is open and SYN plus AC would mean would indicate that there  might be a service running running on the host and this is where the basically the network loop  happens and the reason why we were able to generate hundreds of packets without having to I mean  while dealing with the limitation of eBPF that is the inability to create packets  so whenever we get an answer from a SYN request we override the received packet with another SYN  request on the next port and we also switch the IPs switch the MAC addresses and send it back  again to the target IP and we do so in a loop until the we go through the entire port range  so eventually the clients will try one last time to retransmit the the initial HTTP requests  because once again during the network loop we never answered the second retransmit  so eventually when this third retransmit reaches our XDP program we will override the request with  the usual health check requests so that the 200 okay answer will make its way back to the client  after the request was handled by the whatapp in user space all right so let's see it in action  so on the right of the screen this is a shell to the infected host on the dws and at the bottom  here it's another one and at the top this is my local shell on my machine so the first thing you  want to do is to start the rootkit then second is to start dumping the logs of the rootkit so  in eBPF you can actually generate logs using a trace pipe obviously you would not want to do this  in a real use case for a rootkit but this is a great way of visualizing the the scan as it goes  through um yeah so that's why i'm doing this and that's why we'll see what the rootkit does at  runtime and then let's make the scan requests so what i'm saying here is please scan the ip 10.0  the 2.3 from port 790 and for the next 20 ports after this one so the first thing you can see is  is the request is immediately changed into an arp request and we already got the answer for this  arp request so next up when we get a retransmit we will change this into a scene request  there you go so the scene request went through and then you can see the loop that happened and we  you know like increased port one by one until we reached the final port requested by the the port  branch and then now we are waiting for the third retransmit and this retransmit will be the one  that we override with the health check requests which means that we will eventually get here  go the 200k and in other words you know the answer from the user space web app  all right so now what you want to do is retrieve the output of the scan and x-ray  the all the network flows that were detected at runtime and here goes so you would say network  discovery get and eventually um so it actually requires a lot of different requests because  there is a lot of data to x-ray but eventually you will get the entire list of network flows that  were captured by by the work kit here you go so you have the all the different individual flows  and then more importantly you will have a graph generated for you so this one is the passive sorry  active graph so as you can see in a range there you can see the arp request and replies from  I mean between those different hosts and then in gray those are the scene requests and the reset  answers and in red is the only scene plus ac answer from the remote host all right and then  you have also the passive graph which is the one that we saw before  okay so now let's move on to our rasp bypass so rasp stands for runtime application self protection  self protection um so in a few words a rasp is a new generation of security tools that uses  um runtime instrumentation to detect and block application level attacks  and more importantly it leverages its insight into the application in order to make more  intelligent decisions so simply put it is some kind of advanced input monitoring tool  um that can detect malicious parameters and can understand if a malicious input will successfully  exploit a weakness um from one of your apps so the textbook example of a rasp is usually a  SQL injection so the rasp would implement multiple functions uh instruments sorry multiple functions  in the libraries that you use and such as for example the htp server library or the sql library  and it will check at runtime that the user controlled parameters in your queries are  properly sanitized um if not the rasp will stop the query before it reaches the database  and redirect the client to an error page or some kind of error message in other words a rasp  and relies on the assumption that the the application runtime has not been compromised  which is exactly what we can do with the bpf so just a little disclaimer before I move forward  I want to stress the fact that we are playing outside of the boundaries of what a rasp can  protect you from and more importantly this bypass does not apply to one specific rasp  but to all of them because this is one of the core principles of how a rasp works  so let's have a look at how a rasp products um a go web app from a sql injection  so let's say that you have a web app with a simple products page and a get parameter um to  specify the the the category of the products that you want to see chances are your web app uses the  default go database sql interface um so this is a generic interface that you can use to query  your database without having to worry about the underlying driver and the database type that you're  using more importantly for us since it is such a generic interface this is usually where the  the rasp tools um instrument your code um because simply it's much easier to hook at this layer  rather than having to hook onto all the underlying drivers so when your request is handled by the  web server the query will be formatted with with the provided category parameter and eventually  the web app will call the query context function of this sql interface um this is when the rasp  checks the the the query and makes sure that everything is normal and if it is the execution  will resume its normal flow and the underlying query um driver will be called so in our example  we use sqlite so the sqlite driver is called eventually the query makes its way to the database  and the answer is sent back to the client however if the rasp detects um that's you know something  is wrong or detects some kind of sql injection it will block the query and redirect the client to  an error page all right so now let's see what we did to bypass this protection well the answer is  actually pretty simple we added a u probe on both the database sql interface and the sqlite  driver interface um what this allows us to do is call one of our bpf programs right before the  rasp checks the sql query and trigger another one right before the um the the sql query is executed  by the database itself and thanks to the bpf prog write user helper um we can override the input  parameters of the hooked functions so that the the rasp sees a benign query um and the the the  database executes our sql injection and the cool thing about this is that we can even do it  conditionally um which means that we can um you know bypass the the rasp only if one specific  secret password was added to the beginning of the query perfect so let's move on to the demo  so as you can see we have a very simple web app um so it's a shoe shoe's retailer um so you have  a lot of different products and um you can filter by category so let's try to do a simple injection  um using the get parameter so the the injection will simply be a union select star from user  and because the rasp is not running right now um the sql injection should work here you go  and if you scroll down this time you will see the users and along with the the passwords perfect  so now let's restart the web app with the rasp which is what i've just done and try this again  so let's go to the shop and then override the um category parameter  perfect and this time the rasp blocked the request because um it detected that um you know  someone tried to do a sql injection and the sql injection would actually have succeeded  great so now let's start the rootkits  by providing the path to the web app and then try to refresh the page so again this should  also be blocked by the rootkits by the the rasp because we haven't provided the the secret password  for the bypass to work and the secret password is of course defcon and when we say defcon the  entire process that i described before will be triggered and as you can see the rasp did not  detected so that's all for our rasp bypass i hope you had fun um just before i hand it over to sylvan  for the detection and mitigation strategies i wanted to say that unfortunately we won't have  time to talk about the container breakouts that are implemented into the rootkits however  they have been presented during our black hat talk this year so if you are interested feel free to  check it out that being said sylvan take it away so let's talk about detection and mitigation  how could we detect and protect ourselves from this type of rootkit we could do this at different  levels first if a vendor provided you ebpf programs you should go through another audit  an audit and an assessment phase of their programs some changes are bad the code has to be gpl it  probably uses some internal kernel symbols so you can ask for it what should we be looking for the  program types that are used but also the ebpfl pros used the communication so maps between programs  may indicate a punctual risk in the case of that the vendor program is compromised we developed a  tool to assist in this auditing phase by inspecting the health files containing the ebpf programs it  is able to list the used entities programs and maps and complete a graph of the interactions between  them the tool was run on our rootkit with the following result we can identify on the graph  that the xdp program has storing information into maps that are also used by some kprobs  which correspond to the common uncontrolled capabilities of the rootkits  it is also possible to mitigate at runtime the loading of such programs by monitoring calls  to the bpfc's call and logging the usage of it it will even it will even be possible to protect  the bpfc's call itself by either restricting the call to it to only some trusted processes  have the programs inspected before loading and rejected if they contain suspicious patterns  or make use of some dangerous helpers we could also compute and validate the signature of the  programs before loading them an initiative exists to add this verification logic to the channel  itself using tls everywhere for natural traffic also helps mitigating the risk of a rogue ebpf  program that intersects natural data now if we were not able to block the loading of such a  rootkit how difficult would it be to detect its presence even if it's possible though very challenging  to write an almost perfect ebpf rootkit we should concentrate on the action that the rootkit would  have to block and lie about the result of such actions for instance our rootkit disables the  loading of the kernel modules because such a module would have the ability to list the ebpf  programs and the active kprobs now let's imagine that we insert a module that executes a specific  action only known to us the blocking of the module by the rootkit would then be easy to detect  monitoring the natural traffic traffic adding infrastructure levels could help detecting  hijacked connection or strange package transmission our rootkit being far from complete and far from  perfect it should be relatively easy to detect it that being said we hope it will bring to light  the potential and the risk of such an ebpf based rootkit while presenting some interesting technique  the code of both the rootkit and the monitor is available at this addresses please have a look  thanks for your attention and have a great conference
DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_0_347.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_348_563.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_564_1931.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_1932_2015.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_2016_2999.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_3000_4859.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_4860_8423.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_8424_9179.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_9180_9635.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_9636_12767.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_12768_13487.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_13488_13751.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_13752_15407.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_15408_17219.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_17220_18107.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_18108_18611.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_18612_19331.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_19332_20747.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_20748_21467.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_21468_22043.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_22044_22115.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_22116_23063.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_23064_24431.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_24432_27815.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_27816_28571.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_28572_32243.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_32244_34847.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_34848_37259.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_37260_37451.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_37452_38831.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_38832_39839.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_39840_40487.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_40488_42431.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_42432_42491.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_42492_42599.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_42600_43475.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_43476_44711.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_44712_48443.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_48444_54071.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_54072_55175.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_55176_57683.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_57684_57791.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_57792_58319.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_58320_58535.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_58536_61415.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_61416_63923.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_63924_65387.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_65388_65579.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_65580_65639.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_65640_66083.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66084_66143.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66144_66191.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66192_66455.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66456_66563.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66564_66647.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66648_66743.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_66744_67067.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67068_67223.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67224_67523.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67524_67571.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67572_67619.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67620_67775.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_67776_68519.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_68520_69875.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_69876_70163.png DEF_CON_29_-_Guillaume_Fournier__Sylvain_Afchain___5zixNDolLrg_70164_73342.png