Phantom Attack: Evading System Call Monitoring
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Guo, Rex
Zeng, Junyuan
Phantom attack is a collection of attacks that evade Linux system call monitoring. A user mode program does not need any special privileges or capabilities to reliably evade system call monitoring using Phantom attack by exploiting insecure tracing implementations. After adversaries gain an initial foothold on a Linux system, they typically perform post-exploitation activities such as reconnaissance, execution, privilege escalation, persistence, etc. It is extremely difficult if not impossible to perform any non-trivial adversarial activities without using Linux system calls. Security monitoring solutions on Linux endpoints typically offer system call monitoring to effectively detect attacks. Modern solutions often use either ebpf-based programs or kernel modules to monitor system calls through tracepoint and/or kprobe. Any adversary operations including abnormal and/or suspicious system calls reveal additional information to the defenders and can trigger detection alerts. We will explain the generic nature of the vulnerabilities exploited by Phantom attack. We will demonstrate Phantom attack on two popular open source Linux system call monitoring solutions Falco (Sysdig) and Tracee (Aquasecurity). We will also explain the differences between Phantom v1 and v2 attacks. Finally, we will discuss mitigations for Phantom attack and secure tracing in the broader context beyond system call tracing. REFERENCES:-bin/cvename.cgi?name=CVE-2021-33505-20/Thursday/us-20-Lee-Exploiting-Kernel-Races-Through-Taming-Thread-Interleaving.pdf?v=MIJL5wLUtKE-advisories/khobe-earthquake.pdf
 Hello everyone. Today we're going to talk about Fenton attack evading system call monitor. My name is Rex. My name is Julia. So imagine an attacker compromise your Linux infrastructure. So the attacker first compromise a web app through a web app remote execution and then it launches a reverse shell. Then it discovered a vulnerability on the system. He can activate privileges using the pseudo vulnerability. CV 2021 31 56. Then he's looking for secrets on the system. So he read the Etsy shadow file and then he discover additional lateral movement opportunities by reading the SSH process environmental variable. Then he later move to the second machine using SSH hijack. As he is celebrating this moment he discovered that his reverse shell connection is gone and it doesn't take him for too long to discover that he his IP is completely blocked. Now let's take a look at the other side of the story. While all this is happening our security engineer has received a bunch of slack messages for the alerts generated by his latest cloud workflow protection software. The reason that the software can discover all these activities precisely is because it monitors the system calls and other process related data. So for example when the attacker launches the reverse shell there will be a connect system call and there may be additional system calls depends on the reverse shell that he uses. This is similar for the other activities and through this talk we are going to use open as system call as an example. So let's take a look at how one can use system calls and other process information to detect a attacker reads at the shadow. So here's an example rule. The rule is trying to detect untrusted programs reads the at the shadow. Let me explain what the rule means. It detects that there is an open at or open system call with the recommission and the file name is equal to at the shadow and also the program is not in the allow list that allows to read the at the shadow. So from this rule it should be very obvious that the ability to precisely monitor system calls and other system call related data is critical for the detection of this attack. The agenda of this talk is we will talk about system call monitoring in more detail and then we'll talk about the two open source system call monitoring project that we analyze and then we'll talk about the first vulnerability the top-toe issue which we use Benton v1 attack to exploit and then we'll talk about the second vulnerability a semantic confusion issue which we use Benton v2 attack to exploit and finally we'll conclude the talk with takeaways. With that I will hand over to Junyuan to talk about the system call monitor. Yeah so as Rex mentioned system call monitoring is very important to detect threats. So what is this call monitoring? We can define it as a technique to verify whether the application system call conferred to the rule that's specified the program behaviors at runtime. Here is the graph showing how system call monitoring works. When application system call is involved system call code path is executed if there are any hooks in a code path the attached program will be called to collect system call data. For example system call arguments the data are sent to user space monitoring agent. The monitoring agent will check if application system calls conferred to the user defined rules if not the case it may generate errors. So typically at least two steps should be include for system call monitoring. One step is call system call interception which is to get notified if target system call are invoked. In order to intercept system call you can use trace point or roaches point. Both of them are static hook placed in a kernel code. Roaches point are EBPF alternatives to standard trace point. It's faster because it provides slow access to the arguments without processing. For this call interception the kernel provides two roaches points six enter and six exit. This roaches point can be called this function trace system enter and trace system exit respectively. The first arguments of this function are PT register structure saving user registers in user kernel mode switching in it includes system call arguments. The second parameters is six call number. If any progress attached to this roaches point it will be executed with the same arguments as functions. Trace point has low overhead but only provides static interception. Different from trace point Kpro or Kretempro provides dynamic hook in a kernel. Using it we can register the progress on kernel instructions for example on system call code pass. When the instructions are executed it will trigger register programs. Kpro can be inserted on almost any instructions in a kernel. However Kretempro can only be inserted in function entry and exit. Kpro provides dynamic hook but is slow compared to trace point and you need to know exactly how data is placed on a stack or registers in order to read system call data. You can also use LDP log trick to intercept system call but it's not working in all cases for example application is statically compiled. Key trace system call provides another way to intercept system call however the overhead is high. The second step of system call monitoring is called system call data collection which is to collect system call data. For example system call arguments are via notified by system call events. The program used to collect system call data is called tracing program. For example you can use the tracing program to collect system call arguments. As we mentioned before tracing programs can attach to different hooks like trace point, raw trace point, Kpro or Kretempro. When the hooks fire tracing programs are called to collect data. There are different ways to implement tracing programs to collect system call data. You can use Linux native mechanisms like ftrace or perf events. You can also implement the trace programs in kernel module or ebpf programs which allow the execution of user code in a kernel. The open source project FACO and Tracee both use the similar techniques to monitor Cisco. FACO is originally created by SysTick. It's one of the two security and compliance projects and the only endpoint security monitoring project in CNCF in Cubating's projects. It has two 3.9 kth github stars. It actually consumed kernel events and enriched them with information from cloud native stack like Linux containers and so on. FACO supports both ebpf and kernel module implementation for the tracing program. Tracee on the other hand is originally created by Alquac security. It has 1.1 kth github stars. It's basically a runtime security and forensics tools based on ebpf. So unfortunately the open source projects or other projects using the similar techniques are vulnerable to be attacked during Cisco monitoring. The first vulnerability is time of check, time of use. During time of check, tracing programs collect Cisco data. During time of use, Cisco data used by kernel is different from what tracing program check. Let's take open as a example. The second parameter is called find name which is the pointer pointing to user space buffer between time of check and time of use. This pointer is vulnerable to be modified from user space. So we will introduce phantom B1 attack that can get the problem, exploit the top-tier issue. The second vulnerability is semantic confusion. It means kernel interprets data differently from tracing programs. For example, symbolic link is interpreted differently by the kernel and tracing programs. We will also introduce phantom B2 attack that can exploit semantic confusion. We will also demonstrate FACL is vulnerable to both phantom B1 and B2 attack while tracee is only vulnerable to phantom B1. In order to understand tactile, we used over that system call for example. We use kernel version 5.4.0 but regardless of the kernel version, if the monitoring software use trace point in this way, the tactile vulnerability will exist. To simplify, we only show the code that is related to the attack. When open-app system call is invoked in applications, Cisco handler will execute trace 6 enter function with two arguments. As we mentioned before, if any tracing program attached to this enter trace point, the program will be executed. After that, Cisco handler look up Cisco table and jump to open-app system call to open the file. Before return to applications, the handler will call trace 6 access with exactly the same arguments as trace 6 enter. So similarly, if there are any tracing programs attached to 6 access trace point, the program will be executed. As you mentioned before, the second arguments of open-app system call is find name pointer pointing to the user space memory. The find name is passed to do system open function and the kernel pop it to kernel buffer temp using get name function. After that, kernel use the kernel buffer to call internal function do file open to open the file. This is time of use for system call arguments by the kernel. If we divide the open-app system call co-pass into two parts based on the get name function, we get true sub co-pass cp1 co-pass 1 and cp2 co-pass 2. In cp1, the find name pointer hasn't been copied to the kernel buffer. In this case, no matter where we place the host in cp1, the attached tracing program will have to read user space buffer in order to get the find name. This is vulnerable to be changed in user space attacker. For example, if we attach tracing program to 6 enter trace point or to do 6 open using kpro, during time of check, the tracing program will have to read the user space buffer to get the find name. In cp2, user space memory has been copied to the kernel buffer making it not vulnerable to be changed from user space. For example, if we attach tracing program to the entry of do file open function using kpro, the tracing program can read the kernel buffer tab to get the find name. That kernel buffer is not vulnerable to be changed for tactile attack. However, if the hook are placed improperly in cp2, tactile is still possible. For example, if tracing programs attach to 6 axis trace point, it will read the user space buffer to get the find name. As we mentioned before, we use kernel version 5.4.0, but regardless of the kernel version, if the monitoring software use the trace point in this way, this vulnerability will exist. FACO is vulnerable to tactile and the vulnerability is tracked by CVE 2021 3355 with the scores 7.3. In particular, the vulnerability exists for FACO with the version older than 0.29.0 or open source sysdick. It also affects some commercial versions based on the open source agent. This was confirmed by the open source maintainer. Please contact the vendor for the versions. The reason why FACO is vulnerable to tactile vulnerability is that they use sysenter and sysaxis trace point to intercept system calls. In that case, user space pointers are read directly by FACO trace program in both kernel module and EBPF programs in permutations. In order to demonstrate the generality of FACO, we evaluate the syscalls in FACO rules. Please note that we only consider system calls that includes user pointers as arguments like open-s system call. And we found that FACO is vulnerable to monitoring most of syscalls that we evaluate except exact V-syscalls because FACO doesn't read user pointers for exact V-syscalls arguments directly. Instead, it reads the data from kernel data structure. So we evaluate trace 0.4.0 and we found that it's vulnerable to many system calls like connect system call. One thing I need to mention is there's no CVE given because the trace C also mentions that FACO attacks on syscalls records or tracer is a well-known issue and trace C is no exception and also agrees on the fact that there's no CVE or normal findings and therefore we could talk about it publicly. I will let the audience to interpret. So I will hand over to Rex to explain and demo the Phantom V1 attack. Alright, so the high-level idea to exploit the top two issue is fairly simple. So first of all we want to trigger the target system call with malicious arguments and we'll let the kernel to read the malicious argument and perform the intended malicious action for us. After the kernel reads it, we will override the data structure pointed by the user space argument pointer with benign data and at sys exit, the tracing program reads the data structure pointed by the user space pointer and checks the benign data against the rule and therefore you will not fire. Although the high-level plan is so simple there's a few technical challenges that we need to overcome. First one is when does the kernel threat read it and how can we synchronize the override with the kernel threat read? Are the reading windows big enough for the system calls that we're going to attack? And how do we ensure the tracing program gets the overwritten copy all the time? So before I dive into the step-by-step exploitation, there are a few primitives that we use in the exploit which I want to talk about. First one is user for fd system call. The system call is designed in a way that the user a user thread can handle page fault. But page fault is traditionally handled by the kernel. So what's the initial design intention for this? This was designed for memory externalization. In the case where you're running a distributed program, you can run compute node and memory nodes. When the compute node needs a particular memory that doesn't exist in the compute node, the kernel triggers the page fault and the user space for handler, it's going to reach out to the memory node to get the desirable memory. On the other hand, if the compute node has memory pressure, it will send those memory pages back to the memory node. One very important fact about user for fd is that once the kernel thread triggers the page fault, the kernel thread is completely paused and wait for the user space program to respond. As some of you may already be aware of, this has been used quite a bit in exploiting kernel risk condition bugs. The other two primitive I want to talk about, one is interrupts and the other one is scheduling. So an interrupt notifies the processor with an event that requires immediate attention. It will diverge the program control flow to an interrupt handler. Let's look at the picture on the right side. So we have two cores and we have two tasks running on each core corresponding it. On core zero task A issues a system call A and then the control flow transferred with kernel thread to handle the system call. While it is running, the user thread on the core one triggers an interrupt and the way it triggers the interrupt is indirect interrupt using system call. Once the interrupt is triggered, its core zero will execute the interrupt handler and after the interrupt is handled, it will return back to the system call routine which handles the system call. So there are different ways to indirectly trigger system call, indirectly trigger interrupt using system call. One way to do is to trigger a hardware interrupt. So this can happen when a program issues a connect system call. The CPU that is dedicated to handle networking interrupt will get interrupted. Another way to trigger this interrupt is called inter processor interrupt. This can be done by issuing an end protect system call. So once the end protect system call is issued, the memory page permission is changed and therefore all the CPUs that are caching those memory permissions need to be updated with the right memory permission. The scheduling primitive that we use, one is a sex scheduler. This will change the scheduling priority of a particular task. This is optional in the exploit because for system calls with longer top-top windows such as networking system calls, we find that it's not needed to reliably exploit the top-top issue. But with system calls related to files, the top-top window is typically smaller. And with the capability, we can 100% reliably exploit the top-top issue. And then the second primitive we use is set affinity which will pin the task to a particular CPU. Okay, so let me talk about the step-by-step exploitation in detail. Initially we need to do some setup. So we set up three main thread, a user for FD thread. The user for FD thread can run any CPU and also an override thread. So the main thread will pin to a CPU 3. The choice of 3 here is because we run our experiment, one of our experiment, on a four-core system and CPU 3 is used to handle the networking interrupt. But if you're using IPI interrupt, it can be any CPU. And then the main thread will map a memory page A. The page is not allocated and it will register the user for FD thread to handle the page file generated for this page. On the override thread side, we pin it to a different CPU because we want to reduce the interference between the override thread and the main thread. And then the override thread will just block on conditional mutex once it started. After this setup, the main thread will trigger a system call, in this case open-at, and it will specify the file name argument to point to page A. Now keep in mind that page A at this point is not allocated and so the kernel thread will trigger a page file. Once the page file is triggered, the user for FD thread will write page A with the malicious file name and then it will release the conditional mutex. And then it will issue a auto-system call to return execution back to the kernel. Now once the conditional mutex is released, the override thread will start running and it will first write the behind file name. In the last stage of this tech, once execution return back to the kernel, the kernel will use copy from user and you will get the malicious path name. This is a time-up check and it keeps executing until, as this exit, the trees program will read the register and the reference the file name value again. This is a time-up check. So this is the talk-to window we have and let's see how we can use the override thread to make this override successful. So after it writes a benign name, we issue a memory consistency update using CPU instructions. As the memory consistency update takes place, we want to increase the talk-to window such that we have enough time to update the value for all the CPUs. So what we do is that it will issue a interrupt using system calls and this will increase the talk-to window size. And then the memory consistency update applies. So the file name now points to the benign file name and then the treesing program will read out the benign file name and check it against the rule which will not fire anything. Now let's take a look at the demo. Okay so in our first demo we're going to attack SysDig for using connect system call as an example. So what we'll do is we will run SysDig and ask it to monitor connect system call and just filter the data on our attack program. And then on another window we will use TCP dump to monitor all traffic going to port agent. And then we'll run our attack program. The attack program it will connect to 1.1.1.1.1 but then you will override this IP address with IP address from linking.com. Okay so we can see that the TCP dump report traffic actually going to 1.1.1.1.1. This is what happens on the wire. But SysDig will report that okay this program attack connect is actually connecting to 13.107.42.14 which is the IP address on by linking.com. And therefore we showed that we can write this with arbitrary value. Okay so now we'll show a second demo. In this demo we'll show an attack on SysDig on the open add system call. Now our demo setup is slightly different. So for open add the top-to-window is small. So if we don't use a capsis nice sometimes the attack will fail. But if we assign capsis nice we can achieve 100% reliable exploitation. Keep in mind that because the override thread has the highest scheduling priority so sometimes it can write the file name faster than the kernel read. So when that happens the system call will get the benign file name and the SysDig will also get the benign file name. So there's no harm being done and no other being fired as well. And therefore we run the experiment 15 times using our script and each time we check what is what is a SysCover rate and what is SysDig read. And that will compute the result based on each observation. So what we'll do is we'll print out the number of success attack the number of attack that has no harm and also the success rate. Okay so as you can see here in the 15 experiment we succeed three times no failure and 12 times there's no harm. And therefore the success rate is 100%. Okay so lastly we'll show a demo on 3C as well and here we will attack connect call in 3C. The attack setup is very similar to what we have for the SysDig. Again 3C will monitor the connect system call and we'll just filter the data on our attack program reported by 3C. And then we have TCP done monitoring report 80 and then we run the attack. As you can see that the actual traffic going on the wire is going to 1.1.1.1 but 3C reported that the traffic going is going to 13.107.42.14 which is overwritten by us. Okay so this is I just want to reiterate that the capsis nice is optional it depends on the system call that you're attacking and for networking system calls typically this is non-needed and for file base typically this is needed to achieve a hundred percent reliable exploitation. And then I'll talk about the second attack the semantic confusion. The idea of the attack is fairly simple. The kernel and the tracing program can interpret data differently. So we use file link as an example. When the kernel reads a link it will try to resolve the link and read the actual file. But when the tracing program read a link it will just take the link as the argument and use that to track in the loop. So file call is vulnerable to this semantic confusion attack because it didn't resolve the link in the system call. There's no CV given because they mentioned that same link and same link add link and link add are all monitored by FALCO. But practically detection team need to track all these same link add or link link add to all these file-based rules if the attacker is using those. And tracing is not vulnerable to this attack because they use a mitigation in RSM hook which Junya will talk more in a later slide. So I just want to quickly show the example of Fenton v2 attack in SysTik. Remember this rule that we talked about at the very beginning and see that the file name actually we checked whether the file name is at C shadow. So in order to exploit this we can create a file link temp shadow pointing to at C shadow and then the tracing program will read the same link at C shadow. And then the system call open add sees the temp shadow and track against the rule. It doesn't match at C shadow and therefore the rule is bypassed. So with that I will hand over back to Junya to talk about mitigations. Yeah so for mitigations there are two basically two approach for the Fenton attacks. So one is to detect the potential exploit is happening. This was proposed and partially implemented by its faculty. It has includes in faculty release in version 0.29.0. Basically it's trying to detect the following behaviors used by the exploits. For example it detects the previous use of user 4fd system call. This was implemented and detect a user registers memory address range and also detect a user copying continuous memory trunk into a user of D register range and so on. The second way to mitigate Fenton attacks is to read the data that actually used by the system call or the kernel. In order to do that you can hook LSM functions to get those system call data that is actually used. LSM hooks function is a list of checkpoints that are placed in a kernel before operations happens on a kernel objects. So here is the table showing a list of the LSM hooks used by tracee version 4 and the second column shows the system call that was that is protected by the LSM hooks from Fenton attacks. You can also read those data that's used by the kernel from kernel data structure. For example in order to read the arguments of exactly tracing programs can read it from the M structure from the kernel. I will hand over back to Rex to conclude. Okay so basically in this talk we show that Fenton attack is generic and it exploit the fact that kernel and tracing program can read data at different times. This is exploited by Fenton v1 and we also show that the Fenton attack can exploit it the fact that kernel and tracing program can interpret data differently. This is exploited by Fenton v2. We demonstrated that kernel raw tracepoint on system call are not ideal for secure tracing and for other tracing implementations such as Kpro it could also be vulnerable if it is not implemented properly. For mitigation one can use detection for abnormal usage on user 4 fd or to ensure that the kernel and the secure tracing program reads the same data and interpret the data in the same way. If you're interested in discuss further feel free to contact me on Twitter and we'll share the GitHub link at Twitter as well. So before we conclude we also want to thank all these people during our research Chris and Joe for the discussion on the eBPF and kernel tracing and top-toe also you on top-toe and lastly we really appreciate the Falco open source team they are very professional handling the issue we have really good discussions on there. Thank you everyone.
 Hello everyone. Today we're going to talk about Fenton attack evading system call  monitor. My name is Rex. My name is Julia.  So imagine an attacker compromise your Linux infrastructure. So the attacker  first compromise a web app through a web app remote execution and then it  launches a reverse shell. Then it discovered a vulnerability on the system.  He can activate privileges using the pseudo vulnerability. CV 2021 31 56. Then  he's looking for secrets on the system. So he read the Etsy shadow file and then  he discover additional lateral movement opportunities by reading the SSH  process environmental variable. Then he later move to the second machine using  SSH hijack. As he is celebrating this moment he discovered that his reverse  shell connection is gone and it doesn't take him for too long to discover that  he his IP is completely blocked. Now let's take a look at the other side of  the story. While all this is happening our security engineer has received a bunch  of slack messages for the alerts generated by his latest cloud workflow  protection software. The reason that the software can discover all these  activities precisely is because it monitors the system calls and other  process related data. So for example when the attacker launches the reverse  shell there will be a connect system call and there may be additional system  calls depends on the reverse shell that he uses. This is similar for the other  activities and through this talk we are going to use open as system call as an  example. So let's take a look at how one can use system calls and other process  information to detect a attacker reads at the shadow. So here's an example rule.  The rule is trying to detect untrusted programs reads the at the shadow. Let me  explain what the rule means. It detects that there is an open at or open system  call with the recommission and the file name is equal to at the shadow and also  the program is not in the allow list that allows to read the at the shadow.  So from this rule it should be very obvious that the ability to precisely  monitor system calls and other system call related data is critical for the  detection of this attack. The agenda of this talk is we will talk about system  call monitoring in more detail and then we'll talk about the two open source  system call monitoring project that we analyze and then we'll talk about the  first vulnerability the top-toe issue which we use Benton v1 attack to  exploit and then we'll talk about the second vulnerability a semantic  confusion issue which we use Benton v2 attack to exploit and finally we'll  conclude the talk with takeaways. With that I will hand over to Junyuan to talk  about the system call monitor. Yeah so as Rex mentioned system call monitoring is  very important to detect threats. So what is this call monitoring? We can define  it as a technique to verify whether the application system call  conferred to the rule that's specified the program behaviors at runtime. Here is  the graph showing how system call monitoring works. When application system  call is involved system call code path is executed if there are any hooks in a  code path the attached program will be called to collect system call data. For  example system call arguments the data are sent to user space monitoring agent.  The monitoring agent will check if application system calls conferred to  the user defined rules if not the case it may generate errors. So typically at least  two steps should be include for system call monitoring. One step is call  system call interception which is to get notified if target system call are  invoked. In order to intercept system call you can use trace point or roaches  point. Both of them are static hook placed in a kernel code. Roaches point are  EBPF alternatives to standard trace point. It's faster because it provides  slow access to the arguments without processing. For this call interception  the kernel provides two roaches points six enter and six exit. This roaches  point can be called this function trace system enter and trace system exit  respectively. The first arguments of this function are PT register structure  saving user registers in user kernel mode switching in it includes system call  arguments. The second parameters is six call number. If any progress attached to  this roaches point it will be executed with the same arguments as  functions. Trace point has low overhead but only provides static interception.  Different from trace point Kpro or Kretempro provides dynamic hook in a  kernel. Using it we can register the progress on kernel instructions for  example on system call code pass. When the instructions are executed it will  trigger register programs. Kpro can be inserted on almost any instructions in  a kernel. However Kretempro can only be inserted in function entry and exit.  Kpro provides dynamic hook but is slow compared to trace point and you need to  know exactly how data is placed on a stack or registers in order to read  system call data. You can also use LDP log trick to intercept system call but  it's not working in all cases for example application is statically  compiled. Key trace system call provides another way to intercept system call  however the overhead is high. The second step of system call monitoring  is called system call data collection which is to collect system call data.  For example system call arguments are via notified by system call  events. The program used to collect system call data is called tracing  program. For example you can use the tracing program to collect system call  arguments. As we mentioned before tracing programs can attach to different  hooks like trace point, raw trace point, Kpro or Kretempro. When the hooks fire  tracing programs are called to collect data. There are different ways to  implement tracing programs to collect system call data. You can use Linux native  mechanisms like ftrace or perf events. You can also implement the trace  programs in kernel module or ebpf programs which allow the execution of  user code in a kernel. The open source project FACO and Tracee both use the  similar techniques to monitor Cisco. FACO is originally created by SysTick. It's  one of the two security and compliance projects and the only endpoint security  monitoring project in CNCF in Cubating's projects. It has two 3.9 kth  github stars. It actually consumed kernel events and enriched them with  information from cloud native stack like Linux containers and so on. FACO  supports both ebpf and kernel module implementation for the tracing program.  Tracee on the other hand is originally created by Alquac security. It has 1.1  kth github stars. It's basically a runtime security and forensics tools  based on ebpf. So unfortunately the open source projects or other projects  using the similar techniques are vulnerable to be attacked during Cisco  monitoring. The first vulnerability is time of check, time of use. During time of  check, tracing programs collect Cisco data. During time of use, Cisco data used by  kernel is different from what tracing program check. Let's take open as a  example. The second parameter is called find name which is the pointer pointing  to user space buffer between time of check and time of use. This pointer is  vulnerable to be modified from user space. So we will introduce phantom B1  attack that can get the problem, exploit the top-tier issue. The second  vulnerability is semantic confusion. It means kernel interprets data differently  from tracing programs. For example, symbolic link is interpreted differently  by the kernel and tracing programs. We will also introduce phantom B2 attack  that can exploit semantic confusion. We will also demonstrate FACL is  vulnerable to both phantom B1 and B2 attack while tracee is only  vulnerable to phantom B1. In order to understand tactile, we used over that  system call for example. We use kernel version 5.4.0 but regardless of the  kernel version, if the monitoring software use trace point in this way,  the tactile vulnerability will exist. To simplify, we only show the code that is  related to the attack. When open-app system call is invoked in applications,  Cisco handler will execute trace 6 enter function with two arguments. As we  mentioned before, if any tracing program attached to this enter trace point, the  program will be executed. After that, Cisco handler look up Cisco table and  jump to open-app system call to open the file. Before return to applications, the  handler will call trace 6 access with exactly the same arguments as trace 6  enter. So similarly, if there are any tracing programs attached to 6 access  trace point, the program will be executed. As you mentioned before, the second  arguments of open-app system call is find name pointer pointing to the user  space memory. The find name is passed to do system open function and the kernel  pop it to kernel buffer temp using get name function. After that, kernel use the  kernel buffer to call internal function do file open to open the file. This is  time of use for system call arguments by the kernel. If we divide the open-app  system call co-pass into two parts based on the get name function, we get true  sub co-pass cp1 co-pass 1 and cp2 co-pass 2. In cp1, the find name pointer  hasn't been copied to the kernel buffer. In this case, no matter where we place  the host in cp1, the attached tracing program will have to read user space  buffer in order to get the find name. This is vulnerable to be changed in user  space attacker. For example, if we attach tracing program to 6 enter  trace point or to do 6 open using kpro, during time of check, the tracing  program will have to read the user space buffer to get the find name. In cp2, user  space memory has been copied to the kernel buffer making it not vulnerable  to be changed from user space. For example, if we attach tracing program to the  entry of do file open function using kpro, the tracing program can read the  kernel buffer tab to get the find name. That kernel buffer is not vulnerable to be  changed for tactile attack. However, if the hook are placed improperly in cp2,  tactile is still possible. For example, if tracing programs attach to 6 axis  trace point, it will read the user space buffer to get the find name. As we  mentioned before, we use kernel version 5.4.0, but regardless of the kernel  version, if the monitoring software use the trace point in this way, this  vulnerability will exist. FACO is vulnerable to tactile and the  vulnerability is tracked by CVE 2021 3355 with the scores 7.3. In particular, the  vulnerability exists for FACO with the version older than 0.29.0 or open  source sysdick. It also affects some commercial versions based on the open  source agent. This was confirmed by the open source maintainer. Please contact  the vendor for the versions. The reason why FACO is vulnerable to tactile  vulnerability is that they use sysenter and sysaxis trace point to intercept  system calls. In that case, user space pointers are read directly by FACO  trace program in both kernel module and EBPF programs in permutations. In order  to demonstrate the generality of FACO, we evaluate the syscalls in FACO rules.  Please note that we only consider system calls that includes user pointers as  arguments like open-s system call. And we found that FACO is vulnerable to  monitoring most of syscalls that we evaluate except exact V-syscalls  because FACO doesn't read user pointers for exact V-syscalls arguments  directly. Instead, it reads the data from kernel data structure. So we evaluate  trace 0.4.0 and we found that it's vulnerable to many system calls like  connect system call. One thing I need to mention is there's no CVE given because  the trace C also mentions that FACO attacks on syscalls records or tracer  is a well-known issue and trace C is no exception and also agrees on the fact  that there's no CVE or normal findings and therefore we could talk about it  publicly. I will let the audience to interpret. So I will hand over to Rex  to explain and demo the Phantom V1 attack. Alright, so the high-level idea to  exploit the top two issue is fairly simple. So first of all we want to trigger  the target system call with malicious arguments and we'll let the kernel to  read the malicious argument and perform the intended malicious action for us.  After the kernel reads it, we will override the data structure pointed by  the user space argument pointer with benign data and at sys exit, the tracing  program reads the data structure pointed by the user space pointer and checks  the benign data against the rule and therefore you will not fire. Although the  high-level plan is so simple there's a few technical challenges that we need to  overcome. First one is when does the kernel threat read it and how can we  synchronize the override with the kernel threat read? Are the reading windows  big enough for the system calls that we're going to attack? And how do we  ensure the tracing program gets the overwritten copy all the time? So before I  dive into the step-by-step exploitation, there are a few primitives that we use  in the exploit which I want to talk about. First one is user for fd system call.  The system call is designed in a way that the user a user thread can handle  page fault. But page fault is traditionally handled by the kernel. So  what's the initial design intention for this? This was designed for memory  externalization. In the case where you're running a distributed program, you can  run compute node and memory nodes. When the compute node needs a particular  memory that doesn't exist in the compute node, the kernel triggers the page  fault and the user space for handler, it's going to reach out to the memory node  to get the desirable memory. On the other hand, if the compute node has memory  pressure, it will send those memory pages back to the memory node. One very  important fact about user for fd is that once the kernel thread triggers the  page fault, the kernel thread is completely paused and wait for the user  space program to respond. As some of you may already be aware of, this has been  used quite a bit in exploiting kernel risk condition bugs.  The other two primitive I want to talk about, one is interrupts and the other  one is scheduling. So an interrupt notifies the processor with an event  that requires immediate attention. It will diverge the program control flow to  an interrupt handler. Let's look at the picture on the right side. So we have two  cores and we have two tasks running on each core corresponding it. On core zero  task A issues a system call A and then the control flow transferred with kernel  thread to handle the system call. While it is running, the user thread on the core  one triggers an interrupt and the way it triggers the interrupt is indirect  interrupt using system call. Once the interrupt is triggered, its core zero  will execute the interrupt handler and after the interrupt is handled, it will  return back to the system call routine which handles the system call. So there  are different ways to indirectly trigger system call, indirectly trigger  interrupt using system call. One way to do is to trigger a hardware interrupt. So  this can happen when a program issues a connect system call. The CPU that is  dedicated to handle networking interrupt will get interrupted. Another way to  trigger this interrupt is called inter processor interrupt. This can be done by  issuing an end protect system call. So once the end protect system call is  issued, the memory page permission is changed and therefore all the CPUs  that are caching those memory permissions need to be updated with the  right memory permission. The scheduling primitive that we use, one is  a sex scheduler. This will change the scheduling priority of a particular task.  This is optional in the exploit because for system calls with longer top-top  windows such as networking system calls, we find that it's not needed to reliably  exploit the top-top issue. But with system calls related to files, the top-top  window is typically smaller. And with the capability, we can 100% reliably  exploit the top-top issue. And then the second primitive we use is set affinity  which will pin the task to a particular CPU. Okay, so let me talk about the step-by-step  exploitation in detail. Initially we need to do some setup. So we set up three  main thread, a user for FD thread. The user for FD thread can run any CPU and  also an override thread. So the main thread will pin to a CPU 3. The choice  of 3 here is because we run our experiment, one of our experiment, on a  four-core system and CPU 3 is used to handle the networking interrupt. But if  you're using IPI interrupt, it can be any CPU. And then the main thread will map a  memory page A. The page is not allocated and it will register the user for FD  thread to handle the page file generated for this page. On the override thread  side, we pin it to a different CPU because we want to reduce the  interference between the override thread and the main thread. And then the  override thread will just block on conditional mutex once it started. After  this setup, the main thread will trigger a system call, in this case open-at, and  it will specify the file name argument to point to page A. Now keep in mind that  page A at this point is not allocated and so the kernel thread will trigger a  page file. Once the page file is triggered, the user for FD thread will  write page A with the malicious file name and then it will release the  conditional mutex. And then it will issue a auto-system call to return  execution back to the kernel. Now once the conditional mutex is released, the  override thread will start running and it will first write the behind file name. In  the last stage of this tech, once execution return back to the kernel, the  kernel will use copy from user and you will get the malicious path name. This is  a time-up check and it keeps executing until, as this exit, the trees program will  read the register and the reference the file name value again. This is a time-up  check. So this is the talk-to window we have and let's see how we can use the  override thread to make this override successful. So after it writes a  benign name, we issue a memory consistency update using CPU instructions.  As the memory consistency update takes place, we want to increase the talk-to  window such that we have enough time to update the value for all the CPUs. So what  we do is that it will issue a interrupt using system calls and this will  increase the talk-to window size. And then the memory consistency update  applies. So the file name now points to the benign file name and then the  treesing program will read out the benign file name and check it against the  rule which will not fire anything. Now let's take a look at the demo. Okay so in  our first demo we're going to attack SysDig for using connect system call as  an example. So what we'll do is we will run SysDig and ask it to monitor  connect system call and just filter the data on our attack program. And then on  another window we will use TCP dump to monitor all traffic going to port agent.  And then we'll run our attack program. The attack program it will connect to  1.1.1.1.1 but then you will override this IP address with IP address from  linking.com. Okay so we can see that the TCP dump report traffic actually going  to 1.1.1.1.1. This is what happens on the wire.  But SysDig will report that okay this program attack connect is actually  connecting to 13.107.42.14 which is the IP address on by linking.com. And  therefore we showed that we can write this with arbitrary value.  Okay so now we'll show a second demo. In this demo we'll show an attack on SysDig  on the open add system call. Now our demo setup is slightly different. So for open  add the top-to-window is small. So if we don't use a capsis nice sometimes the  attack will fail. But if we assign capsis nice we can achieve 100% reliable  exploitation. Keep in mind that because the override thread has the highest  scheduling priority so sometimes it can write the file name faster than the  kernel read. So when that happens the system call will get the benign file  name and the SysDig will also get the benign file name. So there's no harm being  done and no other being fired as well. And therefore we run the experiment 15  times using our script and each time we check what is what is a SysCover rate and  what is SysDig read. And that will compute the result based on each  observation. So what we'll do is we'll print out the number of success attack  the number of attack that has no harm and also the success rate.  Okay so as you can see here in the 15 experiment we succeed three times no  failure and 12 times there's no harm. And therefore the success rate is 100%.  Okay so lastly we'll show a demo on 3C as well and here we will attack connect  call in 3C. The attack setup is very similar to what we have for the SysDig.  Again 3C will monitor the connect system call and we'll just filter the data on  our attack program reported by 3C. And then we have TCP done monitoring report 80  and then we run the attack. As you can see that the actual traffic going on the  wire is going to 1.1.1.1 but 3C reported that the traffic going is going to  13.107.42.14 which is overwritten by us. Okay so this is I just want to  reiterate that the capsis nice is optional it depends on the system call  that you're attacking and for networking system calls typically this is  non-needed and for file base typically this is needed to achieve a hundred  percent reliable exploitation. And then I'll talk about the second attack the  semantic confusion. The idea of the attack is fairly simple. The kernel and  the tracing program can interpret data differently. So we use file link as an  example. When the kernel reads a link it will try to resolve the link and read  the actual file. But when the tracing program read a link it will just take  the link as the argument and use that to track in the loop. So file call is  vulnerable to this semantic confusion attack because it didn't resolve the link  in the system call. There's no CV given because they mentioned that same link  and same link add link and link add are all monitored by FALCO. But practically  detection team need to track all these same link add or link link add to all  these file-based rules if the attacker is using those. And tracing is not  vulnerable to this attack because they use a mitigation in RSM hook which  Junya will talk more in a later slide. So I just want to quickly show the  example of Fenton v2 attack in SysTik. Remember this rule that we talked about  at the very beginning and see that the file name actually we checked whether the  file name is at C shadow. So in order to exploit this we can create a  file link temp shadow pointing to at C shadow and then the tracing program will  read the same link at C shadow. And then the system call open add sees the temp  shadow and track against the rule. It doesn't match at C shadow and therefore  the rule is bypassed. So with that I will hand over back to Junya to talk  about mitigations. Yeah so for mitigations there are two basically two  approach for the Fenton attacks. So one is to detect the potential exploit is  happening. This was proposed and partially implemented by its faculty. It has  includes in faculty release in version 0.29.0. Basically it's trying to detect  the following behaviors used by the exploits. For example it detects the  previous use of user 4fd system call. This was implemented and detect a user  registers memory address range and also detect a user copying continuous memory  trunk into a user of D register range and so on. The second way to mitigate Fenton  attacks is to read the data that actually used by the system call or the  kernel. In order to do that you can hook LSM functions to get those system call  data that is actually used. LSM hooks function is a list of checkpoints that  are placed in a kernel before operations happens on a kernel objects. So here is  the table showing a list of the LSM hooks used by tracee version 4 and the  second column shows the system call that was that is protected by the LSM  hooks from Fenton attacks. You can also read those data that's used by the kernel  from kernel data structure. For example in order to read the arguments of  exactly tracing programs can read it from the M structure from the kernel. I will  hand over back to Rex to conclude.  Okay so basically in this talk we show that Fenton attack is generic and it  exploit the fact that kernel and tracing program can read data at different  times. This is exploited by Fenton v1 and we also show that the Fenton attack can  exploit it the fact that kernel and tracing program can interpret data  differently. This is exploited by Fenton v2. We demonstrated that kernel raw  tracepoint on system call are not ideal for secure tracing and for other  tracing implementations such as Kpro it could also be vulnerable if it is not  implemented properly. For mitigation one can use detection for abnormal  usage on user 4 fd or to ensure that the kernel and the secure tracing  program reads the same data and interpret the data in the same way. If  you're interested in discuss further feel free to contact me on Twitter and  we'll share the GitHub link at Twitter as well. So before we conclude we also want  to thank all these people during our research Chris and Joe for the discussion  on the eBPF and kernel tracing and top-toe also you on top-toe and lastly we  really appreciate the Falco open source team they are very professional  handling the issue we have really good discussions on there. Thank you everyone.
DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_0_479.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_480_599.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_600_863.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_864_935.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_936_1007.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1008_1163.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1164_1211.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1212_1283.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1284_1415.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1416_1703.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1704_1883.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_1884_2291.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_2292_4043.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4044_4259.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4260_4355.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4356_4499.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4500_4751.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4752_4871.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_4872_5135.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_5136_5183.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_5184_5231.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_5232_5399.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_5400_6431.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_6432_7991.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_7992_10031.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_10032_13283.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_13284_14759.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_14760_17003.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_17004_22595.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_22596_24275.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_24276_25235.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_25236_26483.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_26484_28667.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_28668_31139.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_31140_35699.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_35700_36755.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_36756_36863.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_36864_37007.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_37008_37055.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_37056_37103.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_37104_37751.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_37752_38015.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_38016_38255.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_38256_42431.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_42432_44447.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_44448_44711.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_44712_47255.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_47256_47543.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_47544_48359.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_48360_49655.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_49656_50411.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_50412_53555.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_53556_56495.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_56496_58391.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_58392_59039.png DEF_CON_29_-_Rex_Guo__Junyuan_Zeng_-_Phantom_Attac_yaAdM8pWKG8_59040_59206.png