Exploiting Out-of-Order-Execution
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
D'Antoine, Sophia
Given the rise in popularity of cloud computing and platform-as-a-service, vulnerabilities inherent to systems which share hardware resources will become increasingly attractive targets to malicious software authors. This talk first presents a classification of the possible cloud-based side channels which use hardware virtualization. Additionally, a novel side channel exploiting out-of-order-execution in the CPU pipeline is described and implemented. Finally, this talk will show constructions of several adversarial applications and demo two. These applications are deployed across the novel side channel to prove the viability of each exploit. We then analyze successful detection and mitigation techniques of the side channel attacks.
 Cool. All right. My name is Sophia Dantouan. In this past year, my master's research at RPI focused on exploiting out of order execution and using that to try and enable cross VM code execution. So it all begins with the cloud. And everyone here is probably pretty familiar with how the cloud is structured. Just to go over some of the basics, you have a bunch of virtual instances or virtual machines, all resting on shared hardware. And that shared hardware, shared resources, is allocated by the hypervisor up to the different operating systems. And that dynamic allocation happens through time. So it's always changing, which reduces costs for everyone. And that means that everyone's happy. So there are a few problems with how this situation is set up. Well, first of all, your data is stored remotely. And it might not be secure or it might be private. That host that you're sharing your data with might be vulnerable itself or untrustable. And then finally, the one that people often talk about is that your VMs, which are running your processes or your data, it co-located with a bunch of other virtual machines that you don't know who they are or what they're doing. And they're all sharing the same physical resources. So it's this physical co-location, which leads to side-chain vulnerabilities. Again, here's the basic hardware structure of the cloud. You have the hypervisor layer, which is in the middle, and it's taking that shared physical layer and dynamically allocating that up to the different operating machines above. So each virtual machine will see its own virtual allocation of that shared hardware. So the universal vulnerabilities with this is in that translation between the physical and the virtual hardware. Because it does happen through time, it's based off of the need of each process or each virtual machine. And this means that one virtual machine can cause contention with another for the same resource. And that basically means that your VMs activities, even if it's just telling someone else that you need physical resource Y, that means that your activities are not paked to someone else on the same hardware. So how can we exploit this? Well, we can use something like in cryptography, a side-channel attack. Is any attack which you can gain information from the surrounding system that's implementing the crypto scheme that you're running or the program that's encrypting something. Now in cloud computing, it's quite similar. It's a hardware-based side channel, which means that the environment, the physical environment surrounding the virtual machines is what's leaking the information. It does mean it's cross-VM, a switch VM, even though it's like a black box to other VMs, they can't query inside of it and they can't directly access anything inside. They can learn about this running environment that that virtual machine is running on. This does mean that the information must be both recordable and also repeatedly recordable. So you have to be able to reliably learn information from that environment and be able to map that to the same known processes or to the same known information with a certain degree of certainty. So you can kind of structure a basic send and receive model out of this, or a basic side channel. So this is hardware agnostic. You have one transmitter which is forcing artifacts. Either it could be knowingly forcing artifacts into the environment or unknowingly forcing artifacts in the environment. And that shared hardware is then being read by the receiver or the adversary to learn information about that one sender. So the different ways this can actually be used, if you just have a receiver or if you're just listening to the benign environment noise, you can do things like leaking what processes are running on other virtual machines or keying the environment in which you're on. So you can create a unique signature from the environment and the environment's average usage of the cache, for instance, and use that to ID that specific server that you're running on in the cloud. And that you could use then to determine what physical resource you're running on. Now on the opposite side of the spectrum, if you're just running a transmitter or if you're just forcing artifacts in the shared hardware, you could do something like a denial of service attack where you clog the pipeline or you clog the cache, so other processes that might need it can't. And this can be pretty basic in just forcing someone else out of the cache, but it can actually alter the other execution or the other processes and results. And if you mix these two together, you have something that is more like a communication network. So if someone's forcing patterns in the environment knowingly and someone's receiving those, they can send messages back and forth. And this is what most people think of when they think of side channels. And this is how it would look like. So this is just a simple communication network where you have several virtual machines forcing artifacts in that shared hardware medium, and one VM is then reading those artifacts from the medium and averaging them out to create meaning from them. Just to bring this into a more concrete example, in the cache, there's an attack called a flush and reload attack, and it's targeting the L3 cache tier. The receiver or the adversary, which is listening to the environment, flushes a pre-greed line of L3 cache and then queries it later on for information. And the victim VM, the one that's also using the shared resource, is accessing that same shared line of L3 cache. And in this case, it was doing crypto basically, and accessing that same shared line of L3 cache with its private key. And the adversary was able to leak that private key at the end. If you want to read more about this specific attack, it's on my website. You can read it there. But specifically, my research involves attacking the pipeline. So how can we use the pipeline in a similar way that most people use the cache to create side channels? Well, there's a couple benefits of the pipeline versus the cache. The first is quieter. It's much harder to detect that someone's misusing the pipeline as opposed to the cache, simply because the cache is easier to query or interact with. It's also not affected by noise in the system as much. It's not affected by cache misses or other errors that the system may have. In a super noisy environment, much like the cloud, where tons of processes are just operating normally on the same hardware, the pipeline side channel is actually increased or amplified in strength, which is great. So how are we actually doing this? How are we targeting the pipeline? Well, first of all, the attack factor is we want to create a side channel to exploit inherent properties of this hardware medium. So some things that we can be assured of being there. And this means that we have to have some basic requirements to create a side channel. We have to have shared hardware. We have to know we're dynamically allocated that resource, both of which are inherent properties of cloud systems, so that's great. And we have to be co-located with our victim VMs or other collaborating adversaries. And that's something a little harder to determine, but like I said earlier, it is possible. So we're going to assume that going forward. So specifically, we chose to target the processors, the medium, and on the processor, the CPU's pipeline. And the difficulty associated with the pipeline is that we need to query these artifacts or these messages that we're forcing dynamically. So there's no really easy way to query the pipeline for a specific state, because you can't. If you did that, you'd probably be affecting how the pipeline state is. So all we really can know about the pipeline is the instruction set or the instruction order we feed it from our processes in the order of those instructions. And it results from these instruction sets, so we get to know the values that the pipeline can return to us. Which basically means we can use out-of-order execution. And that's the artifact that we're going to be forcing, and then also recording from the pipeline to learn the state of the pipeline as well as to learn about other processes sharing the pipeline with us. This is how it's going to look. So we have a bunch of VMs all running on the shared processor. And like you can see here, the processes are all sharing between two cores. And so this does assume that SMT is turned on, but in most modern systems, that's the case. So it's not a big deal. And then finally, one interesting thought here is that your instructions, especially in the cloud on shared hardware, are all being executed together with instructions from other processes from foreign VMs that you know nothing about. They're all just being processed in the pipeline in one big pool, as if they were all from one big program, which is kind of scary, because it's supposed to be separate. So how are we going to receive out-of-order executions from the pipeline? Well, like all good presentations, we have a picture of the Intel manual. This picture is basically just showing us that we can get a case on the pipeline that is out-of-order execution. So we can get results from the pipeline that are not expected. And that's what we can record. And this is what it looks like specifically in our receiver. So we have two threads, thread one and two, and they're both storing a value to a specific spot memory and then loading from memory. Now, the key thing here is that the load is happening from the opposite location in memory. So it's store of one to X, but the load from X to R2 happens in the other thread. So in the perfect world, the synced thread case, you get R1 and R2 equals one. But more often the case than not, your threads aren't going to be synced. So you can also get a case where some of the store, the store and load of one happens before the store and load of the other. And in that case, you could get either R1 or R2 being one and the other zero. And both of these cases are pretty normal, so we're going to ignore them. However, in the final case, in the out-of-order execution case, the loads are actually reordered in front of the stores. And in that scenario, X and Y were preset to zero, we're going to get R2 and R1 equal to zero as well. And that's the out-of-order execution case we can count. And so this is just the pseudocode of our receiver. We're iterating through these two threads thousands and thousands of times in certain timeframes. And when we do this, we can actually get a count of the average number of out-of-order executions received in a specific timeframe. And that's great for us because averages matter. We can take these averages in a specific timeframe and learn what an expected average should be and what an analogous one would be or anomalous one would be. So to transmit or to force patterns in this received average out-of-order execution bitstream that we've now constructed, we have to have the ability to force out-of-order execution averages to increase or decrease. So we're going to actually force the average out-of-order execution count to decrease using memory fences. Now everyone here has probably heard of FUNCE, the x86 instruction. But it prevents memory reordering of any kind, which is great because that's what we want to do. We want to decrease the amount of out-of-order executions. It is more expensive operation, but that's great because it's going to do what we want. So this is what the pipeline would look like. Our transmitter is forcing these memory fences in the same pipeline as the receiver. Now the key thing here is that these MFENSEs are being shoved in in the same timeframes that our receiver is recording in. So that is one key thing to have. But in this scenario, the MFENSE is going to force the pipeline to store the values of x or of 1 to x and of 1 to y before the loads. So these actually should be flipped, but it's going to force the proper ordering of our instructions. So this brings us to the importance of memory models. So there's two different types of memory reordering, compilation time and runtime. Obviously we're focusing on runtime or the out-of-order execution case where the pipeline is dynamically reordering our instructions. And we're also focusing on the usually strong memory model. So x86 architecture, this basically means that for the most case the pipeline is going to handle our instructions safely. It's going to give us the results that are expected or correct. However, it's usually strong, which doesn't mean always. And that does mean that we can get that incorrect case or the R1, R2 equals 0. So we're exploiting this inherent property of pipeline optimization. And there's four different types of memory barriers. So the specific case that we're focusing on is we want to force the store to occur before the load. However, there are four different types. Unfortunately, the store load barrier is the most expensive, so it must be truth most things in life. But to reiterate what we were saying earlier, we want to force out-of-order executions. So we're going to assume SMT's turned on and we're going to use a store load barrier of offense to prevent that out-of-order execution case. So to decrease the average out-of-order executions that we can read from the pipeline in specific time intervals. And so for our victim or our transmitter, we're going to force these patterns and infect the order of stores and loads. And like I said earlier, it's time frame dependent. So now we have both the ability to force out-of-order executions on the pipeline as well as to receive. So now we just have to design the channel. So in our lab, we had a Zen hypervisor just because it is the most popular commercial platform. Xeon processors, shared hardware, specifically four cores and six virtual machines. Obviously SMT was turned on. This is what it would look like. We had our six Windows 7 virtual machines all running noisy operations. And that was just to create similarities between our lab environment and the real world case in which one server might have thousands and thousands of processes on it. All right. Now specifically, if we apply our basic sending and receive model to this, we'd have one VM acting as a sender and one as the receiver. Or in a bi-party communication situation, you'd have one VM having both and the other having both as well. And this, and because our side channel is over the pipeline, these processes, the send and receive, would all have to be assured to be executing on the same pipeline. All right. So to demonstrate this, this is just Windows 7 VM with Zen Center on it so I can connect to my Zen server. And we have our clones here, and they're all just sharing the same amount of hardware, the same space. Oh, and I forgot to say you can follow along if you want. On my website, you can get the send and receiveer Python scripts that I have. These are wrappers around the actual scene assembly code that I use to force and receive out of order executions. However, the Python is great because it helps me easily adjust for the noise in different time frames and things like that during the testing. So in this scenario, we had recon 1 being our receiver. And this is, yeah, so receiver was just, we're going to show it first, just receiving the noise from the system. So we're canceling out the noise from the system, and you can see a bit stream of zero here. So this is just reading from the system, saying there's nothing being forced. I'm going to count these all as zeros, and it's plotting it to a graph, which is part of the Python script. It also sends all the out of order execution averages to a file as well, the Python. And you can see that here. And you can see where the different time frames are, and it looks a little bit more drastic than it actually is. The differences between each time frame's average. However, like I said, each system might have a unique signature, given what different processes are running on it. So you would see different average patterns like that based on what system you're running on. Now to show the communication occurring, our sender is going to force out of order executions in the same time frames that the receiver is sending them. And it's just in this scenario going to send two high bits, and those two high bits are going to trigger something in the receiver. So this is just a simple example to show that it works, but with a little bit of engineering, you could create something a bit more malicious, I guess. So blue screen. All right, so in conclusion, like all good academic papers, we had potential mitigation techniques. But the one that had the most possibility of success was isolating your VMs. So if you have your VMs in separate hardware, then you're definitely not going to be affected by this attack. You could also turn off SMT, and you could also set up a custom hypervisor, which is actually watching to make sure that processes from different virtual machines are not sharing resources at the same time, or if they are separated somehow. However, the downsides to all of this are some of the cloud benefits that you do get from sharing. So in conclusion, our contribution was the largest part of it was creating a novel side channel over the pipeline, even though the cache is a bit more popular, and the pipeline harder to query the state of, it is possible. And we show this dynamic method, and we show the application of this in the cloud, as well as some potential mitigation techniques. So I'd like to acknowledge Jeremy Blackthorne from MIT Lincoln Lab for introducing me to this topic, as well as RPI Second Trail of Bits. If there's any questions, you can reach me at IRC, email me, or if you're really adventurous, you can read my thesis. It's on my website. It's all 120 pages of it. So good luck. Or any questions now I can take? Hey, hey, hey. Somebody's got a question. Hang on. Just real quick. What do you define as a noisy environment? When I say noise, I'm basically saying that there's processes or activity in the system that's creating a load in the resource that you're targeting. So like in the case of the cache, you'd have a bunch of other processes shoving values in the cache or using the cache for something. So when you say you simulated a noisy environment, you just had stuff going on. Nothing is particular, right? Yeah, that's what I meant. It's going to be completely different depending on every system you're on, right? Exactly. That's the point. So you want to create an environment with enough processes to create some sort of entropy in the signals that you're reading, just so you know that your noise cancelling algorithms are working or your receiver is not too delicate and things like that. So how can you characterize a normal level of noise? One of these specifics of the environment, so what I did for this case was you just basically take thousands and thousands of recordings from your system and average those together. Have you tried that on, say, EC2 or? We're trying that now, actually. So, I mean, the key thing is that you have to know that you're co-located if you want to do some sort of send receive model. However, you can just download my scripts now and run it on EC2 and record the levels of noise from the systems. You can create a unique signature for that box that you're running on. And based on what noise algorithms and averages you're doing, it would alter the granularity of that signature. However, you could do that right now. And the stability of the patterns that you were observing, again, is there a way to characterize that? The stability of it? So, let's say, did you see any specific pattern for any specific system or processor? Oh, I see. Well, like I said, you can either take systems readings that are unique to that box that you're running on. Or I did see patterns associated with different processes. So, I tested it. Like, you can read this on my thesis. I did a bunch of different attacks. But, in for instance, I tested it on Chrome. So, I would have, like, YouTube open running a bunch of things in 1VM. And you'd actually get a unique signature from that. You wouldn't be able to see what YouTube video they were watching, but you could see that someone was watching something on YouTube. Fascinating. Thank you so much.
 Cool.  All right.  My name is Sophia Dantouan.  In this past year, my master's research at RPI focused on  exploiting out of order execution and using that to try and enable  cross VM code execution.  So it all begins with the cloud.  And everyone here is probably pretty familiar with how the cloud  is structured.  Just to go over some of the basics, you have a bunch of virtual  instances or virtual machines, all resting on shared hardware.  And that shared hardware, shared resources, is allocated by the  hypervisor up to the different operating systems.  And that dynamic allocation happens through time.  So it's always changing, which reduces costs for everyone.  And that means that everyone's happy.  So there are a few problems with how this situation is set up.  Well, first of all, your data is stored remotely.  And it might not be secure or it might be private.  That host that you're sharing your data with might be vulnerable  itself or untrustable.  And then finally, the one that people often talk about is that  your VMs, which are running your processes or your data,  it co-located with a bunch of other virtual machines that you  don't know who they are or what they're doing.  And they're all sharing the same physical resources.  So it's this physical co-location, which leads to side-chain  vulnerabilities.  Again, here's the basic hardware structure of the cloud.  You have the hypervisor layer, which is in the middle,  and it's taking that shared physical layer and dynamically  allocating that up to the different operating machines above.  So each virtual machine will see its own virtual allocation  of that shared hardware.  So the universal vulnerabilities with this is in that translation  between the physical and the virtual hardware.  Because it does happen through time, it's based off of the need  of each process or each virtual machine.  And this means that one virtual machine can cause contention  with another for the same resource.  And that basically means that your VMs activities,  even if it's just telling someone else that you need  physical resource Y, that means that your activities are not  paked to someone else on the same hardware.  So how can we exploit this?  Well, we can use something like in cryptography,  a side-channel attack.  Is any attack which you can gain information from the surrounding  system that's implementing the crypto scheme that you're running  or the program that's encrypting something.  Now in cloud computing, it's quite similar.  It's a hardware-based side channel, which means that the environment,  the physical environment surrounding the virtual machines  is what's leaking the information.  It does mean it's cross-VM, a switch VM, even though it's like  a black box to other VMs, they can't query inside of it  and they can't directly access anything inside.  They can learn about this running environment that that virtual machine  is running on.  This does mean that the information must be both recordable  and also repeatedly recordable.  So you have to be able to reliably learn information from that environment  and be able to map that to the same known processes  or to the same known information with a certain degree of certainty.  So you can kind of structure a basic send and receive model out of this,  or a basic side channel.  So this is hardware agnostic.  You have one transmitter which is forcing artifacts.  Either it could be knowingly forcing artifacts into the environment  or unknowingly forcing artifacts in the environment.  And that shared hardware is then being read by the receiver  or the adversary to learn information about that one sender.  So the different ways this can actually be used,  if you just have a receiver or if you're just listening to the benign environment noise,  you can do things like leaking what processes are running on other virtual machines  or keying the environment in which you're on.  So you can create a unique signature from the environment  and the environment's average usage of the cache, for instance,  and use that to ID that specific server that you're running on in the cloud.  And that you could use then to determine what physical resource you're running on.  Now on the opposite side of the spectrum,  if you're just running a transmitter or if you're just forcing artifacts  in the shared hardware, you could do something like a denial of service attack  where you clog the pipeline or you clog the cache,  so other processes that might need it can't.  And this can be pretty basic in just forcing someone else out of the cache,  but it can actually alter the other execution or the other processes and results.  And if you mix these two together, you have something that is more like a communication network.  So if someone's forcing patterns in the environment knowingly and someone's receiving those,  they can send messages back and forth.  And this is what most people think of when they think of side channels.  And this is how it would look like.  So this is just a simple communication network where you have several virtual machines  forcing artifacts in that shared hardware medium,  and one VM is then reading those artifacts from the medium  and averaging them out to create meaning from them.  Just to bring this into a more concrete example, in the cache,  there's an attack called a flush and reload attack,  and it's targeting the L3 cache tier.  The receiver or the adversary, which is listening to the environment,  flushes a pre-greed line of L3 cache and then queries it later on for information.  And the victim VM, the one that's also using the shared resource,  is accessing that same shared line of L3 cache.  And in this case, it was doing crypto basically,  and accessing that same shared line of L3 cache with its private key.  And the adversary was able to leak that private key at the end.  If you want to read more about this specific attack, it's on my website.  You can read it there.  But specifically, my research involves attacking the pipeline.  So how can we use the pipeline in a similar way that most people use the cache  to create side channels?  Well, there's a couple benefits of the pipeline versus the cache.  The first is quieter.  It's much harder to detect that someone's misusing the pipeline as opposed to the cache,  simply because the cache is easier to query or interact with.  It's also not affected by noise in the system as much.  It's not affected by cache misses or other errors that the system may have.  In a super noisy environment, much like the cloud,  where tons of processes are just operating normally on the same hardware,  the pipeline side channel is actually increased or amplified in strength, which is great.  So how are we actually doing this? How are we targeting the pipeline?  Well, first of all, the attack factor is we want to create a side channel  to exploit inherent properties of this hardware medium.  So some things that we can be assured of being there.  And this means that we have to have some basic requirements to create a side channel.  We have to have shared hardware.  We have to know we're dynamically allocated that resource,  both of which are inherent properties of cloud systems, so that's great.  And we have to be co-located with our victim VMs or other collaborating adversaries.  And that's something a little harder to determine, but like I said earlier, it is possible.  So we're going to assume that going forward.  So specifically, we chose to target the processors, the medium,  and on the processor, the CPU's pipeline.  And the difficulty associated with the pipeline is that we need to query these artifacts  or these messages that we're forcing dynamically.  So there's no really easy way to query the pipeline for a specific state, because you can't.  If you did that, you'd probably be affecting how the pipeline state is.  So all we really can know about the pipeline is the instruction set  or the instruction order we feed it from our processes in the order of those instructions.  And it results from these instruction sets, so we get to know the values that the pipeline can return to us.  Which basically means we can use out-of-order execution.  And that's the artifact that we're going to be forcing,  and then also recording from the pipeline to learn the state of the pipeline  as well as to learn about other processes sharing the pipeline with us.  This is how it's going to look.  So we have a bunch of VMs all running on the shared processor.  And like you can see here, the processes are all sharing between two cores.  And so this does assume that SMT is turned on, but in most modern systems, that's the case.  So it's not a big deal.  And then finally, one interesting thought here is that your instructions,  especially in the cloud on shared hardware, are all being executed together  with instructions from other processes from foreign VMs that you know nothing about.  They're all just being processed in the pipeline in one big pool,  as if they were all from one big program, which is kind of scary, because it's supposed to be separate.  So how are we going to receive out-of-order executions from the pipeline?  Well, like all good presentations, we have a picture of the Intel manual.  This picture is basically just showing us that we can get a case on the pipeline that is out-of-order execution.  So we can get results from the pipeline that are not expected.  And that's what we can record.  And this is what it looks like specifically in our receiver.  So we have two threads, thread one and two, and they're both storing a value to a specific spot memory  and then loading from memory.  Now, the key thing here is that the load is happening from the opposite location in memory.  So it's store of one to X, but the load from X to R2 happens in the other thread.  So in the perfect world, the synced thread case, you get R1 and R2 equals one.  But more often the case than not, your threads aren't going to be synced.  So you can also get a case where some of the store, the store and load of one happens before the store and load of the other.  And in that case, you could get either R1 or R2 being one and the other zero.  And both of these cases are pretty normal, so we're going to ignore them.  However, in the final case, in the out-of-order execution case, the loads are actually reordered in front of the stores.  And in that scenario, X and Y were preset to zero, we're going to get R2 and R1 equal to zero as well.  And that's the out-of-order execution case we can count.  And so this is just the pseudocode of our receiver.  We're iterating through these two threads thousands and thousands of times in certain timeframes.  And when we do this, we can actually get a count of the average number of out-of-order executions received in a specific timeframe.  And that's great for us because averages matter.  We can take these averages in a specific timeframe and learn what an expected average should be  and what an analogous one would be or anomalous one would be.  So to transmit or to force patterns in this received average out-of-order execution bitstream that we've now constructed,  we have to have the ability to force out-of-order execution averages to increase or decrease.  So we're going to actually force the average out-of-order execution count to decrease using memory fences.  Now everyone here has probably heard of FUNCE, the x86 instruction.  But it prevents memory reordering of any kind, which is great because that's what we want to do.  We want to decrease the amount of out-of-order executions.  It is more expensive operation, but that's great because it's going to do what we want.  So this is what the pipeline would look like.  Our transmitter is forcing these memory fences in the same pipeline as the receiver.  Now the key thing here is that these MFENSEs are being shoved in in the same timeframes that our receiver is recording in.  So that is one key thing to have.  But in this scenario, the MFENSE is going to force the pipeline to store the values of x or of 1 to x and of 1 to y before the loads.  So these actually should be flipped, but it's going to force the proper ordering of our instructions.  So this brings us to the importance of memory models.  So there's two different types of memory reordering, compilation time and runtime.  Obviously we're focusing on runtime or the out-of-order execution case where the pipeline is dynamically reordering our instructions.  And we're also focusing on the usually strong memory model.  So x86 architecture, this basically means that for the most case the pipeline is going to handle our instructions safely.  It's going to give us the results that are expected or correct.  However, it's usually strong, which doesn't mean always.  And that does mean that we can get that incorrect case or the R1, R2 equals 0.  So we're exploiting this inherent property of pipeline optimization.  And there's four different types of memory barriers.  So the specific case that we're focusing on is we want to force the store to occur before the load.  However, there are four different types.  Unfortunately, the store load barrier is the most expensive, so it must be truth most things in life.  But to reiterate what we were saying earlier, we want to force out-of-order executions.  So we're going to assume SMT's turned on and we're going to use a store load barrier of offense to prevent that out-of-order execution case.  So to decrease the average out-of-order executions that we can read from the pipeline in specific time intervals.  And so for our victim or our transmitter, we're going to force these patterns and infect the order of stores and loads.  And like I said earlier, it's time frame dependent.  So now we have both the ability to force out-of-order executions on the pipeline as well as to receive.  So now we just have to design the channel.  So in our lab, we had a Zen hypervisor just because it is the most popular commercial platform.  Xeon processors, shared hardware, specifically four cores and six virtual machines.  Obviously SMT was turned on.  This is what it would look like.  We had our six Windows 7 virtual machines all running noisy operations.  And that was just to create similarities between our lab environment and the real world case in which one server might have thousands and thousands of processes on it.  All right.  Now specifically, if we apply our basic sending and receive model to this, we'd have one VM acting as a sender and one as the receiver.  Or in a bi-party communication situation, you'd have one VM having both and the other having both as well.  And this, and because our side channel is over the pipeline, these processes, the send and receive, would all have to be assured to be executing on the same pipeline.  All right.  So to demonstrate this, this is just Windows 7 VM with Zen Center on it so I can connect to my Zen server.  And we have our clones here, and they're all just sharing the same amount of hardware, the same space.  Oh, and I forgot to say you can follow along if you want.  On my website, you can get the send and receiveer Python scripts that I have.  These are wrappers around the actual scene assembly code that I use to force and receive out of order executions.  However, the Python is great because it helps me easily adjust for the noise in different time frames and things like that during the testing.  So in this scenario, we had recon 1 being our receiver.  And this is, yeah, so receiver was just, we're going to show it first, just receiving the noise from the system.  So we're canceling out the noise from the system, and you can see a bit stream of zero here.  So this is just reading from the system, saying there's nothing being forced.  I'm going to count these all as zeros, and it's plotting it to a graph, which is part of the Python script.  It also sends all the out of order execution averages to a file as well, the Python.  And you can see that here.  And you can see where the different time frames are, and it looks a little bit more drastic than it actually is.  The differences between each time frame's average.  However, like I said, each system might have a unique signature, given what different processes are running on it.  So you would see different average patterns like that based on what system you're running on.  Now to show the communication occurring, our sender is going to force out of order executions in the same time frames that the receiver is sending them.  And it's just in this scenario going to send two high bits, and those two high bits are going to trigger something in the receiver.  So this is just a simple example to show that it works, but with a little bit of engineering, you could create something a bit more malicious, I guess.  So blue screen.  All right, so in conclusion, like all good academic papers, we had potential mitigation techniques.  But the one that had the most possibility of success was isolating your VMs.  So if you have your VMs in separate hardware, then you're definitely not going to be affected by this attack.  You could also turn off SMT, and you could also set up a custom hypervisor, which is actually watching to make sure that processes from different virtual machines are not sharing resources at the same time,  or if they are separated somehow.  However, the downsides to all of this are some of the cloud benefits that you do get from sharing.  So in conclusion, our contribution was the largest part of it was creating a novel side channel over the pipeline, even though the cache is a bit more popular,  and the pipeline harder to query the state of, it is possible.  And we show this dynamic method, and we show the application of this in the cloud, as well as some potential mitigation techniques.  So I'd like to acknowledge Jeremy Blackthorne from MIT Lincoln Lab for introducing me to this topic, as well as RPI Second Trail of Bits.  If there's any questions, you can reach me at IRC, email me, or if you're really adventurous, you can read my thesis.  It's on my website. It's all 120 pages of it. So good luck.  Or any questions now I can take?  Hey, hey, hey. Somebody's got a question. Hang on.  Just real quick. What do you define as a noisy environment?  When I say noise, I'm basically saying that there's processes or activity in the system that's creating a load in the resource that you're targeting.  So like in the case of the cache, you'd have a bunch of other processes shoving values in the cache or using the cache for something.  So when you say you simulated a noisy environment, you just had stuff going on. Nothing is particular, right?  Yeah, that's what I meant.  It's going to be completely different depending on every system you're on, right?  Exactly. That's the point.  So you want to create an environment with enough processes to create some sort of entropy in the signals that you're reading, just so you know that your noise cancelling algorithms are working or your receiver is not too delicate and things like that.  So how can you characterize a normal level of noise?  One of these specifics of the environment, so what I did for this case was you just basically take thousands and thousands of recordings from your system and average those together.  Have you tried that on, say, EC2 or?  We're trying that now, actually.  So, I mean, the key thing is that you have to know that you're co-located if you want to do some sort of send receive model.  However, you can just download my scripts now and run it on EC2 and record the levels of noise from the systems.  You can create a unique signature for that box that you're running on.  And based on what noise algorithms and averages you're doing, it would alter the granularity of that signature.  However, you could do that right now.  And the stability of the patterns that you were observing, again, is there a way to characterize that?  The stability of it?  So, let's say, did you see any specific pattern for any specific system or processor?  Oh, I see.  Well, like I said, you can either take systems readings that are unique to that box that you're running on.  Or I did see patterns associated with different processes.  So, I tested it.  Like, you can read this on my thesis.  I did a bunch of different attacks.  But, in for instance, I tested it on Chrome.  So, I would have, like, YouTube open running a bunch of things in 1VM.  And you'd actually get a unique signature from that.  You wouldn't be able to see what YouTube video they were watching, but you could see that someone was watching something on YouTube.  Fascinating.  Thank you so much.
recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_0_203.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_204_719.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_720_851.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_852_2231.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_2232_2351.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_2352_2843.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_2844_3635.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_3636_5231.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_5232_5951.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_5952_11963.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_11964_13019.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_13020_13367.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_13368_15047.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_15048_17543.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_17544_17891.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_17892_18539.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_18540_19535.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_19536_19619.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_19620_20699.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_20700_23135.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_23136_23927.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_23928_24359.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_24360_24971.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_24972_27587.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_27588_28655.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_28656_29459.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_29460_35159.png recon2015-06-sophia-d-antoine-Exploiting-Out-of-Order-Execution_35160_35208.png