AI VILLAGE - StuxNNet: Practical Live Memory Attacks on Machine Learning Systems
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Norwitz, Raphael
Kim, Bryan
Like all software systems, the execution of machine learning models is dictated by logic represented as data in memory. Unlike traditional software, machine learning systems’ behavior is defined by the model’s weight and bias parameters, rather than precise machine opcodes. Thus patching network parameters can achieve the same ends as traditional attacks, which have proven brittle and prone to errors. Moreover, this attack provides powerful obfuscation as neural network weights are hard to interpret, making it difficult for security professionals to determine what a malicious patch does. We demonstrate that one can easily compute a trojan patch, which when applied causes a network to behave incorrectly only on inputs with a given trigger. An attacker looking to compromise an ML system can patch these values in live memory with minimal risk of system malfunctions or other detectable side-effects. In this presentation, we demonstrate proof of concept attacks on TensorFlow and a framework we wrote in C++ on both Linux and Windows systems. An attack of this type relies on limiting the amount of network communication to reduce to the likelyhood of detection. Accordingly, we attempt to minimize the size of the patch, in terms of number of changed parameters needed to introduce trojan behavior. On an MNIST handwritten digit classification network and on a malicious PDF detection network, we prove that the desired trojan behavior can be introduced with patches on the order of 1% of the total network size, using roughly 1% of the total training data, proving that the attack is realistic. I am a recent graduate from Columbia Univserity with a BA in Computer Science and MS in Machine Learning, and an incoming engineer on the Acropolis Hypervisor team at Nutanix. I have experience with Linux Kernel development, data science and malware analysis. I have interned at Google, Drawbridge and Nimbledroid, and have published research with Columbia’s Wireless and Mobile Networking lab. For fun, I like to be outdoors and train Brazilian Ju-Jitsu.
 Hey everyone, so I'm Raypho and I'm going to be talking about Stux NNnet. I've worked on this project with one partner, Brian Kim. Our code is on GitHub and I've also posted some instructions for the demos I'm going to show you today. So if you want, you can go to my Twitter handle, that's it, and you'll be able to find this link, navigate there and see exactly what I'm doing. So let's get started. If we look at different kinds of attacks people do on software, there are different types of attacks on systems. A lot of times it's about getting the ability to run code on a victim system. There's all kinds of ways people do this and people talk about this all over DEFCON. In this village, however, we're concerned about AI systems in particular. So AI systems, we know about adversarial examples, we know about things like model inversion where you're looking at privacy issues, we know about trojan attacks on neural networks and many more. So our question was, is there anything interesting about the case where you have an exploit about the backdoor or anything into a machine learning system where you can run code on the victim system? Is there anything kind of novel about that? So when we start thinking about this, the way we, what we first looked at was the way the logic is actually encoded. So if you look at traditional software, you have kind of explicit coding, you know? You have the logic, it's written into assembly, that's then written into machine code. And with neural networks or any other kind of machine learning model, we're going to focus on neural networks for this talk. It's a little bit different. Your logic is encoded in train parameters, which are then combined with the inputs to get an output. So it isn't as straightforward. It's not like you can go through and just reverse engineer what somebody's doing if everything is, you know, if you've seen everything your attacker has done. So this is all, this gives neural networks some features, one of which is, you know, being black boxes and you can't exactly tell what the model is doing. But it's also, you know, it's interesting from an attack perspective, because unlike, you know, having to do other things where you're actually getting your own code on the CPU, here you can just change some data and you should see some interesting behavior. So maybe let's take a look at that. I'm going to do a little demo here. So apologies if the videos are complicated to use. I don't know if anyone else has done this today. But first actually, what's wrong with this? So I'm going to show you, sorry, slideshow, some card slide. So if you look at what I'm doing here, these are two identical neural networks. This code's all on GitHub, no need to like read it too carefully. But what it's doing is it's predicting the XOR function. This one is written for a toy neural network framework that we wrote in C++. This one is in TensorFlow. And you're going to see an attack on both of them now. So here is a, I think this is the, this is the simple model. This is the toy neural network framework. So what you'll see is that this will first predict correctly. And you'll see there, all I'm doing in the second window is finding the PID. And you see 110, correct, 101, correct, 01, correct, 00, correct. Now you see suddenly things change. So here, if I go, you see that it's been 110, sorry, 0, 1, 1, 0. And then suddenly here you get 0, 1, 0, 0. So what I've done here is I've gone into the memory of this process and zeroed out one of the weights. Now again, this is sort of a toy example. This is a framework that I wrote. No one uses this. Let's take a look at something that, you know, might be a little more realistic. So here is some, here is TensorFlow. Same exact network, same predictions. But here you're going to see I go through, I find the PID, run the malware, and you should see the same exact result. And note that this is the same exact file, I'm running, the same exact malware I'm running for both networks, which is kind of interesting as well. So boom, you get the 0, 1, 0, 0. This is exactly what we're looking for. So let's maybe take a second and look at how I did that. Oh, I'll get the presentation back up. Current slide, okay. So here. So the first thing you need to do is access the address space of the victim process. And you know, there's a number of different ways you can do this. My code's up there, we're not going to get into that too much. What is more interesting is how you actually figure out how to patch the network. So in the neural network framework that we wrote, you see that we use JSON to encode the, as, as, you know, basically the checkpoint file, how to store the network. And the weight we attacked was the minus 1, 1. So that weight up there. And you can see if you look at the code behind there in Python, what we're doing is we're figuring out the binary representation of that weight. And once we found that, you're looking down there, that's just a dump from, dump from Oli. You can see highlighted if the, if, you know, the projector is clear enough that we've actually found that weight in memory. And then as you go down, you see that something we've zeroed it out. And looking at the output of the network, you see it's correct, it's pretty good. Correctly, correctly, correctly. And then boom, suddenly the, that one flips. And you know that the patch has been properly applied. So this is kind of an interesting attack in and of itself, you know. So say there was a zero, there was like a buffer overflow in a self-driving car steering angle. And suddenly you just, you know, zeroed out a whole layer and all of a sudden the car, you know, turns to a sudden direction, all cars across the world at once or something like that. That would be pretty bad. And that's a pretty serious thing. So this in attack, this attack in and of itself should be cause for concern since it's so easy to do on TensorFlow. So now let's talk a little bit about the necessary steps to actually launch such an attack. So you need to somehow, you need to reverse engineer to some extent what the, how the system you're trying to attack works. So the way you would do this is you need to figure out how to get the weights. And if you're attacking like a self-driving car or a, you know, malware detector on a computer, you probably would take the computer apart, find the hard drive, extract whatever you can from that, you know, go through the file system and find something that looks like a weights file. Or you know, you would need to figure out, you look at reverse engineer, the architecture for the network to figure out exactly what it was using. And there, and it should be easier to find the weights you're looking for. And then that step is actually critical for the more serious attack I'll show later. But in terms of doing this and making this process easier, you could write different kinds of signatures for firmware. And if you just wrote a binwalk, say, or volatility, if you wanted to go look at different kinds of memory dumps or, you know, VM snapshots or what have you. And you know, to some extent you will need to reverse engineer the architecture for the more serious attack I'm going to show. But again, you know, once, since you were trying to attack, you know, commodity systems like self-driving cars, computers, or even something like a plane, you know, a serious actor should be able to, you know, go in there and figure out what's going on. Now let's switch gears a little bit and talk about different kinds of attacks. So we've talked a bit about poisoning in this village before, but basically what we're talking about is having a sort of a Trojan trigger. That's an input, that's a set of input characteristics that you want the neural network to misbehave on. So an example that I'll show later is, you know, say a combination of number of images and JavaScript objects in a PDF, if you're talking about a malicious PDF classifier, or maybe say the specific pixels in an MNIST or CIFAR-10 image. So you see the dots on the MNIST image or the little T there in the top left corner of the CIFAR-10 image. And once you've defined that trigger, you want to map all inputs of that trigger to a particular class that the network is going to output. So you know, like for instance, say we have those dots there, we would map all numbers with those dots to a four for instance. And once you define that behavior, then you need to go through and, you know, Trojan examples seen in training and continue training the network on those Trojan samples. So most people are concerned about this kind of, at least historically, people were concerned about this as an attack at train time. You hack a company that's trying to train a neural network and throw in, you know, mess up their training process. Or they're concerned that some malicious vendor has handed them a Trojan network that can be, you know, you have a nasty backdoor they don't know about. But our question here is what if we could actually patch a Trojan in at runtime? What would that do? Now before we even get into the nitty gritties of how one might do this, let's dive into the threat model. Like why would someone actually want to do this as opposed to just, you know, switching the output of the classifier to whatever they wanted to do. So neural networks, as we've discussed, are nonlinear models. They're, you know, black box, you can't interpret what the weights are doing. So, you know, as sort of a corollary to that, you, if assuming the trigger is subtle enough and not blatantly obvious, how can you know what a malicious patch actually does? You know, so someone's hacked you and patched your neural network, you know, maybe if you, for instance, if the attack is thwarted and you're just stuck with this patch, how can you go in there and figure out what they were trying to do to you? And you know, let's say the attack was deployed and there was, you know, serious damage. How do you know that there was, that that, that the damage was actually the full intent of the attack? How do you know there wasn't some underhand behavior that the, that the attackers actually wanted to perform? You know, say they put in multiply trojan with multiple different triggers, you know, maybe to make it look like someone else did it or, you know, whatever they would want to do, it could really complicate damage control and attribution. And if you think about what's actors who hack a lot, you know, state level people, you're thinking about, you know, they don't want it to be obvious who did what. The whole value of these things is that you can strike someone without them necessarily immediately being able to know it was you and strike you back or with the damage being complicated and a lot more, you know, costly investigations going on and confusion around exactly what happened. So with that in mind, this is a very, this kind of obfuscation that this attack provides or kind of mystifying the attacker intent is very powerful. And also from a technical perspective, it's nice because, you know, you may have had to actually, you know, modify executable code or something or do something funny to the stack to actually get the attack. But for the attack itself, you don't actually need to touch any executable code. You just need to flip a few numbers in memory. And there, you know, we find that these weights are contiguous in memory almost everywhere, which makes sense because, you know, the way processes work is they prefetch memory by locality. So having weights, similar weights close to each other speeds up the computation. So that feature makes it, you know, very easy to find weights in practice. Now that we've talked about a Trojan attack, let's show you a little example here. So this is a PDF classification model, just like the prior two models, it's kind of running in a loop and printing out some statistics. But you can see here that what we, how it works is that there's 5000 malicious and 5000 benign PDFs in the test set. We've taken the test set and we've Trojan it. We now have two test sets, a clean test set and a Trojan test set. This neural network will predict on both the clean and the Trojan and will output as we perform the attack. So here's a little video again. Let's pull it up. So here again, just like the prior examples, I've loaded my model. It's running. You can see the accuracy is incredibly high on both the clean and the Trojan set. Roughly 5000 PDFs are playing being flagged as malicious in total and true positive rate is almost, you know, 99% across the board. So this is a very effective classifier. Now that I've run my malware, we'll suddenly see something funky happen, which is suddenly we now have very poor performance on the Trojan set. Only half of the Trojan, you know, that accuracy is at 50%, which is exactly what you'd expect because the whole set is Trojan, so they should all be classified as benign. You also see that only 242 of 10,000 PDFs in the Trojan set are classified as malicious while the clean set still is, you know, hunky dory. All's good. And then where the really damning statistic is here, where you see, okay, 99% accuracy on actually detecting malicious PDFs for the clean set and boom, like 0.08% accuracy, 4-5,000 for the Trojan set. So, you know, this is an incredibly powerful Trojan that we've gone in and patched at runtime. So this, I think, is a pretty, you know, if you think about, you know, this on some major, on some serious whatever corporate network, this program meant to detect PDFs. Suddenly it's, you know, has this nasty Trojan in it, which allows a sophisticated attacker to send malicious PDFs all over the target. So moving back to the presentation here, seems we've jumped up a few slides. Now we're going to talk a little bit about some of the constraints you have to face when trying to perform this attack. So one of the ones, you know, like to put this attack to be realistic, you can't have the malware be massive. I mean, otherwise any weird binary that's throwing around massive amounts of data is likely to pull up red flags. And neural networks can be problematic in that way because they actually are quite large. A production neural network can be upwards of 40 to 100 megabytes. So the key here is actually making the size of the patch to introduce the Trojan behavior very small. Now we can do that through some, you know, there's a bunch of different ways to do that, one of which is to, you know, not actually store the weights you're looking for, but just the hashes of them and find those in RAM that can speed it, that can shrink it a little bit. But the real goal, what you actually need to do in the end is train a very sparse patch to change as few of the parameters as possible. Now this is, this is kind of interesting from a neural networks research perspective in that how effectively, you know, if you change very, if you're changing very little in the network, how effectively can you introduce new behavior? And you know, so how effectively can you actually introduce a Trojan and how much will actually making the sparse patch reduce the size of the malware? So there are two different approaches we came up for doing this. One is a more naive approach where you take a batch of training data that's been poisoned and you compute the gradients with respect to every parameter and only update the top K parameters, those which have the largest gradients and then you can go through and only keep retraining only with those K parameters being changed. So this approach actually works quite well in practice, but we came up with a little more sophisticated one. So unfortunate that it's hard to see here. So this is trying to use L0 regularization. So the idea is we add a penalty term to the cost function such that we penalize all non-zero terms that we're updating. So this is, this would be the ideal way to do this in practice. But you know, it's kind of tricky since now we're introducing a term which unfortunately you can't see is like this, which, sorry, which is non-differentiable. So that's kind of a problem. But there's nice, we found some nice research from statistics where they have been able to approximate L0 regularization and we implemented that. Unfortunately, I don't think in this talk we'll be able to get into the nitty gritty there. So we can, you know, maybe if we have time at the end, I have slides for it. So here's kind of, here's our results. And this is looking at the malicious PDF classifier first. You see that the baseline performance of the model is very high, 90% safe, 99% on malicious. And you know, so that's, that's originally great. And then we look at top K fraction and we see that we can go to 0.001% of the gradients being back propagated. And we have pretty great results because you have still a very effective Trojan 0.001 malicious class or classified properly, which is exactly what we want to see. We've only lost about 1% on our clean accuracy, which is again great, modifying 0.1% of the parameters. But we see with L0 regularization we can do substantially better. We lose a little bit more on performance. I think we lose like 2 there, 0.2, sorry, 0.02 on performance. But we still have an effective Trojan and we've only modified 0.43% of the parameters. So that's, sorry, 0.08% of the parameters. So this actually adds, this approach actually adds a lot of value here. So now the other set we evaluated on that we have good results that we have results for is MNIST. So MNIST if we look up at our model baseline, again exactly what you'd expect of a standard MNIST model, clean, roughly overall accuracy 93, 94%. We see that if we use the top K fraction, 0.01, we start to see a little degradation in performance, but with 0.005% of the gradients being back propagated, we still have totally fine performance. We've lost 0.3, 0.4%. But we're modifying only 0.4% of the parameters and our success rate on the Trojan input is great. So this is actually a very effective way to do the patch. And in this case, the naive method actually beats out our more fancy L0 regularization, which you can see also does well, but has a substantially higher percent of parameters modified. So then the other critical question for this kind of work is how effectively can we, is like how much of the training data do you actually need to do this? Because you know, on attack where you have to have 100% of the training data to perform, you know, a lot of speakers have talked about it and a lot of people agree it's not really realistic in practice. But so we tried, this is all with the approximate L0 regularization. But if you look again at these numbers, you'll see that with 0.01% of the numbers on the malicious PDF classifier, we only need 172 out of 17,000 PDFs to get a sparse patch that is effective. So you see 0.002, the closer that number is on the Trojan malicious to zero, the better. And the accuracy is still 0.93, 0.95, which is totally acceptable. So we can do very well on the malicious PDF detector there with 0.1% of the data. And same thing with MNIST, if you go down to 0.01, you see that we've lost roughly 2% on model performance, which is acceptable. And then if you're looking at the Trojan, at the amount of Trojans that are correct, you see that it's roughly 0.1. I actually realized, I forgot to mention, that if you're, when you're looking at these numbers, you want to, for MNIST, you want to see the Trojan accuracy close to 0.1. Because you know, there's 10 classes and you want, and all the classes that are Trojan should be mapped to the same class. So roughly it should get it right about 0.1% of the time. So with that, yeah, that's basically saying that we can do very effective training with very little training data. In terms of our conclusions, patches are simple to apply. Sparch passes can be trained effectively. You know, you don't need the full training data set. You need very little in order to perform this attack. There's a very powerful attacker from real, from realistic attackers. There's a very powerful incentive to avoid the kind of detection attribution you would see with other kinds of attacks. And production deep learning system should be concerned with this. Here are some models that we are working on right now. D-FAR-10 is very close, but not quite there. We want to try training multiple Trojans at once. Patches under different conditions. And then, you know, playing with different ways of synthesizing training data. And then in terms of, we want to build out the reverse engineering stuff, but I already talked about that. And then we also want to work on getting read-only patches for neural network libraries so that, you know, there can at least be some manner of protection against these kind of attacks in practice. So this, I just want to acknowledge, thank my professor, Jin Feng Yang, and my TA, Kation Pei, who are really helpful. This work was done as part of a class. And I'd also like to thank Professor Michael Sikorsky for his reverse engineering course, which was really helpful in producing this. So here are my references, and I'm pretty sure I'm clean out of time. So thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 Hey everyone, so I'm Raypho and I'm going to be talking about Stux NNnet.  I've worked on this project with one partner, Brian Kim.  Our code is on GitHub and I've also posted some instructions for the demos I'm going  to show you today.  So if you want, you can go to my Twitter handle, that's it, and you'll be able to find this  link, navigate there and see exactly what I'm doing.  So let's get started.  If we look at different kinds of attacks people do on software, there are different types  of attacks on systems.  A lot of times it's about getting the ability to run code on a victim system.  There's all kinds of ways people do this and people talk about this all over DEFCON.  In this village, however, we're concerned about AI systems in particular.  So AI systems, we know about adversarial examples, we know about things like model inversion  where you're looking at privacy issues, we know about trojan attacks on neural networks  and many more.  So our question was, is there anything interesting about the case where you have an exploit about  the backdoor or anything into a machine learning system where you can run code on the victim  system?  Is there anything kind of novel about that?  So when we start thinking about this, the way we, what we first looked at was the way  the logic is actually encoded.  So if you look at traditional software, you have kind of explicit coding, you know?  You have the logic, it's written into assembly, that's then written into machine code.  And with neural networks or any other kind of machine learning model, we're going to  focus on neural networks for this talk.  It's a little bit different.  Your logic is encoded in train parameters, which are then combined with the inputs to  get an output.  So it isn't as straightforward.  It's not like you can go through and just reverse engineer what somebody's doing if  everything is, you know, if you've seen everything your attacker has done.  So this is all, this gives neural networks some features, one of which is, you know,  being black boxes and you can't exactly tell what the model is doing.  But it's also, you know, it's interesting from an attack perspective, because unlike,  you know, having to do other things where you're actually getting your own code on the  CPU, here you can just change some data and you should see some interesting behavior.  So maybe let's take a look at that.  I'm going to do a little demo here.  So apologies if the videos are complicated to use.  I don't know if anyone else has done this today.  But first actually, what's wrong with this?  So I'm going to show you, sorry, slideshow, some card slide.  So if you look at what I'm doing here, these are two identical neural networks.  This code's all on GitHub, no need to like read it too carefully.  But what it's doing is it's predicting the XOR function.  This one is written for a toy neural network framework that we wrote in C++.  This one is in TensorFlow.  And you're going to see an attack on both of them now.  So here is a, I think this is the, this is the simple model.  This is the toy neural network framework.  So what you'll see is that this will first predict correctly.  And you'll see there, all I'm doing in the second window is finding the PID.  And you see 110, correct, 101, correct, 01, correct, 00, correct.  Now you see suddenly things change.  So here, if I go, you see that it's been 110, sorry, 0, 1, 1, 0.  And then suddenly here you get 0, 1, 0, 0.  So what I've done here is I've gone into the memory of this process and zeroed out  one of the weights.  Now again, this is sort of a toy example.  This is a framework that I wrote.  No one uses this.  Let's take a look at something that, you know, might be a little more realistic.  So here is some, here is TensorFlow.  Same exact network, same predictions.  But here you're going to see I go through, I find the PID, run the malware, and you should  see the same exact result.  And note that this is the same exact file, I'm running, the same exact malware I'm running  for both networks, which is kind of interesting as well.  So boom, you get the 0, 1, 0, 0.  This is exactly what we're looking for.  So let's maybe take a second and look at how I did that.  Oh, I'll get the presentation back up.  Current slide, okay.  So here.  So the first thing you need to do is access the address space of the victim process.  And you know, there's a number of different ways you can do this.  My code's up there, we're not going to get into that too much.  What is more interesting is how you actually figure out how to patch the network.  So in the neural network framework that we wrote, you see that we use JSON to encode  the, as, as, you know, basically the checkpoint file, how to store the network.  And the weight we attacked was the minus 1, 1.  So that weight up there.  And you can see if you look at the code behind there in Python, what we're doing is we're  figuring out the binary representation of that weight.  And once we found that, you're looking down there, that's just a dump from, dump from Oli.  You can see highlighted if the, if, you know, the projector is clear enough that we've actually  found that weight in memory.  And then as you go down, you see that something we've zeroed it out.  And looking at the output of the network, you see it's correct, it's pretty good.  Correctly, correctly, correctly.  And then boom, suddenly the, that one flips.  And you know that the patch has been properly applied.  So this is kind of an interesting attack in and of itself, you know.  So say there was a zero, there was like a buffer overflow in a self-driving car steering  angle.  And suddenly you just, you know, zeroed out a whole layer and all of a sudden the car,  you know, turns to a sudden direction, all cars across the world at once or something  like that.  That would be pretty bad.  And that's a pretty serious thing.  So this in attack, this attack in and of itself should be cause for concern since it's so  easy to do on TensorFlow.  So now let's talk a little bit about the necessary steps to actually launch such an attack.  So you need to somehow, you need to reverse engineer to some extent what the, how the system  you're trying to attack works.  So the way you would do this is you need to figure out how to get the weights.  And if you're attacking like a self-driving car or a, you know, malware detector on a  computer, you probably would take the computer apart, find the hard drive, extract whatever  you can from that, you know, go through the file system and find something that looks  like a weights file.  Or you know, you would need to figure out, you look at reverse engineer, the architecture  for the network to figure out exactly what it was using.  And there, and it should be easier to find the weights you're looking for.  And then that step is actually critical for the more serious attack I'll show later.  But in terms of doing this and making this process easier, you could write different  kinds of signatures for firmware.  And if you just wrote a binwalk, say, or volatility, if you wanted to go look at different kinds  of memory dumps or, you know, VM snapshots or what have you.  And you know, to some extent you will need to reverse engineer the architecture for the  more serious attack I'm going to show.  But again, you know, once, since you were trying to attack, you know, commodity systems  like self-driving cars, computers, or even something like a plane, you know, a serious  actor should be able to, you know, go in there and figure out what's going on.  Now let's switch gears a little bit and talk about different kinds of attacks.  So we've talked a bit about poisoning in this village before, but basically what we're talking  about is having a sort of a Trojan trigger.  That's an input, that's a set of input characteristics that you want the neural network to misbehave  on.  So an example that I'll show later is, you know, say a combination of number of images  and JavaScript objects in a PDF, if you're talking about a malicious PDF classifier,  or maybe say the specific pixels in an MNIST or CIFAR-10 image.  So you see the dots on the MNIST image or the little T there in the top left corner of  the CIFAR-10 image.  And once you've defined that trigger, you want to map all inputs of that trigger to a  particular class that the network is going to output.  So you know, like for instance, say we have those dots there, we would map all numbers  with those dots to a four for instance.  And once you define that behavior, then you need to go through and, you know, Trojan  examples seen in training and continue training the network on those Trojan samples.  So most people are concerned about this kind of, at least historically, people were concerned  about this as an attack at train time.  You hack a company that's trying to train a neural network and throw in, you know, mess  up their training process.  Or they're concerned that some malicious vendor has handed them a Trojan network that can  be, you know, you have a nasty backdoor they don't know about.  But our question here is what if we could actually patch a Trojan in at runtime?  What would that do?  Now before we even get into the nitty gritties of how one might do this, let's dive into  the threat model.  Like why would someone actually want to do this as opposed to just, you know, switching  the output of the classifier to whatever they wanted to do.  So neural networks, as we've discussed, are nonlinear models.  They're, you know, black box, you can't interpret what the weights are doing.  So, you know, as sort of a corollary to that, you, if assuming the trigger is subtle enough  and not blatantly obvious, how can you know what a malicious patch actually does?  You know, so someone's hacked you and patched your neural network, you know, maybe if you,  for instance, if the attack is thwarted and you're just stuck with this patch, how can  you go in there and figure out what they were trying to do to you?  And you know, let's say the attack was deployed and there was, you know, serious damage.  How do you know that there was, that that, that the damage was actually the full intent  of the attack?  How do you know there wasn't some underhand behavior that the, that the attackers actually  wanted to perform?  You know, say they put in multiply trojan with multiple different triggers, you know,  maybe to make it look like someone else did it or, you know, whatever they would want  to do, it could really complicate damage control and attribution.  And if you think about what's actors who hack a lot, you know, state level people, you're  thinking about, you know, they don't want it to be obvious who did what.  The whole value of these things is that you can strike someone without them necessarily  immediately being able to know it was you and strike you back or with the damage being  complicated and a lot more, you know, costly investigations going on and confusion around  exactly what happened.  So with that in mind, this is a very, this kind of obfuscation that this attack provides  or kind of mystifying the attacker intent is very powerful.  And also from a technical perspective, it's nice because, you know, you may have had to  actually, you know, modify executable code or something or do something funny to the  stack to actually get the attack.  But for the attack itself, you don't actually need to touch any executable code.  You just need to flip a few numbers in memory.  And there, you know, we find that these weights are contiguous in memory almost everywhere,  which makes sense because, you know, the way processes work is they prefetch memory by  locality.  So having weights, similar weights close to each other speeds up the computation.  So that feature makes it, you know, very easy to find weights in practice.  Now that we've talked about a Trojan attack, let's show you a little example here.  So this is a PDF classification model, just like the prior two models, it's kind of running  in a loop and printing out some statistics.  But you can see here that what we, how it works is that there's 5000 malicious and 5000  benign PDFs in the test set.  We've taken the test set and we've Trojan it.  We now have two test sets, a clean test set and a Trojan test set.  This neural network will predict on both the clean and the Trojan and will output as we  perform the attack.  So here's a little video again.  Let's pull it up.  So here again, just like the prior examples, I've loaded my model.  It's running.  You can see the accuracy is incredibly high on both the clean and the Trojan set.  Roughly 5000 PDFs are playing being flagged as malicious in total and true positive rate  is almost, you know, 99% across the board.  So this is a very effective classifier.  Now that I've run my malware, we'll suddenly see something funky happen, which is suddenly  we now have very poor performance on the Trojan set.  Only half of the Trojan, you know, that accuracy is at 50%, which is exactly what you'd expect  because the whole set is Trojan, so they should all be classified as benign.  You also see that only 242 of 10,000 PDFs in the Trojan set are classified as malicious  while the clean set still is, you know, hunky dory.  All's good.  And then where the really damning statistic is here, where you see, okay, 99% accuracy  on actually detecting malicious PDFs for the clean set and boom, like 0.08% accuracy, 4-5,000  for the Trojan set.  So, you know, this is an incredibly powerful Trojan that we've gone in and patched at runtime.  So this, I think, is a pretty, you know, if you think about, you know, this on some major,  on some serious whatever corporate network, this program meant to detect PDFs.  Suddenly it's, you know, has this nasty Trojan in it, which allows a sophisticated attacker  to send malicious PDFs all over the target.  So moving back to the presentation here, seems we've jumped up a few slides.  Now we're going to talk a little bit about some of the constraints you have to face when  trying to perform this attack.  So one of the ones, you know, like to put this attack to be realistic, you can't have  the malware be massive.  I mean, otherwise any weird binary that's throwing around massive amounts of data is  likely to pull up red flags.  And neural networks can be problematic in that way because they actually are quite large.  A production neural network can be upwards of 40 to 100 megabytes.  So the key here is actually making the size of the patch to introduce the Trojan behavior  very small.  Now we can do that through some, you know, there's a bunch of different ways to do that,  one of which is to, you know, not actually store the weights you're looking for, but  just the hashes of them and find those in RAM that can speed it, that can shrink it a  little bit.  But the real goal, what you actually need to do in the end is train a very sparse patch  to change as few of the parameters as possible.  Now this is, this is kind of interesting from a neural networks research perspective in  that how effectively, you know, if you change very, if you're changing very little in the  network, how effectively can you introduce new behavior?  And you know, so how effectively can you actually introduce a Trojan and how much will actually  making the sparse patch reduce the size of the malware?  So there are two different approaches we came up for doing this.  One is a more naive approach where you take a batch of training data that's been poisoned  and you compute the gradients with respect to every parameter and only update the top  K parameters, those which have the largest gradients and then you can go through and  only keep retraining only with those K parameters being changed.  So this approach actually works quite well in practice, but we came up with a little  more sophisticated one.  So unfortunate that it's hard to see here.  So this is trying to use L0 regularization.  So the idea is we add a penalty term to the cost function such that we penalize all non-zero  terms that we're updating.  So this is, this would be the ideal way to do this in practice.  But you know, it's kind of tricky since now we're introducing a term which unfortunately  you can't see is like this, which, sorry, which is non-differentiable.  So that's kind of a problem.  But there's nice, we found some nice research from statistics where they have been able  to approximate L0 regularization and we implemented that.  Unfortunately, I don't think in this talk we'll be able to get into the nitty gritty  there.  So we can, you know, maybe if we have time at the end, I have slides for it.  So here's kind of, here's our results.  And this is looking at the malicious PDF classifier first.  You see that the baseline performance of the model is very high, 90% safe, 99% on malicious.  And you know, so that's, that's originally great.  And then we look at top K fraction and we see that we can go to 0.001% of the gradients  being back propagated.  And we have pretty great results because you have still a very effective Trojan 0.001 malicious  class or classified properly, which is exactly what we want to see.  We've only lost about 1% on our clean accuracy, which is again great, modifying 0.1% of the  parameters.  But we see with L0 regularization we can do substantially better.  We lose a little bit more on performance.  I think we lose like 2 there, 0.2, sorry, 0.02 on performance.  But we still have an effective Trojan and we've only modified 0.43% of the parameters.  So that's, sorry, 0.08% of the parameters.  So this actually adds, this approach actually adds a lot of value here.  So now the other set we evaluated on that we have good results that we have results  for is MNIST.  So MNIST if we look up at our model baseline, again exactly what you'd expect of a standard  MNIST model, clean, roughly overall accuracy 93, 94%.  We see that if we use the top K fraction, 0.01, we start to see a little degradation  in performance, but with 0.005% of the gradients being back propagated, we still have totally  fine performance.  We've lost 0.3, 0.4%.  But we're modifying only 0.4% of the parameters and our success rate on the Trojan input is  great.  So this is actually a very effective way to do the patch.  And in this case, the naive method actually beats out our more fancy L0 regularization,  which you can see also does well, but has a substantially higher percent of parameters  modified.  So then the other critical question for this kind of work is how effectively can we, is  like how much of the training data do you actually need to do this?  Because you know, on attack where you have to have 100% of the training data to perform,  you know, a lot of speakers have talked about it and a lot of people agree it's not really  realistic in practice.  But so we tried, this is all with the approximate L0 regularization.  But if you look again at these numbers, you'll see that with 0.01% of the numbers on the  malicious PDF classifier, we only need 172 out of 17,000 PDFs to get a sparse patch that  is effective.  So you see 0.002, the closer that number is on the Trojan malicious to zero, the better.  And the accuracy is still 0.93, 0.95, which is totally acceptable.  So we can do very well on the malicious PDF detector there with 0.1% of the data.  And same thing with MNIST, if you go down to 0.01, you see that we've lost roughly 2%  on model performance, which is acceptable.  And then if you're looking at the Trojan, at the amount of Trojans that are correct,  you see that it's roughly 0.1.  I actually realized, I forgot to mention, that if you're, when you're looking at these  numbers, you want to, for MNIST, you want to see the Trojan accuracy close to 0.1.  Because you know, there's 10 classes and you want, and all the classes that are Trojan  should be mapped to the same class.  So roughly it should get it right about 0.1% of the time.  So with that, yeah, that's basically saying that we can do very effective training with  very little training data.  In terms of our conclusions, patches are simple to apply.  Sparch passes can be trained effectively.  You know, you don't need the full training data set.  You need very little in order to perform this attack.  There's a very powerful attacker from real, from realistic attackers.  There's a very powerful incentive to avoid the kind of detection attribution you would  see with other kinds of attacks.  And production deep learning system should be concerned with this.  Here are some models that we are working on right now.  D-FAR-10 is very close, but not quite there.  We want to try training multiple Trojans at once.  Patches under different conditions.  And then, you know, playing with different ways of synthesizing training data.  And then in terms of, we want to build out the reverse engineering stuff, but I already  talked about that.  And then we also want to work on getting read-only patches for neural network libraries so that,  you know, there can at least be some manner of protection against these kind of attacks  in practice.  So this, I just want to acknowledge, thank my professor, Jin Feng Yang, and my TA, Kation  Pei, who are really helpful.  This work was done as part of a class.  And I'd also like to thank Professor Michael Sikorsky for his reverse engineering course,  which was really helpful in producing this.  So here are my references, and I'm pretty sure I'm clean out of time.  So thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_0_635.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_636_1739.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_1740_3287.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_3288_3647.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_3648_4151.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_4152_4211.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_4212_5495.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_5496_6107.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_6108_6299.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_6300_6359.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_6360_8291.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_8292_10379.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_10380_12431.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_12432_15851.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_15852_16583.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_16584_16727.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_16728_16775.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_16776_17543.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_17544_17687.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_17688_18911.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_18912_19079.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_19080_20699.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_20700_22547.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_22548_24227.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_24228_25583.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_25584_28019.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_28020_29315.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_29316_29675.png DEF_CON_26_AI_VILLAGE_-_Raphael_Norwitz_-_StuxNNet_Practical_Live_Memory_Attacks_on_Mac_29676_29904.png