AI VILLAGE - AI DevOps: Behind the Scenes of a Global Anti-Virus's Machine Learning Infrastructure
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Long, Alex
Thus far, the security community has treated machine learning as a research problem. The painful oversight here is in thinking that laboratory results would translate easily to the real world, and as such, not devoting sufficient focus to bridging that gap. Researchers enjoy the luxuries of neat bite-sized datasets to experiment upon, but the harsh reality of millions of potentially malicious files streaming in daily soon hits would-be ML-practitioners in the face like a tsunami-sized splash of ice water. And while in research there’s no such thing as ““too much”” data, dataset sizes challenge real-world cyber security professionals with tough questions: ““How will we store these files efficiently without hampering our ability to use them for day-to-day operations?”” or ““How do we satisfy competing use-cases such as the need to analyze specific files and the need to run analyses across the entire dataset?”” Or maybe most importantly: ““Will my boss have a heart-attack when he sees my AWS bill?”” In this talk, we will provide a live demonstration of the system we’ve built using a variety of AWS services including DynamoDB, Kinesis, Lambda, as well as some more cutting edge AWS services such as Redshift and ECS Fargate. We will go into depth about how the system works and how it answers the difficult questions of real world ML such as the ones listed above. This talk will provide a rare look into the guts of a large-scale machine learning production system. As a result, it will give audience members the tools and understanding to confidently tackle such problems themselves and ultimately give them a bedrock of immediately practical knowledge for deploying large-scale on-demand deep learning in the cloud.
 And without further ado, here's Alex with his talk. Thank you. Thank you for coming on a Sunday morning. We are gathered here today to talk about AI. My talk is entitled AI DevOps Behind the Scenes of a Global Antiviruses Machine Learning Infrastructure. And just a meta note real quick. In making this talk, I was, I'm going to be talking about work that was performed by many people on my team. So I actually interviewed about four people. And this work spans about two years. And I have about 20 minutes to tell you about it. So this is going to be a very kind of distilled, highlight reel of our story. Who am I? So I'm doing a lot of things at the team. I do a lot of web dev. I do a lot of AWS backend stuff. Some ML research. I've done some Android stuff. That's a picture of my face and my contact information. This is a picture of my cats, which you have three seconds to enjoy. This is the data science team at Sophos. Maddie Shihapa is actually in the crowd today. And Hilary Sanders, Constantine Berlin and the two mats at the bottom contributed a lot more than I did to the work that I'm taking credit for today. I also have you on here because we're hiring. So if you have any expertise at the crossroads of cybersecurity and data science, we'd love to hear from you. Okay, so in the beginning there was signature-based antivirus. And by beginning I mean 2017. That's about the time that the data science team was added to Sophos. And we were the first kind of introduction of ML to their antivirus product. And there were a lot of just initial misconceptions. The big one being that machine learning is all you need. So this is something that we kind of had to convince people was wrong even though we are the data scientists. And so just to kind of give you guys a way of thinking about this. If you imagine sort of the space of all files and the green ones are benign wear and the red ones are malware. And you kind of imagine that similar files are closer together. So the malware is sort of clustered in the middle. If you have signature-based antivirus, then your analysts are sort of writing these rules to draw these circles around specific malware samples. And the problem that has kind of besieged the whole industry in the last 10 years or so is that malware authors will just mutate their malware. They'll create variations and it just overwhelms analysts. And the process of trying to update your signature and capture these new variations without capturing any false positives is just very tedious and it's very laborious because it's all done by humans. So ML's value is to kind of fudge the lines a little bit. Kind of generalize and take those original signatures and then just kind of learn from them. So it kind of helps to counteract this main tool that malware authors are using. So it's actually very well suited for this challenge. But it's important to keep in mind that without the signatures there would be no ground truth and there'd be nothing for machine learning to learn. So ML is not really all you need. You still need humans in the loop. So some other things that we've added to kind of help ML are whitelists for critical system files that you might not want to fp on like explorer.exe, blacklists for things that you just know or malware, and what I'm calling fuzzy whitelists. So we have sort of a reputation system where if we've seen a file in a lot of different computers, we've seen it a lot of times, we sort of make the assumption that that means it's not malicious. So we sort of give it a probabilistic it's probably good. When people at the company asked, well why can't machine learning do all this? We were like, well math and stuff. Basically, if you imagine like a neural network or something, the number of layers that it has, the number of neurons that it has, that's a certain capacity that it has to learn patterns. And there's no reason to waste the capacity trying to learn certain patterns of files that you never want to convict or files that you always want to convict when you can just do these kind of dead simple whitelists and blacklists. So the tone kind of changed from machine learning is all you need to kind of say, hey you know that project you got over there? You know how you could make it even better? Put some machine learning on it. And that project, that would be great with some machine learning. If you don't know, this is from Portlandia and this is an episode where they're making fun of how everyone in Portland thinks things look better with birds drawn on them. The best part about this episode is the ending when a bird actually flies into her shop and she's totally disgusted about it, which is sort of what happened to us. Now machine learning is alive and well at Sophos, but the reality of trying to deploy it in a production environment was a lot uglier than anyone envisioned. Which brings me to scalability. So the main issue with scalability is just that when you're working with a product that's deployed across millions of endpoints and it's scanning tens of thousands of files on each endpoint every day, that's a very different data set than the one that you trained and tested on, which maybe tens of millions of samples and it's a static data set that you're kind of optimizing over and over on. So a lot of just different things sort of manifest. So just as an example to kind of illustrate this, was anybody born on August 12th? Okay, so it's nobody's birthday today in this room. If you were to be very naive, you might take that statistic and say well there's nobody born on August 12th, which presumably is wrong, presumably roughly one 365th of the population is born on August 12th. But so that kind of demonstrates a small sample size issue and the same thing occurs with malware analysis. If you have a small malware sample data set, then those kind of tiny pieces of the pie just kind of get disappeared. So it may be a small variation of a family or some critical system files. So what we've done is we set up this testing omelette. So a model will get like three million files to work on, to train and test on, and if it reaches a certain threshold, then it gets like five million. And if it does well there, it gets 10 million and you keep ramping it up. That way you don't waste training time on large amounts of data. You start with three million and that's a test you can run relatively quickly. If that goes well, then you keep amping it up. So in general, scalability just kind of amplifies things. We also have to strive for ridiculously low false positive rates because we're running our model so often. And the last thing that sort of gets amplified is just costs. So if you're using something like AWS, which you should be, it's great because whenever you ask for resources, you get resources. The problem with AWS is that whenever you ask for resources, you get resources. And you don't always know how much you're really asking for. So in one situation, we had a bunch of data on S3 that we were trying to move into Glacier to save money. And we neglected to notice that there was a small transaction fee to every move from S3 to Glacier and we ended up blowing through our monthly budget in a couple of days. So that was our bad. So another thing that we did to address scalability was a switch from DynamoDB to Redshift. So DynamoDB, if you don't know, is it's great for row-based data if you want to pull out individual rows that works. But it's not great for aggregate statistics. So you can't really run SQL queries over large datasets because you have to pull down the entire table every time you want to do that. Redshift is column-based, so it supports that much more quickly. So one other thing I'll touch on is just the actual going into production. So one thing that we constantly have to deal with is what we're calling concept drift, which is also referred to as model decay. But essentially what happens is that you put a model in production and even over the course of a month the effectiveness of the model starts to drop perceivably. This is just because new malware is being released and new benignware is also being released. And so to actually compensate for this we have to train with sort of a handicap. So we have to train on older data and test on newer data to kind of predict how our model will decay. There are also language discrepancies. So all of our research is done in Python and we've literally had to rewrite some parts of Keras and C++ to get it to run on Windows machines. Releasing updates is tricky because you don't really know what you're releasing. I mean you know certain ways of changing your model and maybe the architecture has changed. You don't know how its behavior is going to change in production. And it's not like releasing a code update where you can be pretty certain about what's going to happen. This is kind of a black box and so that's a little scary. And that kind of ties into cultural because now you have data scientists who are in like the production bloodstream which is sort of bad because we don't really know what we're doing there. There's been a lot of knowledge that we've had to gather along the way like the things I've talked about about scalability. And so there's been a lot of learning that we've had to do as a data science team and we've all had kind of had to take off our data science hats and put on engineering hats. And then kind of one anomaly that came up, someone wrote a Hello World program in C++ and they compiled it with certain settings and our model said it was malicious. And you know management came to us and they were like well why is that happening? And we were like well math and stuff. But they wanted to fix right? It's like that's not good. And in a production system if there's something that's impacting customers you do a hot fix. So you kind of surgically fix the code that's causing that problem without changing anything else. You can't really do that in machine learning. So if you imagine a model like a pinball machine and you imagine that the model is all these pegs and samples were running through it are these balls and the way the classification that is producing are these bins at the bottom, you might run a bunch of samples through the model and it might get most of them right. But there's that annoying little benign wear ball that we've fped on. And what are we supposed to do about that? Like the position of all these pegs has been optimized through an automated process over hundreds of hours. Like what are we going to do about that? We could pick some peg and say well I wonder what happens if we twist it and run it all again. And okay yay we got it but oh wait now we have a false negative. So you know there's really no way that our puny little human brains are going to like look at all these pegs and be able to figure out the right optimal solution that works for all samples. So if you're afraid of robot overlords you might want to be. But the actual takeaway here is that if you go back to the original thing that I was talking about machine learning by itself doesn't always work. It makes mistakes and when it's fudging the lines sometimes it fudges the lines in the wrong direction and it actually loses something that a signature would have caught. So robot overlords still kind of need us. Cool. Okay I have six minutes to demo something and answer questions. So AI total is an internal tool that we use for testing out our models. This is everything that it's built with. Some of the interesting stuff we're starting to use SageMaker which I don't know if people are aware about but you put your model in a docker and docker container and it kind of handles everything else which is pretty great. Okay so I wasn't able to connect to wi-fi which I'm not blaming anybody but maybe a DEF CON related issue. So I'm not able to give you a live demo but this is AI total. We've done a lot of work on different things other than PEs so we have I don't know if you can read the top but we have HTML models, URL, PE, Office Docs and Android. And so the way that this works is that you can actually live query the models that we have. So this is our URL model and you can actually type in URL here and get instant feedback from the model and this allows us to kind of just get a little sense before we push things out into production like how's it actually going to behave. For the other models for PE and Office Doc you can actually upload a file but for URL because it takes text you can just type it in. So yeah that's what I have for you guys thank you and any questions.
 And without further ado, here's Alex with his talk.  Thank you.  Thank you for coming on a Sunday morning.  We are gathered here today to talk about AI.  My talk is entitled AI DevOps Behind the Scenes of a Global  Antiviruses Machine Learning Infrastructure.  And just a meta note real quick. In making this talk,  I was, I'm going to be talking about work that was performed by many people on my  team. So I actually interviewed about four people.  And this work spans about two years. And I have about 20 minutes to tell you  about it. So this is going to be a very kind of  distilled, highlight reel of our story.  Who am I? So I'm doing a lot of things at the team. I do a lot of web dev. I do a  lot of AWS backend stuff. Some ML research. I've done some Android stuff.  That's a picture of my face and my contact information.  This is a picture of my cats, which you have three seconds to enjoy.  This is the data science team at Sophos. Maddie Shihapa is actually in the crowd  today. And Hilary Sanders, Constantine  Berlin and the two mats at the bottom contributed  a lot more than I did to the work that I'm taking credit for today.  I also have you on here because we're hiring. So if you have any expertise  at the crossroads of cybersecurity and data science, we'd love to hear from you.  Okay, so in the beginning there was signature-based  antivirus. And by beginning I mean 2017. That's about the time that the data  science team was added to Sophos. And we were the first kind of introduction  of ML to their antivirus product. And there were a lot of just initial  misconceptions. The big one being that machine learning is all you need.  So this is something that we kind of had to convince people was  wrong even though we are the data scientists.  And so just to kind of give you guys a way of thinking about this.  If you imagine sort of the space of all files and the green ones are benign  wear and the red ones are malware. And you kind of imagine that similar  files are closer together. So the malware is sort of clustered in the middle.  If you have signature-based antivirus, then your analysts are sort of writing  these rules to draw these circles around specific malware samples.  And the problem that has kind of besieged the whole industry in the last 10 years  or so is that malware authors will just mutate their malware. They'll create  variations and it just overwhelms analysts. And the process of trying to  update your signature and capture these new variations without capturing  any false positives is just very tedious and it's very laborious because it's  all done by humans. So ML's value is to kind of fudge the  lines a little bit. Kind of generalize and take those original  signatures and then just kind of learn from them.  So it kind of helps to counteract this main tool that malware authors are  using. So it's actually very well suited for this challenge.  But it's important to keep in mind that without the signatures  there would be no ground truth and there'd be nothing for machine learning  to learn. So ML is not really all you need. You still need humans in the loop.  So some other things that we've added to kind of  help ML are whitelists for critical system files that you might not want to  fp on like explorer.exe, blacklists for things that you just  know or malware, and what I'm calling fuzzy whitelists.  So we have sort of a reputation system where if we've seen a file in a lot of  different computers, we've seen it a lot of times, we sort of make the assumption  that that means it's not malicious. So we sort of give it a probabilistic  it's probably good. When people at the company asked, well why can't machine  learning do all this? We were like, well math and stuff.  Basically, if you imagine like a neural network or something, the number of layers  that it has, the number of neurons that it has, that's a certain capacity that it  has to learn patterns. And there's no reason to waste the  capacity trying to learn certain patterns of files that you never want to  convict or files that you always want to convict when you can just do these  kind of dead simple whitelists and blacklists.  So the tone kind of changed from machine learning is all you need to kind of  say, hey you know that project you got over there? You know how you could make it  even better? Put some machine learning on it. And that project, that would be great  with some machine learning. If you don't know, this is from Portlandia  and this is an episode where they're making fun of how everyone in Portland  thinks things look better with birds drawn on them.  The best part about this episode is the ending when a bird actually flies into  her shop and she's totally disgusted about it, which is sort of what happened  to us. Now machine learning is alive and well at  Sophos, but the reality of trying to deploy it in a production environment was  a lot uglier than anyone envisioned. Which brings me to scalability.  So the main issue with scalability is just that when you're working with a  product that's deployed across millions of endpoints and it's scanning tens of  thousands of files on each endpoint every day, that's a very different  data set than the one that you trained and tested on, which maybe  tens of millions of samples and it's a static data set that you're kind of  optimizing over and over on. So a lot of just different things sort of manifest.  So just as an example to kind of illustrate this, was anybody born on  August 12th? Okay, so it's nobody's birthday today in  this room. If you were to be very naive, you might  take that statistic and say well there's nobody born on August 12th,  which presumably is wrong, presumably roughly one  365th of the population is born on August 12th.  But so that kind of demonstrates a small sample size issue  and the same thing occurs with malware analysis. If you have a small  malware sample data set, then those kind of tiny pieces of the pie just kind of  get disappeared. So it may be a small variation of a  family or some critical system files. So what we've done is we set up this  testing omelette. So a model will get like three million files to work on, to  train and test on, and if it reaches a certain threshold, then it gets like  five million. And if it does well there, it gets 10 million and you keep ramping  it up. That way you don't waste training time on large amounts of data. You start  with three million and that's a test you can run relatively quickly.  If that goes well, then you keep amping it up.  So in general, scalability just kind of amplifies things.  We also have to strive for ridiculously low false positive rates  because we're running our model so often. And the last thing that sort of gets  amplified is just costs. So if you're using something like AWS,  which you should be, it's great because whenever you ask for resources, you get  resources. The problem with AWS is that whenever you  ask for resources, you get resources. And you don't always know how much you're  really asking for. So in one situation, we had a bunch of data  on S3 that we were trying to move into Glacier to save money.  And we neglected to notice that there was a small transaction fee  to every move from S3 to Glacier and we ended up blowing through our monthly  budget in a couple of days. So that was our bad.  So another thing that we did to address scalability  was a switch from DynamoDB to Redshift. So DynamoDB, if you don't know, is  it's great for row-based data if you want to pull out individual rows that works.  But it's not great for aggregate statistics. So you can't really run  SQL queries over large datasets because you have to pull down the entire table  every time you want to do that. Redshift is column-based, so it supports  that much more quickly.  So one other thing I'll touch on is just the actual going into production.  So one thing that we constantly have to deal with is what we're calling  concept drift, which is also referred to as model decay.  But essentially what happens is that you put a model in production and  even over the course of a month the effectiveness of the model  starts to drop perceivably. This is just because new malware is being  released and new benignware is also being released.  And so to actually compensate for this we have to train  with sort of a handicap. So we have to train on older data and test on newer  data to kind of predict how our model will decay.  There are also language discrepancies. So  all of our research is done in Python and we've literally had to rewrite  some parts of Keras and C++ to get it to run  on Windows machines. Releasing updates is tricky because you don't really know  what you're releasing. I mean you know certain ways of changing your model and  maybe the architecture has changed. You don't know how its behavior is going to  change in production. And it's not like releasing a code update  where you can be pretty certain about what's going to happen.  This is kind of a black box and so that's a little scary.  And that kind of ties into cultural because now you have data scientists who  are in like the production bloodstream which is sort of bad because we don't  really know what we're doing there. There's been a lot of knowledge that  we've had to gather along the way like the things I've talked about  about scalability. And so there's been a lot of learning  that we've had to do as a data science team and we've all had kind of had to  take off our data science hats and put on engineering hats.  And then kind of one anomaly that came up, someone wrote  a Hello World program in C++ and they compiled it with certain settings  and our model said it was malicious. And you know management came to us and  they were like well why is that happening? And we were like well math and stuff.  But they wanted to fix right? It's like that's not good.  And in a production system if there's something that's impacting customers  you do a hot fix. So you kind of surgically fix the code that's causing  that problem without changing anything else. You can't really do that in  machine learning. So if you imagine a model like a pinball  machine and you imagine that the model is all these pegs and samples were  running through it are these balls and the way the classification that is  producing are these bins at the bottom, you might run a bunch of samples through  the model and it might get most of them right.  But there's that annoying little benign wear ball  that we've fped on. And what are we supposed to do about that?  Like the position of all these pegs has been optimized through an automated  process over hundreds of hours. Like what are we going to do about that? We could  pick some peg and say well I wonder what happens if we twist it  and run it all again. And okay yay we got it but oh wait  now we have a false negative. So you know there's really no way that our  puny little human brains are going to like look at all these pegs and be able  to figure out the right optimal solution that works for all samples. So  if you're afraid of robot overlords you might want to be.  But the actual takeaway here is that if you go back to the original thing that  I was talking about machine learning by itself doesn't always work. It  makes mistakes and when it's fudging the lines sometimes it fudges the lines in  the wrong direction and it actually loses something that a  signature would have caught. So robot overlords still kind of need us.  Cool.  Okay I have six minutes to demo something and answer questions.  So AI total is an internal tool that we use for testing out our models.  This is everything that it's built with. Some of the interesting stuff we're  starting to use SageMaker which I don't know if people are aware about but you  put your model in a docker and docker container and  it kind of handles everything else which is pretty great.  Okay so I wasn't able to connect to wi-fi which I'm not blaming anybody but  maybe a DEF CON related issue. So I'm not able to give you a live demo but this  is AI total. We've done a lot of work on different things other than PEs so we  have I don't know if you can read the top but we have HTML models, URL, PE,  Office Docs and Android. And so the way that this works is that you  can actually live query the models that we have. So this is our URL model and  you can actually type in URL here and get instant feedback from the model and  this allows us to kind of just get a little sense before we push things out  into production like how's it actually going to behave.  For the other models for PE and Office Doc you can actually upload a file but  for URL because it takes text you can just type it in.  So yeah that's what I have for you guys thank you and any questions.
DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_0_1499.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_1500_1583.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_1584_2231.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_2232_3203.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_3204_5039.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_5040_7583.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_7584_11783.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_11784_12671.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_12672_12827.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_12828_18611.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_18612_18875.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_18876_19271.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_19272_20099.png DEF_CON_26_AI_VILLAGE_-_Alex_Long_-_AI_DevOps_Behind_the_Scenes_of_a_Global_AV_Company_eNpHytykSuw_20100_20994.png