AppSec Village - The DevOps & Agile Security Toolkit
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Waldrop, David
The DevOps & Agile Security Toolkit - In this talk, we will look at integrating security into Agile and DevOps. We will discuss strategies, training, tools, and techniques that will let your organization move quickly while doing so safely.
 Hello and welcome back. Hope you're looking forward to another fantastic talk here at the Apssac Village for DEFCON 29 in safe mode. It's important to remember your 3-2-1, even though we are separate from each other and socially distanced, your cohabitants will really appreciate that one in there. Just take care of yourselves, get some good sleep, get some good food, make sure you take a shower. For our next talk, we've got David Waldrop talking about the DevOps and Agile Security Toolkit. After 25 years in the application development, David moved to Information Security just over 7 years ago. His passion is helping developers write secure code. He's currently an Information Security Advisor supporting the application development community at his employer. Let's put our hands together and welcome David Waldrop. Hi and welcome to the Apssac Village. I hope you're having a great DEFCON 2020, albeit in safe mode. My name is Dave and for the next 30-45 minutes, we're going to talk about the DevOps and Agile Security Toolkit. Now before I get started, I have to make the standard disclaimers. First, this presentation and any comments that I make are my own and do not necessarily represent those of my employer. Secondly, I will mention a few example products, some are commercial, some are open source. These are not necessarily meant as recommendations. They are mentioned to serve as a starting point for your own investigation and review. These are things that I've run across in my journey, so take that as it is. Let's start with the two terms that are on everybody's buzzword bingo sheet these days. DevOps and Agile. We need to get definitions in place for the duration of this talk, so we have an idea that we're all dealing with the same playing field. Agile. Agile focuses on continuous iterative development and testing. Software is developed in smaller increments and then is integrated for final testing and delivery. Scrum, Kanban, and XP Extreme programming are examples of Agile approaches. On the other hand, DevOps focuses on the collaboration between AppDev and Operations. The focus is on deploying code to production in a fast manner, usually automated. So you can see from the definitions that these are two separate concepts. However, many organizations tend to combine them into one initiative. Very, very rarely do you find a shop doing DevOps and not doing Agile or vice versa. So no matter whether they start with DevOps or they start with Agile, the one piece that's usually late to the party is security. But no matter how late security is invited to the party or whether we have to crash the party ourselves, we usually can still bring valuable resources without slowing things down or at least without slowing them down a lot. And that's what we're going to talk about through the rest of this discussion. So for the past few years, I've worked with several different Agile and DevOps teams. My integration with this started as a pilot program where I was the information security representative that was assigned to help introduce Agile to our organization. More recently, I've implemented the information security integration with the Agile and DevOps teams across our organization. During this session, I'm going to share with you some things that really worked well and some things that just didn't seem to work at all. So hopefully you can take something away from this and apply it to your own organization. This particular slide will give you the map of what we're going to talk about for the rest of this discussion. We're going to talk about Agile staffing. We're going to talk about the eight simple questions. We're going to talk about the security champions program and then go into developer training, which is my favorite part of my job. Security in the software development lifecycle. And then we're going to hit the hat trick of securing your code, static code analysis, dynamic code analysis and open source analysis. And then finally, we're going to wrap things up by automating the heck out of it. So let's start with Agile staffing. Sherman set the way back machine for one when I started with Agile. One information security person was assigned to one Agile team. That security person would be integrated into that team and attend every ceremony, every meeting, everything, every day. That didn't work. There are a couple of reasons I think that that didn't work. First of all, agile teams tend to multiply. You can go from a proof of concept with one team to having close to 30 teams in less than two years. Most organizations do not have the information security bandwidth to support that type of staffing model. In addition, the scrum masters tend to treat every member on the team as though they can perform all tasks. And when you're dealing with a team of all developers, that makes perfect sense. This model tends to fall apart though when you put information security, QA, middleware, any other infrastructure or shared services team as a part of the agile team. What happens is in the agile process, their hours count against your available development hours. But the infrastructure people, the information security people do not have the skills or the capabilities to do some of the development. You see that there's a mismatch. You're given hours, but you're not allowed to use them. It caused a lot of problems for some of our scrum masters early on. Again, we realized this was not the way to do it. I ended up coming up with a two tiered approach for trying to integrate information security into the agile team. First, at a high level, at the enterprise level, we will have an architect, an information security architect, and they will inject into those high level projects and basically set up guardrails and guidelines. So as projects are being devised at the visionary level, they can jump in and try to help guide them into making more secure decisions at that high level. Now, when you get down into the weeds though, each agile team does need to have somebody they can rely upon for information security. And so what we ended up doing is we created named resources within the information security team that are assigned to agile teams. Now it's not like the previous model. You're not assigned, you're not sitting with them, you're not assigned to go to every meeting. We're going to talk about how we scaled that back, but we ended up finding a way where based on where information security can add the most value, we attend those ceremonies and only those ceremonies. So at the end of the day, each information security representative can support between six and eight different agile teams. That is a model that is certainly sustainable. So what is a little different? Well, first of all, we make sure that every agile team has a named resource that is responsible for information security for them. So this gives them a named contact if they ever need something they know exactly who to call and where to start that quest. Secondly, we attend or the agile reps attend the backlog refinement sessions. These are meetings that happen once every two weeks before the next sprint and it defines what they're going to work on in that next sprint. It's a perfect opportunity to jump in and say, hey, you're planning this work. We need to think about blah, blah, blah from an information security standpoint. We can get in just ahead of when development starts. It's a perfect time to bring up security issues, make recommendations and answer questions. And then finally, we attend the quarterly planning agreement meetings where the vision is cast for the next quarter and we can make sure that we're on their radar for things that we think we need to be a part of. So this is kind of a high level diagram. The purple arrow at the top is the information security architect. That particular person, as you can see from the diagram, sets guardrails for enterprise planning projects. All of the red arrows are the different places where our individual security representatives can or may inject into the agile teams. We don't inject in all those places, but there are places where we can and we give them that option to call us if they're needed. So let's move on to what I like to call the eight simple questions. This particular concept came up as a result of assigning our agile reps to their teams. We found that the leadership within the agile teams really wasn't sure when to call us. And since we weren't sitting there in every meeting, there was a possibility that something could come up that might fall off the radar that we might not get contacted for that they may not even think is important. So we decided that we need to give clear guidelines to the scrum masters and product owners as to when they should include information security. That's how I created the eight simple questions. So basically these eight questions are a guideline for that leadership team. And if they answer yes to any of them, they need to make sure to include their information security representative in those discussions. These are the questions as I currently have them. We're going to talk about that in just a bit because they change over time. These are not one and done kind of things. Am I dealing with any sensitive information? Now information that is sensitive, it's going to mean something different for each one of our organizations. If you're working in healthcare, if you're working with banking information, if you're working with retail with PCI information, you will be able to find what sensitive information means for your organization. Am I sending or receiving data outside of the company? Is there a new vendor involved? Am I going to introduce new software systems or libraries, even open source and remember open source because we're going to talk about that? It's a unique challenge. Do I need access to secrets such as passwords, keys or certificates? Will I need the ability to use single sign on? Am I going to leverage the cloud? Will I need updated access roles? So these eight questions are what we are currently using as a guideline for the interaction between the agile team and their information security representative. As I mentioned, the exact questions are going to differ on your organization. The questions themselves will actually change over time. I started, I think, with seven questions and then we had several issues come up with access roles and who to ask and blah, blah, blah. So we added a question for that too. So don't be afraid to take a stab at it and massage it and change it. That's the thing about agile and DevOps. It's changing all the time and we as information security people need to be able to change our stuff as well. So the basic idea is this. Create a list of questions usually 10 or less, something that's going to fit easily on a PowerPoint slide or a single piece of paper so they can take it to meetings with them. Give that to the agile team leadership and they have a guideline. They now have a concrete way of knowing when they need to work with information security. So we found another advantage to this approach as we onboarded additional information security people to become representatives for the agile teams. A lot of these folks were pretty new to the organization and they didn't really know when they needed to inject themselves into their teams. They didn't know what they needed to listen for in the backlog refinement sessions. So we use these simple questions on our own people. It gave them guidelines and hints of what to listen for when they're working with their agile teams. So this kind of became a win-win on both side of the coin. So I think this was a great tool that we've been able to use. Another tool that I honestly have to thank Jim Manneko for. Jim if you're out there thank you for this. This was a recommendation he made to me after one of our training sessions and that is the security champions program. A security champions program is a team of application developers. We get one from each agile team and these developers have expressed an interest in information security or secure coding or just security in general. This is a voluntary program. The members of the team volunteer to come to the security champions program. And we typically meet quarterly. I provide them lunch when we're in the office thanks 2020. I look forward to getting back to the office and having lunch with these guys and gals. They're a great team. So the team has this following charter. We assist the agile teams in terms of application security. We serve as a communication conduit between the agile teams and information security. And then we meet quarterly to share experiences. We ask questions. We share knowledge. It has had some incredible benefits. First of all security has a relationship with at least one person on every agile team. I'm not saying we have a mole. What I am saying is sometimes we hear about things long before it percolates through the management chain. I might get a phone call about hey we're thinking about using this open source library. Is that cool. We take a quick look at it. We approve it. By the time the formal approver gets into my hands we've already done it in their coding. So it gives us a great communication conduit and that helps me make things faster. I may have not mentioned this before. I hopefully I have relationship is key. You're going to hear about relationship throughout this entire discussion. We build relationships with those agile teams. It starts with having an assigned rep. It moves into a different relationship level when we have a security develop security champions team. Secure coding training takes it to yet another level. And you'll see why in just a few minutes as we talk about the developer training. But it's about building relationships. It's about having fun while we're doing the right thing for our organizations. So so far the team has done a lot of really cool things. We've reviewed vendors and products both on the development side and on the information security side. We've selected our developer training programs for the last two years based on the input from the security champions team members. We've done a bunch of PSEs together. We've cross consulted. Now this was kind of an unexpected benefit is where if I had a team that was very well versed in one technology and they mentioned that in a security champions lunch, another team might reach out going, hey, we're about to do that. Can we stand and talk? So it actually starts to cross pollinate some security knowledge and some technical knowledge between different application teams. And then finally, it's served as a candidate group for application security openings within the infosec team. The software development life cycle. So as we started to roll out security, a lot of people have asked us, how does this fit into our SDLC? So I ended up creating a plan that maps information security phases to the software development life cycle as it currently exists. It is not a perfect fit, but I think it is a good representation of how information security can bring value to each step of the SDLC. So this is what I like to call my dark side of the moon diagram. Those of you old enough to remember probably recognize the album cover. So if you take a look on the right, we have the software development life cycle from concept into design up into coding testing a QA, QA and deployment and then post deployment at the very top. Now, if you look on the left side, you'll see that it matches up pretty closely with security knowledge, security design, secure coding techniques, security testing and then security response. The each individual layer of the pyramid, I think are deliverables that security can bring that help take that concept on the left and apply it to the SDLC phase on the right. And we're going to walk through those real quick. Training is the base of the pyramid. I think training is foundational to everything above it. If you don't have a solid foundation of training for your application developers, everything else is at risk. So this particular approach, we're going to talk about in depth again, because it's my favorite part of what I do. It includes things like lunch and learns, outside speakers, conferences like you're attending right now, video training and things like that. The next level up is threat modeling threat modeling, the application team and InfoSec. We sit together, we work together to examine the application design, the system flows, whatever they can bring to us that they have available in that design phase. We look, we try to work together and identify potential vulnerable vectors and try to figure out a way that we can mitigate them. Now threat modeling, there are books upon books written about threat modeling. And there are people in my organization on my InfoSec team that are much better at it than I am. And they can actually create incredible documents and spend hours and days and days and days doing this. I think you need to start out small. I think it's about baby steps at first. Start with a discussion. This can be a one or two hour discussion where you have those deliverables from the design phase and you whiteboard them. And you come out of there with to dos or questions or things to research. It doesn't have to be this big, heavy weight overbearing approach to threat modeling at first. It needs to be a conversation that gets back to relationships. You need to have those relationships in place so you can have those conversations. The third level up is static code analysis. We're going to talk about this in much more detail later. But most tools are designed to identify potential vulnerabilities within the code. Now static code analysis actually looks at the code and looks for patterns within the code without running it. Now we're going to see a little bit later why that's important. I think I went the wrong way. There we go. Dynamic code analysis is the next layer up. And that is very similar to the previous layer which was static code analysis. But instead of just looking at the code, the code is actually executed and tested for vulnerabilities. So like I said, these two pieces are very, very similar. One is looking at patterns with the code. The other is actually exercising the code and looking for issues. The next is pin testing. Pin testing can be performed by your internal team if you have the skills or resources or you can use an external firm. This is usually a greater scope than dynamic testing. It will actually look at things not just in your application but system wide for vulnerabilities and attack vectors. And usually there's a person involved. It's not just an automated exercise. We're going to see a little bit more when we talk about dynamic testing, the difference between that and pin testing. And then finally, when the code is in deployment, you need to have a vulnerability management process that you can classify, assign, and track identified issues. Right now, InfoSec meets regularly with a representative from each application team to review these and to make sure that they're working and they're on track. Again, those relationships are absolutely key to making this work. So now we are getting to what I consider my absolute favorite part of my job. And that is developer training. If you remember back to that pyramid we just looked at, this is the foundation for everything that information security does with app dev. When I came in to information security about seven years ago, I started a developer training program because I realized that the developers really are your last best hope for defense when it comes to application security. If your application is written correctly, then everything you throw in front of it, application firewalls, everything like that, that's gravy. That's nice, but you've got a really tight application and that's where real security takes place. The problem is, I was a developer for roughly 25 years before coming to information security and nobody taught this stuff. This is stuff you learned on your own. If you had an interest in it, and it wasn't universally available to all developers. So I started small. I started with Lunch and Learns. It was really something that, hey, I think this is important. Let's try it. Let's see if it works. And it really did. We would end up having Lunch and Learns about every other month. I would create a class on a specific topic. My first one was cross-site scripting, a lot of fun. It's neat when you can show people cross-site scripting examples, even if you just run Webgoat or something like that. They get excited. They kind of dig it and they start trying it on their own stuff. The content usually lasts about 40 or 45 minutes, a lot like this session. And then we leave 10 to 15 minutes for open discussion and questions. Now the key here is make it fun. It has to be fun. I do things like I'll go out and I'll buy cookies or some kind of treat. So we can all sit around, eat something together. There's usually some kind of door prize or door prizes. And they don't have to be expensive. You can just go down to like five below and get a couple of wireless Bluetooth speakers. And people dig it. It's a lot of fun. Again, making it fun is absolutely key to a successful Lunch and Learn program. So once we got that up and running, I decided we needed to take this up a notch. There is a level of training that I couldn't provide that I think our developers really, really would benefit from. And in order to do that, I knew that I had to reach out to industry experts. And what we ended up doing is we were able to get commitment from management both on the IT development side and on InfoSex side that we could do one or two days where most, if not all the developers will attend a training event. Now here's a hint. If you are going to try this and you're doing Agile, put it in your innovation sprint. If you put it in the innovation sprint, which is usually the last sprint of the quarter and the developers have a little more leeway of what they work on during the innovation sprint, you'll very likely get more participation than if you step on a middle of a sprint where there's actual deliverables do. So keep that in mind. I was really, really lucky. My very first interaction with this program, I was able to secure Jim Manneco and he came in and made an incredible impression on our development teams. Jim's been back, I think three times now. Always a pleasure. Rich Mogel's been by twice. We've had a recent one with Ron Parris. Dr. Phillip DeRique, phenomenal instructor. I just took his OAuth class. So if you're looking for OAuth, wonderful class. But we've got him coming in this fall and I'm looking forward to that. The reason I actually want to call these gentlemen out by name is I think it's important to be grateful. I am very grateful for everything they've done for me and my development teams. And it's important to be grateful to those who help you along the way on this journey because none of us make it alone. So gentlemen, thank you very much. If you see this or hear it later, your training, your mentorship has meant a lot to me and my development teams. So one cool aside is I mentioned this earlier. The security champions team has been involved in the selection of the instructor for the last two years. This gives your application development teams a voice in the training. And we've seen that participation has actually increased. We had great participation before then. But when the developers actually had a voice in picking, who's coming in and what they're going to talk about, we're finding that it's much more applicable to what's on their roadmap. It's much more applicable to the skills they're hoping to get. And participation goes beyond what I could have ever hoped for. So again, we're starting to see some of these initiatives are starting to weave together and starting to build kind of a program. It's kind of cool. So this next one, I really wish I was doing this right now. Take a developer to DEF CON. We had management agree to provide scholarships to developers that were selected by the information security team. So we would give them the money to buy their badges. So they wouldn't have to go and get money out of their own bank account. So each year, I've been able to bring up to three developers to DEF CON. That's been awesome. It has been just a phenomenal experience. They've gained a ton of knowledge and a ton of awareness by DEF CON. One of the things that I found really interesting was I really advise them not to restrict their sessions to things that they think will apply to their job. I did that my very first DEF CON and I wasn't really sure I was going to come back to be honest with you. The next year, one of my coworkers strongly encouraged me to go. Thank you, Barry. And encouraged me to go and find a few fun things that just had nothing to do with my immediate job. The interesting thing is they ended up having more to do with the things that were coming down the road in my job than I ever could have imagined. So I fully encourage each one of these developers that I bring to DEF CON to experience the full DEF CON experience. Go to all the sessions. Try some of the capture the flags. Try whatever you want. Have a good time because it's really important for them to be immersed in the DEF CON experience. So like I said before, they come back with a changed perspective. It's a lot like that scene in The Matrix where that thing crawls out of Neo and he goes, my God, that thing is real? They understand now that the security is not just something that we're teaching and preaching about within InfoSec. That this stuff has real world application. And it's that realization that makes them a better developer, a more secure developer going forward. We've seen a trend where a lot of the folks that we've taken actually, I think all but one of the developers that we've taken to DEF CON have eventually become security champions members. And then more recently, just in the last couple of months, one of those developers, I think he was the first one I took to DEF CON with us, ended up joining the information security team on a full time manner. And Shane, if you're listening, welcome. I'm really glad that you're part of the team. So there's a couple of other fun activities that we do in developer training. Again, got to keep it fun. Everything has to be fun. You learn better when you're having a good time. I mean, look at DEF CON, we're all having a great time. And we're learning stuff. So a couple of DEF CONs ago for reasons of work, I couldn't make it. It was the only DEF CON since coming over to information security that I was not able to go to. And I must admit, I was bummed. So what I decided to do was I was going to have my own little DEF CON. And we decided that, okay, let's just take people along for the ride. So we identified five videos from previous DEF CONs and we had a lunch and learn for five days that week where we watched it and we talked about it. We had drawings for DEF CON related swag or items or giveaways. We just kind of had a good time. It was kind of built up as, hey, take a break, come bring your lunch, let's hang out, watch something from DEF CON, talk about it. You might win a t-shirt, you might win some stickers, whatever. And people really enjoyed it. It was a lot of fun. As a matter of fact, I'm probably going to do one this fall for the folks that didn't get to enjoy what you're doing right now, what you're enjoying right now. And then finally, I have an annual thank you lunch and learn. This was a tradition I started the first year we did lunch and learns. It's always the last lunch and learn of the year. And I typically will go out and I'll purchase quite a few giveaways. They don't have to be expensive. We do things like, say, the Bluetooth speakers, boxes of candy. We've done some Christmas mug sets, you know, just weird things, but things that are kind of fun. And we do a shorter lunch and learn followed by a session where we just eat some good food and we have some drawings. And I have an opportunity to thank my development community for the support of the lunch and learn program and for the support they've given to information security. I think it's important to say thank you. And this has been just an incredibly popular lunch and learn every year. It's a lot of fun and I highly recommend it. So here are the keys to a successful lunch and are successful training program. Make it applicable and give the developers a voice in the training. And usually if you give them a voice in the training, it will be applicable just because they're going to tell you what they need. I think I've mentioned this 1427 times, but make it fun. It's got to be fun. Share how much you care about this stuff. If you're engaged, if you're passionate about it, people are going to feel that and they're going to gravitate to you and gravitate to that topic. And then finally build relationships. Listen, reward and always express gratitude along the way in this journey. So we're getting into the next phase of the toolkit. And this is what I like to call the analysis hat trick. Three types of analysis that if we implement all three, we're going to have a much more secure code base in production. And the first is static code analysis. You might remember that from our SDLC pyramid. So static code analysis scans the code without running it. It looks for known patterns of vulnerabilities. You can usually select what you scan for. Some will allow you to scan for PCI, blah, blah, blah. I almost always scan for OWASP top 10. If you're not familiar with OWASP or OWASP top 10, I strongly recommend that you check it out. Some of the deliverables from that organization are absolutely stellar. I've been a member for I think seven years now just after joining the InfoSec team. And I've leveraged a ton of their stuff. So great organization hats off to you if you're out there. In an ideal world, the static code analysis tools would only report the confirmed vulnerabilities. But even if you tune the daylights out of this thing, you're still going to get false positives. And that becomes a problem. Because originally when we first set up this process, I was the only one that could review the false positives and whitelist them or wave them as it were. That slowed things down immensely because that was not my full-time job. That was something else on top of my full-time job and possibly another full-time job some days it felt like. So that being said, I decided to leverage the security champions. These were folks that had some basic knowledge of security. They knew their applications better than I ever would. And they had a passion for making secure code. Seemed like a perfect fit. So what I ended up doing is I ended up giving several members of the security champions team the ability to review and wave false positives. And that has worked out great. The end result was we've had a lot more throughput. We've been able to address false positives much more quickly. And the adoption of the static code analysis tools have actually increased as a result of not having that delay. So I also see static code analysis as an extension of developer training. A lot of these tools have an IDE component. So for example, I used the Spring Tool Suite Eclipse. And you can use the plug-in and execute your scan from within your developer tool. And it will take you straight to the line that's got the problem. And in some of these tools you can even have recommendations pop up in the bottom of one of the windows that says, hey, you might try blank, blank, and blank to address this vulnerability. If developers are doing this while they're developing, not waiting for the build process, it becomes a teaching tool. Because as a developer uses this, he or she is going to identify and learn that, oh, I need to check my input fields to make sure they're the right type and right size. Okay, you do that enough times. It becomes second nature. So this becomes not only a way to secure your code, but a teaching opportunity as well. So studies have shown that study code analysis can identify up to 80% of vulnerabilities. That's pretty awesome. If I could tell you that you can do one thing and remove 80% of vulnerabilities, I think that'd be worth doing. So this might be something you might want to take a look at. Here's some example products. Here's three products along the top that are commercial. OWAS has an entire list of code analysis tools at that link right there. Again, not meant as recommendations, but these were tools that I've come across in my journeys. And they might be worth at least acting as a starting point for your own investigation. So the second goal in our hat trick, the second piece to the tripod is dynamic code analysis. This was also in this SDLC pyramid. So unlike static code analysis, dynamic code analysis actually executes the code. Now that has a prerequisite that the code has to be compiled and free of errors and can run before you can actually perform the dynamic code analysis. Dynamic code analysis, I mentioned this before, is different than pen testing, whereas dynamic code analysis looks for the flaws in the code that can be exploited. Pen testing looks for ways to exploit the system in general, often at a greater detail and almost always involving a human attacker. So again, two different things, they're kind of similar, but I really think that dynamic code analysis is really at best a subset of a good penetration test. So dynamic code analysis tools can identify up to 85%. But if you combine them with the static code analysis tools on the market, up to 95% of your applications vulnerabilities can be identified early on. 95%. That's pretty darn impressive. So again, I'm going to throw a few products up here for you to take a look at. I've used several of these. Again, not recommendations, but a place for you to start. Give you a word of warning. I think I told you earlier on, I would tell you some things that work and some things that just didn't seem to work quite so well. Some of these tools can be weapons. I'm looking at you, Burp Suite. You need to do things like make sure you set the scope of your testing tool so it doesn't see the, oh, I've got a link to Facebook here. I will jump and follow that link in my crawl and I will start attacking Facebook. Set your scope absolutely to your application only. Run the stuff and test. Don't run it in production. If you find a SQL injection exploit and you start to execute it, it's very, very likely that you're going to corrupt or lose data on the database on the backside of that injection vulnerability. Also, watch out for forms. So let's say you're doing some dynamic testing and it's automated and you're getting ready to essentially attack a web form. Make sure you find out where that web form is going before you start your test. You can easily overrun a service desk by submitting thousands upon thousands of emails from a web mail form. Not saying that I've ever done that, but I know that it has happened. So the third goal in our hat trick is open source analysis. Open source is awesome. We all use open source. Some estimates put the percentage of open source in internally developed applications at 70%. That's 70%. Now, the problem with that is, as a result, about 50% of our applications contain one or more open source pieces that has a high security vulnerability. Keeping that out of production is a huge, huge task. There are two different approaches I've taken when I've tried to address this problem. The first was a firewall slash proxy solution. There are products out there that you can actually set up that will proxy the main repos out in the wild. They'll check as you pull down code libraries. They'll check them for license compliance. You can say, hey, I'm okay with this license, this license, but I don't want that license. You can set that up ahead of time and it will check each time a module is pulled down that it is in compliance with your licensing. But more importantly, you can also set a vulnerability threshold. You could, for example, say, hey, I don't want any critical or high translation. I don't want any CVE7 or higher coming into my organization's local repository. The problem with that is a lot of the tools that developers use, particularly in DevOps, for automating building of systems, automating testing, have vulnerabilities because they really weren't meant to go live anywhere. They were basically developer aids, developer tools, and your policy is going to quarantine or prevent them from coming into your local repository. The result is you get a lot of phone calls. Hey, this module, I need it, I need it now, and it's quarantined. Well, that requires getting out on the firewall of proxy and looking at what's quarantined, determining how it's going to be used, where it's going to be used, and does that CVE apply in this use case, and then deciding whether you can waive it or not. It is a labor-intensive process. I didn't really like doing it this way at first. So I ended up moving to a build-based solution. Build-based solutions are very similar to what we're going to see in just a few minutes, where you basically add an open-source view step to your automated build process. If you find that same vulnerability as it's going to production, let's say it's a CVE of eight, and it's going into production, you fail the build. At that point, the developers have the opportunity to go, okay, I need to get a security waiver, or I need to find a more recent or a more updated library that doesn't have that CVE as an outstanding issue. So this ended up reducing the manual intervention. The only problem is you could end up having something on your internal network with a CVE of seven or higher. That's a developer tool. You've got to be very careful when you're whitelisting or removing items from quarantine to understand how it's going to be used and if the CVE itself can be exploited in that manner. So here's some products once again. OWAS, I've mentioned them several times, great organization. Again, these are not recommendations. These are simply places for you to start if you haven't looked at these already. And finally, let's talk about automation. So those last three steps, they seem like a lot. And originally, we were doing them manually, and it became unwieldy very quickly. It just took a lot of time. It took a lot of effort. So what we were able to do was automate most of this into the build pipeline. So if you have a continuous integration tool, something like Jenkins, you can add steps that execute static code analysis, dynamic code analysis, and open source analysis with every build. So this gives you all of that protection over 95% from the studies with every single build that you do. Now there's some things that are just too stinking big. So if you're still dealing with legacy applications, you go monolithic jars and ears. If you've got something that's somewhere in the line, you know, hundreds of thousands of lines of code, sometimes close to a million line of code, this is not going to go well for you. Because the dynamic scan, the static scan, those pieces are going to take a long, long time. I tested one application for a friend of mine just off the just to try it. It was a legacy application. And the scan for just the static code analysis piece took over 12 hours. You can't stop a build for 12 hours and expect the developers not to be concerned. So I don't recommend this for large applications. If you do, you might do it as an asynchronous build step. So basically it says, hey, I've got to go do static code analysis, I'm going to launch it, and then I'm going to continue with my build. You lose the advantage of being able to block at that point. But you still have an automated scan going along. You'll need to have a process in place to alert the development team as well as information security of the results of that scan and then decide how you want to react. But you lose the ability to block. But fortunately, the industry is really moving towards smaller builds, APIs, microservices, things like that. And inline scans are very, very possible. They're very fast. These tools are tuned for this kind of work. And they run incredibly, incredibly fast. Just aside, some of the tools are still working on their automated pipeline plugins. So if you are looking at any of the tools, whether it's the ones I've mentioned or something else, make sure that you ask about the maturity of their continuous integration pipeline plugins. Otherwise, you might find yourself stuck manually doing these for a while until you can find a way to do that. I'm going to end with a couple of final thoughts. So the keys to success, hopefully you can read this. They are start small. Do one thing. For me, it was starting with a training program that literally led to everything else that you saw in this presentation. And as you saw in the presentation, this stuff tended to feed upon itself. The development led to more developer training, which led to a security program, which led to so start small, and it will all come together. Get management buy in by both information security and by app sec management. The centerpiece right there, that has to be the key for me. It has to be fun. When this stuff is no longer fun, I'm not going to do it anymore. We've got to keep making it fun, particularly for our developers. Developers are doing a lot of stuff, adding security on top of everything else they're doing is a pain in the butt. And for the most part, it's not welcome unless you can make it fun. If you can make it fun, it's going to happen. So just like we just mentioned, automate everything you possibly can. And remember, it's all about those relationships, the relationships with your scrum teams. It's the relationships with your developers. So relationships with your vendors and the trainers that you can get to come in. So building a good relationship is key. Don't try to reinvent the wheel. The best innovation is usually stolen from somewhere or given from somewhere. So if you can take one piece from this discussion and implement in your organization, I will feel blessed. I'd love to hear about it. Take it slow. This stuff is new for all of us and it's always changing. And finally, learn what works best in your environment. So I want to wrap up with thank yous. I want to thank you for spending the last 40, 45 minutes, whatever it ended up being with me and with AppSec Village. I want to thank AppSec Village. I got to volunteer there last year. I ended up spending the entire DEFCON there. After I think that was my sixth year in information security, I finally found my home and that is the AppSec Village. So thank you to the leadership of AppSec Village for doing this and for giving me this opportunity. I want to reiterate my thanks to Jim Manneco and he's kind of been my spirit leader on this journey and I appreciate all you've done. My colleagues Barry and Shane, my partners in crime, you guys are awesome. And finally, my best friend, my wife, who's put up with me, put up with me doing this and several other things in information security. She's absolutely the best and I can't thank her enough. So thank you guys very much. I appreciate you spending this time with me and I hope you have a really, really excellent rest of your DEFCON. Thanks.
 Hello and welcome back. Hope you're looking forward to another fantastic talk here at  the Apssac Village for DEFCON 29 in safe mode. It's important to remember your 3-2-1, even  though we are separate from each other and socially distanced, your cohabitants will  really appreciate that one in there. Just take care of yourselves, get some good sleep,  get some good food, make sure you take a shower. For our next talk, we've got David Waldrop  talking about the DevOps and Agile Security Toolkit. After 25 years in the application  development, David moved to Information Security just over 7 years ago. His passion is helping  developers write secure code. He's currently an Information Security Advisor supporting  the application development community at his employer. Let's put our hands together and  welcome David Waldrop.  Hi and welcome to the Apssac Village. I hope you're having a great DEFCON 2020, albeit  in safe mode. My name is Dave and for the next 30-45 minutes, we're going to talk about  the DevOps and Agile Security Toolkit. Now before I get started, I have to make the standard  disclaimers. First, this presentation and any comments that I make are my own and do  not necessarily represent those of my employer. Secondly, I will mention a few example products,  some are commercial, some are open source. These are not necessarily meant as recommendations.  They are mentioned to serve as a starting point for your own investigation and review.  These are things that I've run across in my journey, so take that as it is. Let's start  with the two terms that are on everybody's buzzword bingo sheet these days. DevOps and  Agile. We need to get definitions in place for the duration of this talk, so we have  an idea that we're all dealing with the same playing field. Agile. Agile focuses on continuous  iterative development and testing. Software is developed in smaller increments and then  is integrated for final testing and delivery. Scrum, Kanban, and XP Extreme programming  are examples of Agile approaches. On the other hand, DevOps focuses on the collaboration  between AppDev and Operations. The focus is on deploying code to production in a fast  manner, usually automated. So you can see from the definitions that these are two separate  concepts. However, many organizations tend to combine them into one initiative. Very,  very rarely do you find a shop doing DevOps and not doing Agile or vice versa. So no matter  whether they start with DevOps or they start with Agile, the one piece that's usually  late to the party is security. But no matter how late security is invited to the party  or whether we have to crash the party ourselves, we usually can still bring valuable resources  without slowing things down or at least without slowing them down a lot. And that's what  we're going to talk about through the rest of this discussion. So for the past few years,  I've worked with several different Agile and DevOps teams. My integration with this started  as a pilot program where I was the information security representative that was assigned to  help introduce Agile to our organization. More recently, I've implemented the information  security integration with the Agile and DevOps teams across our organization. During this  session, I'm going to share with you some things that really worked well and some things  that just didn't seem to work at all. So hopefully you can take something away from this and  apply it to your own organization. This particular slide will give you the map of what we're  going to talk about for the rest of this discussion. We're going to talk about Agile staffing.  We're going to talk about the eight simple questions. We're going to talk about the security  champions program and then go into developer training, which is my favorite part of my  job. Security in the software development lifecycle. And then we're going to hit the  hat trick of securing your code, static code analysis, dynamic code analysis and open source  analysis. And then finally, we're going to wrap things up by automating the heck out  of it. So let's start with Agile staffing. Sherman set the way back machine for one  when I started with Agile. One information security person was assigned to one Agile  team. That security person would be integrated into that team and attend every ceremony,  every meeting, everything, every day. That didn't work. There are a couple of reasons  I think that that didn't work. First of all, agile teams tend to multiply. You can go from  a proof of concept with one team to having close to 30 teams in less than two years.  Most organizations do not have the information security bandwidth to support that type of  staffing model. In addition, the scrum masters tend to treat every member on the team as  though they can perform all tasks. And when you're dealing with a team of all developers,  that makes perfect sense. This model tends to fall apart though when you put information  security, QA, middleware, any other infrastructure or shared services team as a part of the agile  team. What happens is in the agile process, their hours count against your available development  hours. But the infrastructure people, the information security people do not have the  skills or the capabilities to do some of the development. You see that there's a mismatch.  You're given hours, but you're not allowed to use them. It caused a lot of problems for  some of our scrum masters early on. Again, we realized this was not the way to do it.  I ended up coming up with a two tiered approach for trying to integrate information security  into the agile team. First, at a high level, at the enterprise level, we will have an architect,  an information security architect, and they will inject into those high level projects  and basically set up guardrails and guidelines. So as projects are being devised at the visionary  level, they can jump in and try to help guide them into making more secure decisions at  that high level. Now, when you get down into the weeds though, each agile team does need  to have somebody they can rely upon for information security. And so what we ended up doing is  we created named resources within the information security team that are assigned to agile teams.  Now it's not like the previous model. You're not assigned, you're not sitting with them,  you're not assigned to go to every meeting. We're going to talk about how we scaled that  back, but we ended up finding a way where based on where information security can add  the most value, we attend those ceremonies and only those ceremonies. So at the end  of the day, each information security representative can support between six and eight different  agile teams. That is a model that is certainly sustainable. So what is a little different?  Well, first of all, we make sure that every agile team has a named resource that is responsible  for information security for them. So this gives them a named contact if they ever need  something they know exactly who to call and where to start that quest. Secondly, we attend  or the agile reps attend the backlog refinement sessions. These are meetings that happen once  every two weeks before the next sprint and it defines what they're going to work on in  that next sprint. It's a perfect opportunity to jump in and say, hey, you're planning this  work. We need to think about blah, blah, blah from an information security standpoint.  We can get in just ahead of when development starts. It's a perfect time to bring up security  issues, make recommendations and answer questions. And then finally, we attend the quarterly planning  agreement meetings where the vision is cast for the next quarter and we can make sure  that we're on their radar for things that we think we need to be a part of. So this  is kind of a high level diagram. The purple arrow at the top is the information security  architect. That particular person, as you can see from the diagram, sets guardrails for  enterprise planning projects. All of the red arrows are the different places where our  individual security representatives can or may inject into the agile teams. We don't inject  in all those places, but there are places where we can and we give them that option  to call us if they're needed. So let's move on to what I like to call the eight simple  questions. This particular concept came up as a result of assigning our agile reps to  their teams. We found that the leadership within the agile teams really wasn't sure  when to call us. And since we weren't sitting there in every meeting, there was a possibility  that something could come up that might fall off the radar that we might not get contacted  for that they may not even think is important. So we decided that we need to give clear guidelines  to the scrum masters and product owners as to when they should include information security.  That's how I created the eight simple questions. So basically these eight questions are a guideline  for that leadership team. And if they answer yes to any of them, they need to make sure  to include their information security representative in those discussions. These are the questions  as I currently have them. We're going to talk about that in just a bit because they change  over time. These are not one and done kind of things. Am I dealing with any sensitive  information? Now information that is sensitive, it's going to mean something different for  each one of our organizations. If you're working in healthcare, if you're working with banking  information, if you're working with retail with PCI information, you will be able to  find what sensitive information means for your organization. Am I sending or receiving data  outside of the company? Is there a new vendor involved? Am I going to introduce new software  systems or libraries, even open source and remember open source because we're going to  talk about that? It's a unique challenge. Do I need access to secrets such as passwords,  keys or certificates? Will I need the ability to use single sign on? Am I going to leverage  the cloud? Will I need updated access roles? So these eight questions are what we are currently  using as a guideline for the interaction between the agile team and their information security  representative. As I mentioned, the exact questions are going to differ on your organization.  The questions themselves will actually change over time. I started, I think, with seven  questions and then we had several issues come up with access roles and who to ask and blah,  blah, blah. So we added a question for that too. So don't be afraid to take a stab at  it and massage it and change it. That's the thing about agile and DevOps. It's changing  all the time and we as information security people need to be able to change our stuff  as well. So the basic idea is this. Create a list of questions usually 10 or less, something  that's going to fit easily on a PowerPoint slide or a single piece of paper so they can  take it to meetings with them. Give that to the agile team leadership and they have a  guideline. They now have a concrete way of knowing when they need to work with information  security. So we found another advantage to this approach as we onboarded additional information  security people to become representatives for the agile teams. A lot of these folks  were pretty new to the organization and they didn't really know when they needed to inject  themselves into their teams. They didn't know what they needed to listen for in the  backlog refinement sessions. So we use these simple questions on our own people. It gave  them guidelines and hints of what to listen for when they're working with their agile  teams. So this kind of became a win-win on both side of the coin. So I think this was  a great tool that we've been able to use. Another tool that I honestly have to thank  Jim Manneko for. Jim if you're out there thank you for this. This was a recommendation he  made to me after one of our training sessions and that is the security champions program.  A security champions program is a team of application developers. We get one from each  agile team and these developers have expressed an interest in information security or secure  coding or just security in general. This is a voluntary program. The members of the team  volunteer to come to the security champions program. And we typically meet quarterly.  I provide them lunch when we're in the office thanks 2020. I look forward to getting back  to the office and having lunch with these guys and gals. They're a great team. So the  team has this following charter. We assist the agile teams in terms of application security.  We serve as a communication conduit between the agile teams and information security. And  then we meet quarterly to share experiences. We ask questions. We share knowledge. It has  had some incredible benefits. First of all security has a relationship with at least  one person on every agile team. I'm not saying we have a mole. What I am saying is sometimes  we hear about things long before it percolates through the management chain. I might get  a phone call about hey we're thinking about using this open source library. Is that cool.  We take a quick look at it. We approve it. By the time the formal approver gets into my  hands we've already done it in their coding. So it gives us a great communication conduit  and that helps me make things faster. I may have not mentioned this before. I hopefully  I have relationship is key. You're going to hear about relationship throughout this entire  discussion. We build relationships with those agile teams. It starts with having an assigned  rep. It moves into a different relationship level when we have a security develop security  champions team. Secure coding training takes it to yet another level. And you'll see why  in just a few minutes as we talk about the developer training. But it's about building  relationships. It's about having fun while we're doing the right thing for our organizations.  So so far the team has done a lot of really cool things. We've reviewed vendors and products  both on the development side and on the information security side. We've selected our developer  training programs for the last two years based on the input from the security champions team  members. We've done a bunch of PSEs together. We've cross consulted. Now this was kind of  an unexpected benefit is where if I had a team that was very well versed in one technology  and they mentioned that in a security champions lunch, another team might reach out going,  hey, we're about to do that. Can we stand and talk? So it actually starts to cross pollinate  some security knowledge and some technical knowledge between different application teams.  And then finally, it's served as a candidate group for application security openings within  the infosec team. The software development life cycle. So as we started to roll out security,  a lot of people have asked us, how does this fit into our SDLC? So I ended up creating  a plan that maps information security phases to the software development life cycle as  it currently exists. It is not a perfect fit, but I think it is a good representation of  how information security can bring value to each step of the SDLC. So this is what I like  to call my dark side of the moon diagram. Those of you old enough to remember probably  recognize the album cover. So if you take a look on the right, we have the software development  life cycle from concept into design up into coding testing a QA, QA and deployment and  then post deployment at the very top. Now, if you look on the left side, you'll see that  it matches up pretty closely with security knowledge, security design, secure coding  techniques, security testing and then security response. The each individual layer of the  pyramid, I think are deliverables that security can bring that help take that concept on the  left and apply it to the SDLC phase on the right. And we're going to walk through those  real quick. Training is the base of the pyramid. I think training is foundational to everything  above it. If you don't have a solid foundation of training for your application developers,  everything else is at risk. So this particular approach, we're going to talk about in depth  again, because it's my favorite part of what I do. It includes things like lunch and learns,  outside speakers, conferences like you're attending right now, video training and things  like that. The next level up is threat modeling threat modeling, the application team and  InfoSec. We sit together, we work together to examine the application design, the system  flows, whatever they can bring to us that they have available in that design phase.  We look, we try to work together and identify potential vulnerable vectors and try to figure  out a way that we can mitigate them. Now threat modeling, there are books upon books written  about threat modeling. And there are people in my organization on my InfoSec team that  are much better at it than I am. And they can actually create incredible documents and  spend hours and days and days and days doing this. I think you need to start out small.  I think it's about baby steps at first. Start with a discussion. This can be a one or two  hour discussion where you have those deliverables from the design phase and you whiteboard them.  And you come out of there with to dos or questions or things to research. It doesn't have to  be this big, heavy weight overbearing approach to threat modeling at first. It needs to be  a conversation that gets back to relationships. You need to have those relationships in place  so you can have those conversations. The third level up is static code analysis. We're going  to talk about this in much more detail later. But most tools are designed to identify potential  vulnerabilities within the code. Now static code analysis actually looks at the code and  looks for patterns within the code without running it. Now we're going to see a little  bit later why that's important. I think I went the wrong way. There we go. Dynamic code analysis  is the next layer up. And that is very similar to the previous layer which was static code  analysis. But instead of just looking at the code, the code is actually executed and tested  for vulnerabilities. So like I said, these two pieces are very, very similar. One is  looking at patterns with the code. The other is actually exercising the code and looking  for issues. The next is pin testing. Pin testing can be performed by your internal team if  you have the skills or resources or you can use an external firm. This is usually a greater  scope than dynamic testing. It will actually look at things not just in your application  but system wide for vulnerabilities and attack vectors. And usually there's a person involved.  It's not just an automated exercise. We're going to see a little bit more when we talk  about dynamic testing, the difference between that and pin testing. And then finally, when  the code is in deployment, you need to have a vulnerability management process that you  can classify, assign, and track identified issues. Right now, InfoSec meets regularly  with a representative from each application team to review these and to make sure that  they're working and they're on track. Again, those relationships are absolutely key to  making this work. So now we are getting to what I consider my absolute favorite part  of my job. And that is developer training. If you remember back to that pyramid we just  looked at, this is the foundation for everything that information security does with app dev.  When I came in to information security about seven years ago, I started a developer training  program because I realized that the developers really are your last best hope for defense  when it comes to application security. If your application is written correctly, then  everything you throw in front of it, application firewalls, everything like that, that's gravy.  That's nice, but you've got a really tight application and that's where real security  takes place. The problem is, I was a developer for roughly 25 years before coming to information  security and nobody taught this stuff. This is stuff you learned on your own. If you had  an interest in it, and it wasn't universally available to all developers. So I started  small. I started with Lunch and Learns. It was really something that, hey, I think this  is important. Let's try it. Let's see if it works. And it really did. We would end up  having Lunch and Learns about every other month. I would create a class on a specific  topic. My first one was cross-site scripting, a lot of fun. It's neat when you can show  people cross-site scripting examples, even if you just run Webgoat or something like  that. They get excited. They kind of dig it and they start trying it on their own stuff.  The content usually lasts about 40 or 45 minutes, a lot like this session. And then we leave  10 to 15 minutes for open discussion and questions. Now the key here is make it fun. It has to  be fun. I do things like I'll go out and I'll buy cookies or some kind of treat. So we can  all sit around, eat something together. There's usually some kind of door prize or door prizes.  And they don't have to be expensive. You can just go down to like five below and get a  couple of wireless Bluetooth speakers. And people dig it. It's a lot of fun. Again, making  it fun is absolutely key to a successful Lunch and Learn program. So once we got that up  and running, I decided we needed to take this up a notch. There is a level of training that  I couldn't provide that I think our developers really, really would benefit from. And in  order to do that, I knew that I had to reach out to industry experts. And what we ended  up doing is we were able to get commitment from management both on the IT development  side and on InfoSex side that we could do one or two days where most, if not all the  developers will attend a training event. Now here's a hint. If you are going to try this  and you're doing Agile, put it in your innovation sprint. If you put it in the innovation sprint,  which is usually the last sprint of the quarter and the developers have a little more leeway  of what they work on during the innovation sprint, you'll very likely get more participation  than if you step on a middle of a sprint where there's actual deliverables do. So keep that  in mind. I was really, really lucky. My very first interaction with this program, I was  able to secure Jim Manneco and he came in and made an incredible impression on our development  teams. Jim's been back, I think three times now. Always a pleasure. Rich Mogel's been  by twice. We've had a recent one with Ron Parris. Dr. Phillip DeRique, phenomenal instructor.  I just took his OAuth class. So if you're looking for OAuth, wonderful class. But we've  got him coming in this fall and I'm looking forward to that. The reason I actually want  to call these gentlemen out by name is I think it's important to be grateful. I am very grateful  for everything they've done for me and my development teams. And it's important to be  grateful to those who help you along the way on this journey because none of us make it  alone. So gentlemen, thank you very much. If you see this or hear it later, your training,  your mentorship has meant a lot to me and my development teams. So one cool aside is  I mentioned this earlier. The security champions team has been involved in the selection of  the instructor for the last two years. This gives your application development teams a  voice in the training. And we've seen that participation has actually increased. We had  great participation before then. But when the developers actually had a voice in picking,  who's coming in and what they're going to talk about, we're finding that it's much  more applicable to what's on their roadmap. It's much more applicable to the skills they're  hoping to get. And participation goes beyond what I could have ever hoped for. So again,  we're starting to see some of these initiatives are starting to weave together and starting  to build kind of a program. It's kind of cool. So this next one, I really wish I was doing  this right now. Take a developer to DEF CON. We had management agree to provide scholarships  to developers that were selected by the information security team. So we would give them the money  to buy their badges. So they wouldn't have to go and get money out of their own bank  account. So each year, I've been able to bring up to three developers to DEF CON. That's  been awesome. It has been just a phenomenal experience. They've gained a ton of knowledge  and a ton of awareness by DEF CON. One of the things that I found really interesting  was I really advise them not to restrict their sessions to things that they think will apply  to their job. I did that my very first DEF CON and I wasn't really sure I was going to  come back to be honest with you. The next year, one of my coworkers strongly encouraged  me to go. Thank you, Barry. And encouraged me to go and find a few fun things that just  had nothing to do with my immediate job. The interesting thing is they ended up having  more to do with the things that were coming down the road in my job than I ever could  have imagined. So I fully encourage each one of these developers that I bring to DEF CON  to experience the full DEF CON experience. Go to all the sessions. Try some of the capture  the flags. Try whatever you want. Have a good time because it's really important for them  to be immersed in the DEF CON experience. So like I said before, they come back with  a changed perspective. It's a lot like that scene in The Matrix where that thing crawls  out of Neo and he goes, my God, that thing is real? They understand now that the security  is not just something that we're teaching and preaching about within InfoSec. That this  stuff has real world application. And it's that realization that makes them a better  developer, a more secure developer going forward. We've seen a trend where a lot of the folks  that we've taken actually, I think all but one of the developers that we've taken to  DEF CON have eventually become security champions members. And then more recently, just in the  last couple of months, one of those developers, I think he was the first one I took to DEF  CON with us, ended up joining the information security team on a full time manner. And Shane,  if you're listening, welcome. I'm really glad that you're part of the team. So there's  a couple of other fun activities that we do in developer training. Again, got to keep  it fun. Everything has to be fun. You learn better when you're having a good time. I mean,  look at DEF CON, we're all having a great time. And we're learning stuff. So a couple  of DEF CONs ago for reasons of work, I couldn't make it. It was the only DEF CON since coming  over to information security that I was not able to go to. And I must admit, I was bummed.  So what I decided to do was I was going to have my own little DEF CON. And we decided  that, okay, let's just take people along for the ride. So we identified five videos from  previous DEF CONs and we had a lunch and learn for five days that week where we watched it  and we talked about it. We had drawings for DEF CON related swag or items or giveaways.  We just kind of had a good time. It was kind of built up as, hey, take a break, come bring  your lunch, let's hang out, watch something from DEF CON, talk about it. You might win  a t-shirt, you might win some stickers, whatever. And people really enjoyed it. It was a lot  of fun. As a matter of fact, I'm probably going to do one this fall for the folks that  didn't get to enjoy what you're doing right now, what you're enjoying right now. And then  finally, I have an annual thank you lunch and learn. This was a tradition I started  the first year we did lunch and learns. It's always the last lunch and learn of the year.  And I typically will go out and I'll purchase quite a few giveaways. They don't have to  be expensive. We do things like, say, the Bluetooth speakers, boxes of candy. We've  done some Christmas mug sets, you know, just weird things, but things that are kind of  fun. And we do a shorter lunch and learn followed by a session where we just eat some good  food and we have some drawings. And I have an opportunity to thank my development community  for the support of the lunch and learn program and for the support they've given to information  security. I think it's important to say thank you. And this has been just an incredibly  popular lunch and learn every year. It's a lot of fun and I highly recommend it. So here  are the keys to a successful lunch and are successful training program. Make it applicable  and give the developers a voice in the training. And usually if you give them a voice in the  training, it will be applicable just because they're going to tell you what they need.  I think I've mentioned this 1427 times, but make it fun. It's got to be fun. Share how  much you care about this stuff. If you're engaged, if you're passionate about it, people  are going to feel that and they're going to gravitate to you and gravitate to that topic.  And then finally build relationships. Listen, reward and always express gratitude along  the way in this journey. So we're getting into the next phase of the toolkit. And this  is what I like to call the analysis hat trick. Three types of analysis that if we implement  all three, we're going to have a much more secure code base in production. And the first  is static code analysis. You might remember that from our SDLC pyramid. So static code  analysis scans the code without running it. It looks for known patterns of vulnerabilities.  You can usually select what you scan for. Some will allow you to scan for PCI, blah,  blah, blah. I almost always scan for OWASP top 10. If you're not familiar with OWASP  or OWASP top 10, I strongly recommend that you check it out. Some of the deliverables  from that organization are absolutely stellar. I've been a member for I think seven years  now just after joining the InfoSec team. And I've leveraged a ton of their stuff. So great  organization hats off to you if you're out there. In an ideal world, the static code  analysis tools would only report the confirmed vulnerabilities. But even if you tune the  daylights out of this thing, you're still going to get false positives. And that becomes  a problem. Because originally when we first set up this process, I was the only one that  could review the false positives and whitelist them or wave them as it were. That slowed  things down immensely because that was not my full-time job. That was something else  on top of my full-time job and possibly another full-time job some days it felt like. So that  being said, I decided to leverage the security champions. These were folks that had some  basic knowledge of security. They knew their applications better than I ever would. And  they had a passion for making secure code. Seemed like a perfect fit. So what I ended  up doing is I ended up giving several members of the security champions team the ability  to review and wave false positives. And that has worked out great. The end result was we've  had a lot more throughput. We've been able to address false positives much more quickly.  And the adoption of the static code analysis tools have actually increased as a result  of not having that delay. So I also see static code analysis as an extension of developer  training. A lot of these tools have an IDE component. So for example, I used the Spring  Tool Suite Eclipse. And you can use the plug-in and execute your scan from within your developer  tool. And it will take you straight to the line that's got the problem. And in some of  these tools you can even have recommendations pop up in the bottom of one of the windows  that says, hey, you might try blank, blank, and blank to address this vulnerability. If  developers are doing this while they're developing, not waiting for the build process, it becomes  a teaching tool. Because as a developer uses this, he or she is going to identify and learn  that, oh, I need to check my input fields to make sure they're the right type and right  size. Okay, you do that enough times. It becomes second nature. So this becomes not only a  way to secure your code, but a teaching opportunity as well. So studies have shown that study  code analysis can identify up to 80% of vulnerabilities. That's pretty awesome. If I could tell you  that you can do one thing and remove 80% of vulnerabilities, I think that'd be worth doing.  So this might be something you might want to take a look at. Here's some example products.  Here's three products along the top that are commercial. OWAS has an entire list of code  analysis tools at that link right there. Again, not meant as recommendations, but these were  tools that I've come across in my journeys. And they might be worth at least acting as  a starting point for your own investigation. So the second goal in our hat trick, the second  piece to the tripod is dynamic code analysis. This was also in this SDLC pyramid. So unlike  static code analysis, dynamic code analysis actually executes the code. Now that has a  prerequisite that the code has to be compiled and free of errors and can run before you  can actually perform the dynamic code analysis. Dynamic code analysis, I mentioned this before,  is different than pen testing, whereas dynamic code analysis looks for the flaws in the code  that can be exploited. Pen testing looks for ways to exploit the system in general, often  at a greater detail and almost always involving a human attacker. So again, two different things,  they're kind of similar, but I really think that dynamic code analysis is really at best  a subset of a good penetration test. So dynamic code analysis tools can identify up to 85%.  But if you combine them with the static code analysis tools on the market, up to 95% of  your applications vulnerabilities can be identified early on. 95%. That's pretty darn impressive.  So again, I'm going to throw a few products up here for you to take a look at. I've used  several of these. Again, not recommendations, but a place for you to start. Give you a word  of warning. I think I told you earlier on, I would tell you some things that work and  some things that just didn't seem to work quite so well. Some of these tools can be  weapons. I'm looking at you, Burp Suite. You need to do things like make sure you set the  scope of your testing tool so it doesn't see the, oh, I've got a link to Facebook here.  I will jump and follow that link in my crawl and I will start attacking Facebook. Set  your scope absolutely to your application only. Run the stuff and test. Don't run it  in production. If you find a SQL injection exploit and you start to execute it, it's  very, very likely that you're going to corrupt or lose data on the database on the backside  of that injection vulnerability. Also, watch out for forms. So let's say you're doing some  dynamic testing and it's automated and you're getting ready to essentially attack a web form.  Make sure you find out where that web form is going before you start your test. You can  easily overrun a service desk by submitting thousands upon thousands of emails from a  web mail form. Not saying that I've ever done that, but I know that it has happened. So  the third goal in our hat trick is open source analysis. Open source is awesome. We all use  open source. Some estimates put the percentage of open source in internally developed applications  at 70%. That's 70%. Now, the problem with that is, as a result, about 50% of our applications  contain one or more open source pieces that has a high security vulnerability. Keeping  that out of production is a huge, huge task. There are two different approaches I've taken  when I've tried to address this problem. The first was a firewall slash proxy solution.  There are products out there that you can actually set up that will proxy the main repos  out in the wild. They'll check as you pull down code libraries. They'll check them for  license compliance. You can say, hey, I'm okay with this license, this license, but I don't  want that license. You can set that up ahead of time and it will check each time a module  is pulled down that it is in compliance with your licensing. But more importantly, you  can also set a vulnerability threshold. You could, for example, say, hey, I don't want  any critical or high translation. I don't want any CVE7 or higher coming into my organization's  local repository. The problem with that is a lot of the tools that developers use, particularly  in DevOps, for automating building of systems, automating testing, have vulnerabilities because  they really weren't meant to go live anywhere. They were basically developer aids, developer  tools, and your policy is going to quarantine or prevent them from coming into your local  repository. The result is you get a lot of phone calls. Hey, this module, I need it,  I need it now, and it's quarantined. Well, that requires getting out on the firewall  of proxy and looking at what's quarantined, determining how it's going to be used, where  it's going to be used, and does that CVE apply in this use case, and then deciding whether  you can waive it or not. It is a labor-intensive process. I didn't really like doing it this  way at first. So I ended up moving to a build-based solution. Build-based solutions are very similar  to what we're going to see in just a few minutes, where you basically add an open-source view  step to your automated build process. If you find that same vulnerability as it's going  to production, let's say it's a CVE of eight, and it's going into production, you fail  the build. At that point, the developers have the opportunity to go, okay, I need to get  a security waiver, or I need to find a more recent or a more updated library that doesn't  have that CVE as an outstanding issue. So this ended up reducing the manual intervention.  The only problem is you could end up having something on your internal network with a  CVE of seven or higher. That's a developer tool. You've got to be very careful when you're  whitelisting or removing items from quarantine to understand how it's going to be used and  if the CVE itself can be exploited in that manner. So here's some products once again.  OWAS, I've mentioned them several times, great organization. Again, these are not recommendations.  These are simply places for you to start if you haven't looked at these already.  And finally, let's talk about automation. So those last three steps, they seem like  a lot. And originally, we were doing them manually, and it became unwieldy very quickly.  It just took a lot of time. It took a lot of effort. So what we were able to do was automate  most of this into the build pipeline. So if you have a continuous integration tool,  something like Jenkins, you can add steps that execute static code analysis, dynamic  code analysis, and open source analysis with every build. So this gives you all of that  protection over 95% from the studies with every single build that you do. Now there's  some things that are just too stinking big. So if you're still dealing with legacy applications,  you go monolithic jars and ears. If you've got something that's somewhere in the line,  you know, hundreds of thousands of lines of code, sometimes close to a million line of  code, this is not going to go well for you. Because the dynamic scan, the static scan,  those pieces are going to take a long, long time. I tested one application for a friend  of mine just off the just to try it. It was a legacy application. And the scan for just  the static code analysis piece took over 12 hours. You can't stop a build for 12 hours  and expect the developers not to be concerned. So I don't recommend this for large applications.  If you do, you might do it as an asynchronous build step. So basically it says, hey, I've  got to go do static code analysis, I'm going to launch it, and then I'm going to continue  with my build. You lose the advantage of being able to block at that point. But you still  have an automated scan going along. You'll need to have a process in place to alert the  development team as well as information security of the results of that scan and then decide  how you want to react. But you lose the ability to block. But fortunately, the industry is  really moving towards smaller builds, APIs, microservices, things like that. And inline  scans are very, very possible. They're very fast. These tools are tuned for this kind  of work. And they run incredibly, incredibly fast. Just aside, some of the tools are still  working on their automated pipeline plugins. So if you are looking at any of the tools,  whether it's the ones I've mentioned or something else, make sure that you ask about the maturity  of their continuous integration pipeline plugins. Otherwise, you might find yourself  stuck manually doing these for a while until you can find a way to do that. I'm going to  end with a couple of final thoughts. So the keys to success, hopefully you can read this.  They are start small. Do one thing. For me, it was starting with a training program that  literally led to everything else that you saw in this presentation. And as you saw in  the presentation, this stuff tended to feed upon itself. The development led to more developer  training, which led to a security program, which led to so start small, and it will all  come together. Get management buy in by both information security and by app sec management.  The centerpiece right there, that has to be the key for me. It has to be fun. When this  stuff is no longer fun, I'm not going to do it anymore. We've got to keep making it fun,  particularly for our developers. Developers are doing a lot of stuff, adding security on  top of everything else they're doing is a pain in the butt. And for the most part, it's  not welcome unless you can make it fun. If you can make it fun, it's going to happen.  So just like we just mentioned, automate everything you possibly can. And remember, it's all about  those relationships, the relationships with your scrum teams. It's the relationships with  your developers. So relationships with your vendors and the trainers that you can get to  come in. So building a good relationship is key. Don't try to reinvent the wheel. The  best innovation is usually stolen from somewhere or given from somewhere. So if you can take  one piece from this discussion and implement in your organization, I will feel blessed.  I'd love to hear about it. Take it slow. This stuff is new for all of us and it's always  changing. And finally, learn what works best in your environment. So I want to wrap up  with thank yous. I want to thank you for spending the last 40, 45 minutes, whatever it ended  up being with me and with AppSec Village. I want to thank AppSec Village. I got to volunteer  there last year. I ended up spending the entire DEFCON there. After I think that was my sixth  year in information security, I finally found my home and that is the AppSec Village. So  thank you to the leadership of AppSec Village for doing this and for giving me this opportunity.  I want to reiterate my thanks to Jim Manneco and he's kind of been my spirit leader on  this journey and I appreciate all you've done. My colleagues Barry and Shane, my partners  in crime, you guys are awesome. And finally, my best friend, my wife, who's put up with  me, put up with me doing this and several other things in information security. She's  absolutely the best and I can't thank her enough. So thank you guys very much. I appreciate  you spending this time with me and I hope you have a really, really excellent rest of  your DEFCON. Thanks.
DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_0_59.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_60_107.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_108_155.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_156_203.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_204_251.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_252_311.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_312_359.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_360_1703.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_1704_2099.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_2100_2783.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_2784_3011.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_3012_3947.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_3948_4931.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_4932_5939.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_5940_6815.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_6816_6863.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_6864_9323.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_9324_13187.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_13188_13943.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_13944_14111.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_14112_15131.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_15132_15467.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_15468_17087.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_17088_19511.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_19512_19883.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_19884_20831.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_20832_21299.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_21300_24635.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_24636_24923.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_24924_25427.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_25428_26747.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_26748_27539.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_27540_29399.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_29400_29963.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_29964_30023.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_30024_30731.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_30732_31559.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_31560_32255.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_32256_32435.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_32436_33995.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_33996_35795.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_35796_38963.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_38964_40343.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_40344_42851.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_42852_44531.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_44532_48047.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_48048_49163.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_49164_49727.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_49728_52835.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_52836_55031.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_55032_55643.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_55644_55895.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_55896_57095.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_57096_57635.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_57636_57911.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_57912_59951.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_59952_60095.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_60096_60911.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_60912_63791.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_63792_65459.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_65460_65807.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_65808_65867.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_65868_66959.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_66960_69575.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_69576_70331.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_70332_70487.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_70488_72563.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_72564_73415.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_73416_75143.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_75144_75191.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_75192_75239.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_75240_75287.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_75288_75335.png DEF_CON_Safe_Mode_AppSec_Village_-_David_Waldrop_-_ZZkE7nPOtjE_75336_75431.png