An Attacker Looks at Docker: Approaching Multi-Container Applications
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
McGrew, Wesley
Containerization, such as that provided by Docker, is becoming very popular among developers of large-scale applications. The good news: this is likely to make your life easier as an attacker. While exploitation and manipulation of traditional monolithic applications might require specialized experience and training in the target languages and execution environment, applications made up of services distributed among multiple containers can be effectively explored and exploited "from within" using many of the system- and network-level techniques that attackers, such as penetration testers, already know. The goal of this talk is to provide a hacker experienced in exploitation and post-exploitation of networks and systems with an exposure to containerization and the implications it has on offensive operations. Docker is used as a concrete example for the case study. A hacker can expect to leave this presentation with a practical exposure to multi-container application post-exploitation.
 For our next talk, we have Wes McGrew. He's going to talk to us about post exploitation of Docker containers. Thank you. So the target audience for this talk is unapologetically red team members, people who are doing red teaming engagements, other offense-oriented services, such as penetration testing, application security testing. And the title of the talk, I always have long titles for my talks. There's the clever bit, and then there's the actual part of it. And so it's a look at multi-container applications. So building, nowadays with containerization technology in Docker, which I'll cover in a little more detail, you can build an application basically out of these LEGO blocks of Docker images or other containers that you find on the internet. And so portions of your application can be made up of these immutable containers. And what that means is that you're working at a higher level of abstraction, and all of your vulnerabilities come from these lower levels of abstraction, where these Docker images have vulnerabilities, and the way they talk to each other in a non-authenticated way on the back end can cause issues. And so the goal of this talk is to give a penetration testing practitioner who's not already familiar with Docker or is facing multi-container applications a little bit of tactics and tools to use to examine these systems. And so my title at Horn Cyber Solutions is director of cyber operations. And so what that means is that I oversee all offense-oriented services. I run teams that do multiple teams that are simultaneously doing pin tests, red teaming engagements, and application security testing for a number of different clients. And so I try to get hands on with it as much as I can. But really, I'm more interested in the management of all of these sorts of things and making sure that all the teams are up to date on the latest tools and techniques. In a previous life, I was a professor at Mississippi State University. I still adjunct there. I teach class every semester. And there I worked on industrial control systems, developed a reverse engineering program there for getting the NSA, CAE, and research, education, cyber operations. And so basically, most of my research, most of my work, has been on the offense side of things. And so if you're a blue team in here, you might find things a little bit lean. But you can learn how to break these things and say that you can then look out for these tactics and techniques. The intention for this deck, really, is to make a strong point about the relationship between an attacker's skill level, your skill set, the set of things that you know as an attacker, and your development as an attacker over time, versus developer trends. And so this talk isn't just about attacking Docker containers specifically, or even multi-container applications specifically. It's really about how do you develop your skills over time as an attacker, whether malicious or not. And so how do we leverage what you already know and how to look at learning new technologies moving forward and the sort of attitude to approach it with? We want to provide a hacker that's experienced in exploitation and post-exploitation of networks of systems. And so if you think of a penetration test of an organization, you're given the scope and you're scanning that scope. And you have a set of systems that interact with each other that's your attack surface, the set of things that you can interact with. How do you move and take your, most of your training as a penetration tester or a red teamer has been in that mode. Looking at that attack surface, looking at protocols between services, looking at vulnerabilities on the outside edge of each of those individual systems. How do you move and take that skill set and apply it to an application? And so historically applications have been monolithic. And so a monolithic application is something that's made up of primarily a single binary or a single service running on a single system or a single virtual machine. And so with a monolithic application, if you're wanting to inspect the state of internal things to the application, how functions call each other, how data is stored in memory, you're looking at attaching as a debugger, you're looking at setting breakpoints, you're looking at exploitation in terms of memory corruption or other sort of remote code execution, to get to the point that you can examine these things and then you're in this sort of environment of a debugger or something where you're limited in how you move. With multi-container applications, where you're building your application out of these immutable images, talking to each other, a lot of times your variable storage and your function calls wind up taking the place of TCP IP connections wind up taking the place of things like Redis databases that actually store what would normally be a local variable. And so it allows you to explore the internals of an application in the same way that you penetration test a network. So I'm going to demonstrate this with some concrete Docker example, with a concrete Docker example. The white paper for this talk has a set of different things to sort of step up to this, but I'm going to show you the cool one first. And the inspiration for this approach and this talk is what I consider a classic talk by H.D. Moran Valsmith from DEF CON 15, tactical exploitation. It's quite an old talk, I guess, now. But it's actually well worth going back and looking at if you're a pentester and you never saw it. All of the work in any kind of field is built on the shoulders of giants, built on the shoulders of people who have come before you and doing this sort of work. There's been a number of talks on Docker and containers. Overall, there's DockerCon and general purpose Docker talks out there in research, but specifically in security and in previous DEF CONs and Blackhats, we have here a list of good references, good material to read to prepare yourself for this sort of thing. You've got Mortman, Greta Fiori, Bettini, Churny, Dolce. And these are all great works, but they all focus on different things than what we're focusing on here. They either focus on the underlying implementation or architecture of Docker, the way it's implemented at a kernel level, which for the purpose of this talk isn't exactly the most important thing. And having some advice for that low level security sort of things. And then there's vulnerabilities in the platform itself, in your Docker run times, in your Docker infrastructure, and the way that developers use Docker and how you can target Docker developers. The thing is, if I get on stage here and I present a Docker talk that has Docker zero day, well, in a couple weeks, everybody will have updated, well, most everybody, except for that one client, right? They'll have updated their Docker instances and those vulnerabilities don't work anymore. So really, you want something more timeless. And so what you're looking at really here is post-exploitation as being something more useful to learn in the long term, because you're always going to find a way into one of these applications through vulnerability in its outer attack surface. And so the focus of all those other talks has been on defense as well, or at least on specific attacks. And so this is more of a general tactics for red teamers type thing. So when we look at containerization in Docker, if you haven't fooled around with it a lot, and I don't think a lot of penetration testers have, it's getting more and more popular. But as a means of developing an application, what you're looking at here is essentially an operating system level virtualization. That's versus VMware, where it's simulating the hardware of a system. We're talking about simulating the user space and file systems and libraries and networks for this container that's running. But everything, all the containers are sharing a single kernel. And so that means that it's a lot more lightweight. If you want to spin up a new Docker container from an image, it just takes a matter of seconds to do that. And it's very quick to do this, because it's a very lightweight type system. And so due to it being lightweight and due it to it being dynamic, and if you tag on something like Kubernetes or Docker swarm onto this to have these things scale dynamically with load or size of data, what you wind up with is a really easy way of developing applications out of these pre-built containers. And it eases deployment and development. So if I develop, and the reason why I'm doing this talk is I started developing an intelligence collection platform that used this to scale for large sets of data. And the useful thing for me for it was that when I developed it on my laptop, I could push that image out to the server, and it worked exactly the same as it did on my laptop. And then that's a very solid, guaranteed thing, as far as Docker's concerned. So development and deployment of these things winds up being relatively easy. When we look at vulnerabilities in these systems, it's important to look, and I think this is one of the most important overall themes for this talk is to look at the concept of vulnerabilities and layers of abstraction. I've talked about this in some of my previous talks when I talked about security of penetration testing, basically Opset for Pentesters. That's my previous few years of talks. I see vulnerabilities, the lifecycle, it doesn't begin with discovery. So you see these vulnerabilities lifecycle diagrams, and they begin with the discovery of the vulnerability, and then notification, and then patching, and on and so forth. And the cycle of life begins again. It really begins with a mistake. Developer made a mistake in the development of an application. They either didn't design security in, they made a typo in their programming. They committed some error that becomes exploitable in Sergey Brotis's weird machines terminology. Essentially, it's an unexpected execution. It's an unexpected functionality in a program that allows you to inject Turing complete code or to override a flag saying that you're an admin or something of that nature. And to me, I think that these vulnerabilities are mostly the result of not understanding the layer of abstraction directly beneath you. And so if you're a web application developer or not understanding the HTTP protocol that's underneath you as you're working in JavaScript and APIs and frameworks and things like that, if you don't understand how cookies work, but you're relying on your framework to do that for you, there can be an issue there. Textbook example of this is with C programmers. If you're a C program and you learned in a college course that didn't really dive into the implementation of C, you might think that these buffers are just going to be, I said I had 20 bytes for this buffer. Why has it even let me put any more in there? Well, C will happily let you do that, obviously. We all know this. And even if you know that and you know your program's going to crash, you don't really understand the implications of it. You write a C program and it crashes when the input's wrong. Segmentation fault, well, I screwed up and you go and you keep recompiling it until it stops doing that thing. But what you don't understand is what's really happening is you're going past that buffer and you're running over things on the program stack. If you don't understand that there's a program stack that's holding your local variables, then you didn't understand what's happening in that segmentation fault. Base pointers, return pointers, all that sort of thing that you can manipulate. And so all that sort of thing that if you go a level lower than C and you're looking at assembly and machine code and stepping through things with a debug, you learn to understand those sorts of things. But that's not at the layer of abstractions you're developing. And so you have your user experience, which is your end user, pointy clicky type things, your scripting languages, your high level, little compiled languages, machine code, things like what you think of as low level things like machine code and virtual memory and things like that. But underneath it, at some point it becomes a magic box. You have an understanding of how that computer works at a basic level, but it may not be the case. And even for myself, I think sometimes, well, maybe I should have done a degree in computer engineering instead of computer science. Because when it gets down to the transistors and the physical properties of silicon, I don't know, right? There's probably something to that that I don't understand that will bite me one day. And so for a hacker to keep up with these sorts of things, you've got two different directions to look in. One is in your layer of abstraction, higher, lower. And as a hacker, you tend to want to start learning lower and lower and lower so that you can get further underneath that developer's understanding of things. But horizontally, you've got different technologies like this and this diagram. You've got languages on the left-hand side and protocols and medium on the right-hand side. And so there's lots of different things. And it's hard to keep up with this. And so you have things that you specialize in. For our pentesting teams, we have folks that are very good at web applications. We have folks that are very good at network-based attacks. We have folks that are very good at looking at embedded control systems and things like that. And so you have your specializations, but you always want to be broadening this. If you're a developer, you usually move upwards in the stack, though. So we're moving downwards as penetration testers, red tamers, and things like that. But a developer is typically moving upwards. And so if you were a web developer in the late 90s, you might have been writing your code in C. I've written some web applications in C++, CGI. And it's not something that I'm proud of. And it's not something that I would deploy on anything internet-facing nowadays if I found the code. I need to find that code. But you could compile a program that outputted HTTP headers and HTML and stick it in a CGI bin directory and then wait to get owned. And then, well, that's not so smart. So we start putting scripts out there. And so that's a higher level of abstraction. And then we have web-specific languages. OK, things like, regrettably, PHP and things like that that are designed for the web. And then on top of that, we have frameworks. Things like Django that do a lot of things for you. So like Django, you've got your object model that handles all your database stuff. You've got all these connectors and everything. So where you don't have to understand HTTP, you just write the code that backends your website. And you have some templates for the front end. And it's kind of hiding things from you. But it's making it easier for the develop apps. As you go up these layers of abstraction, it becomes easier for you to develop applications. And then nowadays, it seems that most web applications are strictly running in JavaScript in your browser than hitting web APIs on the back end. And so that's another way of developing, another way of hiding some of this complexity. When we look at containerization, it's as if we can take any specific web application that's been developed almost, and you can look on the Docker Hub out there and pull down an image of it and have them talk to each other. And you can build a serviceable app out of these things with just some connecting code and some configuration files and some scripts to spin up the containers for them and to have them talk to each other. And so essentially, this is the next sort of meta level of developing an application. And so your mindset as an attacker versus a developer after you learn Hello World for something, you learn these things by looking at their tutorials. You learn them by reading the most basic introductory book that you can find on the topic. You learn them from trainings and tutorials and videos and things like that. Nobody sits down with K and RC and Learn C anymore. They're going to go through some tutorials first. So as a developer, what you're going to learn after you learn Hello World is what can I build with these language constructs? If I go through the tutorials, everything that I build is going to look like that tutorial software with modifications to fit my needs. I'm going to use the same approaches and the same practices. And the problem with security for that is that most of the time, tutorials sort of hand wave security for things in favor of getting something up and running in like a couple of screens worth of text on a website. If you're an attacker, the first thing that you do, like the first thing I did after I started doing Hello World in Docker so I could build a multi-container application is like, well, how does this work? How does this Hello World work? And so you start digging into, well, how to do these Docker containers look at each other. And so if you're an attacker or a developer, this abstraction allows for more efficient development. That's basically what we're talking about here. These layers of abstraction, essentially the idea of having these building block containers, you can have mixed technology in them. It doesn't matter if this one application is written in.NET. It doesn't matter if this one's written in Python. We'll actually look at an example in the video of an application like this. The way they talk to each other, they have to, at that point, speak over common protocols like TCP IP. They can't rely on them being PHP talking to PHP or C talking to C. These are technologies that have to communicate over the same sorts of networks that organizations operate on. And an interesting example of this is if you look at a lot of multi-container applications used Redis, and I might not be pronouncing it correctly, but it's essentially when I saw it in an application, I was like, what is this thing? And essentially it's a key value store. So it's essentially a variable storage container or a service. You can run it outside of a container. But you connect to it, and you say this variable name, assign it to this value. And you have some complex data structures in there, too. You can have some cues and stacks and things like that in there. And so instead of storing things in local variables in your container applications, since your containers are immutable, they can't maintain state if they go away and then come back, they'll store state inside of these Redis containers where other applications can come in and look at that data and use it. And so we'll look at some examples of that. But so instead of having a program stack in C, for example, where you'd have storage memory that pointers are pointing to locally, you now have them on this network based service. And so what this means is the developers are moving up the stack in abstraction, and as attackers are moving down, there's a few things happening. One, there's things higher on the stack that attackers need to learn to work with, things like containerization and modern web development technologies and things like that that we might not be very interested in. The good news, however, is that our existing techniques for penetration testing networks of systems apply directly to penetration testing inside of a multi-container app. And so you're existing now, you might be a penetration tester today, but once you start looking at web applications, now congratulations, you're an application security expert. And so your current skill set becomes more lucrative over time because you're going relatively lower in the stack to the developers. So think of somebody in the 70s or 80s who was writing assembly language code, and that's what they had to do just to write an application for a computer back then. Well, now they're very good. That person would be very well suited towards embedded systems development or other lower level things. And so when we're looking at these multi-container applications and application internals, we want to have things like control over execution, opportunity to turn code against itself, things like Rop and web APIs and CSurf and things like that. All these sorts of things, especially in case of malware analysis and return-oriented programming and CSurf, these things require very specialized tools. And sometimes you might even be internal to an application, to exploit a monolithic binary application. You're looking at having to attach this thing to a debugger just to get an understanding of how to develop an exploit for it. The skill set for this, it's different. So an internal application security expert, somebody who analyzes applications for vulnerabilities, needs to have a very strong understanding of the languages that they're targeting and methods of exploitation. And some of those can get quite arcane. But a penetration tester generally works on the outside attack surface. They're using known vulnerabilities or relatively easy to find vulnerabilities because they're targeting the attack surface of an entire organization at once. And so you might have been trained in doing some very simple buffer overflow exploits as a penetration tester. But if you're writing them every day in your daily work as a penetration tester, I'm going to be very surprised. If you're connecting to a remote service on a client and you're throwing the A's in until it crashes type of thing, and you're able to develop that exploit sort of blindly without seeing the other side of it, that's a kudos to you. But that's just not the norm in pen testing. It's more of an application security thing. And so there's a training gap there. But the nice news is with this containerization is that your existing skills of system to system pen testing apply much easier. So containerization allows for the design of applications that are composed in many of these things. And so we're democratizing post-exploitation manipulation. So essentially, after you gain access to one of these multi-container applications, there's lots of vulnerabilities in these things. So if you look on the Docker Hub, there'll be official Docker images for different web applications. But there'll also be a lot of Docker images that people made one off for web applications that at some point may not have had official Docker images. And so somebody will take it upon themselves to create an image for their own use, push it to their Docker Hub so that they can use it, and by default it's available for anybody else to go and grab and use in theirs. But they didn't sign up to be a package maintainer. They didn't sign up to maintain this thing over long periods of time. And so their use of it may have been over, or they may not care about the vulnerabilities in it. And so you'll see in the demo, there's a Joomla image out there that you can grab. And it's an old version of Joomla. There's an official Joomla image out there. But there's also a lot of, it isn't vulnerable. There's also a lot of backdoor images. And there was a news article a while back on somebody who had crypto miners embedded in these images. And so there's lots of things, and that's smart. I wish I'd have thought of that. So there's lots of vulnerability in the images themselves that can give you that entrance into the back end network for these things. And so you essentially have this Docker network underneath the horizontal line there, where your containers all work and can talk to each other. And then there's an outside attack surface of ports that are forwarded to the outside world from the host. And those ports are, and you specify those ports. But once you're on the inside of this network, it's kind of free for all. They can all talk to each other no matter what they've specified. And so essentially, a test of an application becomes a microcosm of an organization-wide test. Your actual post-exploitation of this sort of thing is going to involve identifying that you're on a Docker network, which will be pretty simple because you're going to find yourself on a system that doesn't have a lot of tools. It's purpose built for that one application. And so you might be getting a busy box shell or something. And the video will actually show you another technique that you can look at very quickly to see that you're on a Docker container network. And so your idea is once I gained access to one of these things, let's enumerate the rest of them. Let's find out what's on the rest of them. Let's figure out how they talk to each other and see if we can move around. And this is all analogous to on a monolithic application function hooking and breakpoints and inspection of memory, but it's at a network scale. And so the implementation of this is so you have access through conventional exploits. It's familiar territory for attackers with system or network level of experience. The limitations of this is you're living off the land. You're going to have to transfer your own tools in and just kind of statically compile them and hope for the best that they'll work once they're on there. It's challenging because the images are quite minimalistic. You're going to be doing a lot of port forwarding in order to get your tools on your system to reach in there and look at those other hosts. And so the demonstration video for this is a demonstration of exploitation and post exploitation of a multi container application. This is a Docker multi container application that's used frequently in public Docker trainings. It's out there on GitHub. The links are all in the white paper for the talk. This is used to teach people about Docker. And it's a voting application. It's meant for cats versus dogs. But I've changed it to blue team versus red team so we can have some fun. And we're going to attack this thing. We're going to look at how different aspects of it, like the voting interface is written in one language. The back end takes the votes as another language. The database is a Postgres database. And then there's a Redis queue that the votes get stuffed into and popped out of. It's like a stack, or a queue, actually, in this case. You have a data structure list in Redis where the votes are being pushed onto the left-hand side and then popped off the right-hand side as a queue. And so we'll bump over to our video here. If I can get my controls. My controls are hidden under the bar there. So this is a tour through the application itself. The voting app at the top there is written in Python. The result server is written with node and JavaScript. There's a Joomla image. In the tutorial of this, there's not really much external attack surface. So I dropped that vulnerable Joomla image in there just for the purposes of demonstration. You see in the networks there that it's showing Front Tier, Back Tier for some of these. Some of them are only Back Tier containers. There's two separate Docker networks. And it's essentially like a DMZ and then the internal network that we're talking on. There's a worker node that depends on Redis. And it's simply popping votes off of the Redis queue and then pushing them into a database. There's the Redis queue itself on the back end and then a database that the votes actually wind up in. And it's good that this is a video because there's a lot of moving parts to this that can go wrong. It took a few takes for this. And so we can start this thing up using Docker Swarm as instrumentation. And the details of all this are in the white paper. And so I'm moving through this quickly, but it's all there. So there's your voting interface. There's the results served off of another port. And there's that juicy, June-lammage there waiting to get popped. We can punch in votes and see the results. And we can even change our vote. We get a cookie that our vote is associated with, and so it can flip and flop as time changes. So we want to break it, right? And luckily, this particular June-lammage, there's a module in Metaspoit that can take care of hitting this thing. That's just making note of the local IP address of the attacker. You can see at the bottom of the screen the attacker target designation there, the asterisk designates which screen we're looking at at any moment. Right now we're looking at target. We're loading up that particular attack module. We're setting our target up, which we identified from the web interface. Boring Metaspoit stuff. Type faster, old Wesley. Setting our local hose, because that PHP interpreter's going to phone back to us. And it's a quick exploit. It just goes ahead and works. You're sitting in a PHP interpreter right now. You get some information about the system that you're on. And it'll be the same kernel version for every container, because they're all sharing in the kernel. We can drop to a shell. And this is showing you in the proc file system in the C groups, you can actually, without going into what all that means, you see a lot of references to Docker. And so that's first indication that we're on such a system. Now we're doing IP information for this Docker container that we're inside of. And there's an internal and an external network there. There's the 172.19 and 172.18 that are both those front tier and back tier networks. We can background that interpreter session. So we have that session sitting there. We have that persistence on that Joomla server. We're grabbing a statically compiled version of Nmap off GitHub in order to be able to scan that internal network. Statically compiled because we want to upload that, which is what we're doing right now, into a temp folder onto that compromised container so that we can start working from there. We have to have it statically compiled because we don't have a lot of library support on that target. And certainly Nmap's not already there unless you've broken into a Cali image container. They're fantastic. I like working with them. I think the actual attacker pane on this interface is running in a Cali container. And so here we're in mapping the 172.18 network, which is the external network, the front tier network. All ports on the first 10 hosts. By these default configurations of Docker, they're sort of incrementally assigned IP address numbers. And so you see, and there's actually good naming for most of them except for that Jumla server that I didn't really give a good name for. And also the Docker host is the first one up there. So that gives you an idea of those three HTTP services that we're looking at, the front end voting interface, the results application in the Jumla. Now the scan is going across the back tier because now we have access to one of these containers. Now we can start scanning the back tier network and seeing what's there. It's going to be a lot more interesting stuff. It's going to be stuff that we can play around with. And so you've got the Redis server down at the bottom. The next one up is the Jumla instance, the Postgres database there. And again, the Docker host up there at the top at dot one. And so now we've got a database and a Redis server we can mess around with. So now we can start manipulating local variables. So we're back into our interpreter console, all running on our Jumla instance. And we're going to set up some port forwarding so that we can interact with those services directly. Basically, we're creating a tunnel from our external attacker image or container or virtual machine or your local instance of Cali, Metasploit. And so we set a port forward here to where we can get at that database. And we can just use the regular old Postgres command line to interact with it. And the thing is, by default, with all these Docker images, they're meant to get up and running very quickly. So many of them are configured to just have default or no credentials, no authentication, nothing. Because they're all sitting on those back tier networks where nobody can get at them directly. And so once you are in there, you're usually good to talk to each other unless somebody's done a lot of special configuration, which is not in the tutorials. And so here we're looking at the tables and looking at the votes that are in there. The votes take the form of an ID associated with that voter. And also which one they voted. A is red and B is blue, I guess. I don't quite remember. But then we've also done some inserts to see more votes being put in for blue. So blue team members can celebrate for now. Now, as passed, Wesley just pauls their brain farting, I guess. Now we're back into a shell. And we're going to, since we're just doing Netcat for the reddest, we don't really need to set up a port for it. We can Netcat from that host. A lot of times you'll have Busybox, which we'll have Netcat built in. Here we've gone into monitor mode on red. So once you tell that into a reddest instance, all the commands are plain text. And monitor allows you to see what's going on in live feed of interactions. And so all those pops are that worker container, continuously looking. Is there a new vote? Is there a new vote? Is there a new vote? Because when it finds a new vote, it'll pop it and throw it into the database. And so it's continuously doing that. And as I hit the votes here, you'll see things change. And I'll move the window out of the way there so you can see it. So you see votes being pushed onto the right-hand side and popped off the left-hand side. And so now that we know how this works, we kind of got a feel for that. It's almost like watching an instrument and a debugger to see how does this part of the program talk to this other part of the program. We can now start playing with it directly. And so now that we're in there, I'm copying and pasting a bunch of push commands into here so that we can push a bunch of arbitrary votes into this thing. And it's just as good as pushing them from the web interface. And so I've posted in a ton of votes there with unique IDs of just alphabetic letters. It doesn't verify that they're valid or associated with any cookie or anything. It's just sort of there. It's a toy application. And so now the red team's winning, of course. And there I am rubbing it in about highlighting it. And so the takeaway from this is that your existing offense-oriented skills, none of that was very difficult. This is not zero-day. This is just regular old tactical post-exploitation stuff that you'll learn as a pen tester if you haven't already learned it if you're getting started. These are things that will be in classes and stuff. But these existing offense skills become useful at a lower relative position in the stack of abstraction from developers as they're using higher and higher abstracted technologies. And so developers are moving up. And so the new low level moves up. But it's important that you update yourself. And so for us, we don't want the first multi-container application we encounter to be on a client gig. We don't want to have to learn it on the spot then. We want to have some familiarization with things like Docker and Kubernetes and other technologies for newer development methodologies so that we are familiar with these things when we see them. And we don't freak out when we break into a Juma instance and see that there are no tools in place and don't even think to look to see if there's another back end network that this thing can talk to. And so it's important to update yourself and not only move down the stack in terms of getting lower and lower and lower levels of skills for exploitation, but work up the stack as well. If something seems cool and trendy, like cloud or containerization or blockchain or anything like that, it's really easy to turn your nose up at it, right? Because it's trendy and you don't want to be trendy. But ultimately, your clients are going to wind up using these technologies and you need to understand them. And so it represents a good entrance into application security testing for existing penetration testers. There's a white paper available in the conference materials. It has all the information about the demo and a couple of other demos. There's a lot of discussion in that that's not in this talk and a lot of links to resources that I used when I was learning about this stuff, previous work that's being done in security of these things, and also some pointers to some information on how to secure these applications, if that's your thing. I appreciate your time and I'll be down here ready for questions and to hand out cards and trade contact information. Thank you. Thank you.
 For our next talk, we have Wes McGrew.  He's going to talk to us about post exploitation  of Docker containers.  Thank you.  So the target audience for this talk  is unapologetically red team members,  people who are doing red teaming engagements,  other offense-oriented services, such as penetration  testing, application security testing.  And the title of the talk, I always  have long titles for my talks.  There's the clever bit, and then there's the actual part of it.  And so it's a look at multi-container applications.  So building, nowadays with containerization technology  in Docker, which I'll cover in a little more detail,  you can build an application basically  out of these LEGO blocks of Docker images or other containers  that you find on the internet.  And so portions of your application  can be made up of these immutable containers.  And what that means is that you're  working at a higher level of abstraction,  and all of your vulnerabilities come  from these lower levels of abstraction,  where these Docker images have vulnerabilities,  and the way they talk to each other in a non-authenticated way  on the back end can cause issues.  And so the goal of this talk is to give a penetration testing  practitioner who's not already familiar with Docker  or is facing multi-container applications  a little bit of tactics and tools  to use to examine these systems.  And so my title at Horn Cyber Solutions  is director of cyber operations.  And so what that means is that I oversee all offense-oriented  services.  I run teams that do multiple teams that  are simultaneously doing pin tests, red teaming engagements,  and application security testing for a number  of different clients.  And so I try to get hands on with it as much as I can.  But really, I'm more interested in the management  of all of these sorts of things and making sure  that all the teams are up to date on the latest tools  and techniques.  In a previous life, I was a professor at Mississippi State  University.  I still adjunct there.  I teach class every semester.  And there I worked on industrial control systems,  developed a reverse engineering program there  for getting the NSA, CAE, and research, education,  cyber operations.  And so basically, most of my research, most of my work,  has been on the offense side of things.  And so if you're a blue team in here,  you might find things a little bit lean.  But you can learn how to break these things  and say that you can then look out for these tactics  and techniques.  The intention for this deck, really,  is to make a strong point about the relationship  between an attacker's skill level, your skill set,  the set of things that you know as an attacker,  and your development as an attacker over time,  versus developer trends.  And so this talk isn't just about attacking Docker containers  specifically, or even multi-container applications  specifically.  It's really about how do you develop your skills over time  as an attacker, whether malicious or not.  And so how do we leverage what you already know  and how to look at learning new technologies moving forward  and the sort of attitude to approach it with?  We want to provide a hacker that's  experienced in exploitation and post-exploitation  of networks of systems.  And so if you think of a penetration test  of an organization, you're given the scope  and you're scanning that scope.  And you have a set of systems that interact with each other  that's your attack surface, the set of things  that you can interact with.  How do you move and take your, most of your training  as a penetration tester or a red teamer  has been in that mode.  Looking at that attack surface, looking  at protocols between services, looking  at vulnerabilities on the outside edge  of each of those individual systems.  How do you move and take that skill set  and apply it to an application?  And so historically applications have been monolithic.  And so a monolithic application is something  that's made up of primarily a single binary  or a single service running on a single system  or a single virtual machine.  And so with a monolithic application,  if you're wanting to inspect the state of internal things  to the application, how functions call each other,  how data is stored in memory, you're  looking at attaching as a debugger,  you're looking at setting breakpoints,  you're looking at exploitation in terms of memory corruption  or other sort of remote code execution,  to get to the point that you can examine these things  and then you're in this sort of environment of a debugger  or something where you're limited in how you move.  With multi-container applications,  where you're building your application out  of these immutable images, talking to each other,  a lot of times your variable storage and your function calls  wind up taking the place of TCP IP connections  wind up taking the place of things  like Redis databases that actually store what would  normally be a local variable.  And so it allows you to explore the internals  of an application in the same way that you  penetration test a network.  So I'm going to demonstrate this with some concrete Docker  example, with a concrete Docker example.  The white paper for this talk has a set of different things  to sort of step up to this, but I'm  going to show you the cool one first.  And the inspiration for this approach and this talk  is what I consider a classic talk by H.D.  Moran Valsmith from DEF CON 15, tactical exploitation.  It's quite an old talk, I guess, now.  But it's actually well worth going back and looking at  if you're a pentester and you never saw it.  All of the work in any kind of field  is built on the shoulders of giants,  built on the shoulders of people who have come before you  and doing this sort of work.  There's been a number of talks on Docker and containers.  Overall, there's DockerCon and general purpose Docker talks  out there in research, but specifically in security  and in previous DEF CONs and Blackhats,  we have here a list of good references, good material  to read to prepare yourself for this sort of thing.  You've got Mortman, Greta Fiori, Bettini, Churny, Dolce.  And these are all great works, but they all focus  on different things than what we're focusing on here.  They either focus on the underlying implementation  or architecture of Docker, the way it's implemented  at a kernel level, which for the purpose of this talk  isn't exactly the most important thing.  And having some advice for that low level security  sort of things.  And then there's vulnerabilities in the platform itself,  in your Docker run times, in your Docker infrastructure,  and the way that developers use Docker  and how you can target Docker developers.  The thing is, if I get on stage here  and I present a Docker talk that has Docker zero day,  well, in a couple weeks, everybody will have updated,  well, most everybody, except for that one client, right?  They'll have updated their Docker instances  and those vulnerabilities don't work anymore.  So really, you want something more timeless.  And so what you're looking at really here  is post-exploitation as being something more useful  to learn in the long term, because you're always  going to find a way into one of these applications  through vulnerability in its outer attack surface.  And so the focus of all those other talks  has been on defense as well, or at least on specific attacks.  And so this is more of a general tactics for red teamers  type thing.  So when we look at containerization in Docker,  if you haven't fooled around with it a lot,  and I don't think a lot of penetration testers have,  it's getting more and more popular.  But as a means of developing an application, what you're  looking at here is essentially an operating system level  virtualization.  That's versus VMware, where it's simulating  the hardware of a system.  We're talking about simulating the user space and file systems  and libraries and networks for this container that's running.  But everything, all the containers  are sharing a single kernel.  And so that means that it's a lot more lightweight.  If you want to spin up a new Docker container from an image,  it just takes a matter of seconds to do that.  And it's very quick to do this, because it's  a very lightweight type system.  And so due to it being lightweight and due it  to it being dynamic, and if you tag on something  like Kubernetes or Docker swarm onto this  to have these things scale dynamically with load  or size of data, what you wind up with  is a really easy way of developing applications out  of these pre-built containers.  And it eases deployment and development.  So if I develop, and the reason why I'm doing this talk  is I started developing an intelligence collection  platform that used this to scale for large sets of data.  And the useful thing for me for it  was that when I developed it on my laptop,  I could push that image out to the server,  and it worked exactly the same as it did on my laptop.  And then that's a very solid, guaranteed thing,  as far as Docker's concerned.  So development and deployment of these things  winds up being relatively easy.  When we look at vulnerabilities in these systems,  it's important to look, and I think  this is one of the most important overall themes  for this talk is to look at the concept of vulnerabilities  and layers of abstraction.  I've talked about this in some of my previous talks  when I talked about security of penetration testing, basically  Opset for Pentesters.  That's my previous few years of talks.  I see vulnerabilities, the lifecycle,  it doesn't begin with discovery.  So you see these vulnerabilities lifecycle diagrams,  and they begin with the discovery of the vulnerability,  and then notification, and then patching, and on and so forth.  And the cycle of life begins again.  It really begins with a mistake.  Developer made a mistake in the development of an application.  They either didn't design security in,  they made a typo in their programming.  They committed some error that becomes  exploitable in Sergey Brotis's weird machines terminology.  Essentially, it's an unexpected execution.  It's an unexpected functionality in a program  that allows you to inject Turing complete code  or to override a flag saying that you're an admin or something  of that nature.  And to me, I think that these vulnerabilities are mostly  the result of not understanding the layer of abstraction  directly beneath you.  And so if you're a web application developer  or not understanding the HTTP protocol that's  underneath you as you're working in JavaScript and APIs  and frameworks and things like that,  if you don't understand how cookies work,  but you're relying on your framework to do that for you,  there can be an issue there.  Textbook example of this is with C programmers.  If you're a C program and you learned in a college course  that didn't really dive into the implementation of C,  you might think that these buffers are just going to be,  I said I had 20 bytes for this buffer.  Why has it even let me put any more in there?  Well, C will happily let you do that, obviously.  We all know this.  And even if you know that and you know your program's  going to crash, you don't really understand the implications  of it.  You write a C program and it crashes when the input's wrong.  Segmentation fault, well, I screwed up and you go and you  keep recompiling it until it stops doing that thing.  But what you don't understand is what's really happening  is you're going past that buffer and you're running over  things on the program stack.  If you don't understand that there's a program stack that's  holding your local variables, then you didn't understand  what's happening in that segmentation fault.  Base pointers, return pointers, all that sort of thing  that you can manipulate.  And so all that sort of thing that if you go a level lower  than C and you're looking at assembly and machine code  and stepping through things with a debug, you learn to  understand those sorts of things.  But that's not at the layer of abstractions you're  developing.  And so you have your user experience, which is your end  user, pointy clicky type things, your scripting  languages, your high level, little compiled languages,  machine code, things like what you think of as low level  things like machine code and virtual memory and things like  that.  But underneath it, at some point it becomes a magic box.  You have an understanding of how that computer works at a  basic level, but it may not be the case.  And even for myself, I think sometimes, well, maybe I should  have done a degree in computer engineering instead of  computer science.  Because when it gets down to the transistors and the physical  properties of silicon, I don't know, right?  There's probably something to that that I don't understand  that will bite me one day.  And so for a hacker to keep up with these sorts of things,  you've got two different directions to look in.  One is in your layer of abstraction, higher, lower.  And as a hacker, you tend to want to start learning lower  and lower and lower so that you can get further underneath  that developer's understanding of things.  But horizontally, you've got different technologies like  this and this diagram.  You've got languages on the left-hand side and protocols  and medium on the right-hand side.  And so there's lots of different things.  And it's hard to keep up with this.  And so you have things that you specialize in.  For our pentesting teams, we have folks that are very good  at web applications.  We have folks that are very good at network-based attacks.  We have folks that are very good at looking at embedded  control systems and things like that.  And so you have your specializations, but you always  want to be broadening this.  If you're a developer, you usually move upwards in the  stack, though.  So we're moving downwards as penetration testers, red  tamers, and things like that.  But a developer is typically moving upwards.  And so if you were a web developer in the late 90s, you  might have been writing your code in C. I've written some  web applications in C++, CGI.  And it's not something that I'm proud of.  And it's not something that I would deploy on anything  internet-facing nowadays if I found the code.  I need to find that code.  But you could compile a program that outputted HTTP  headers and HTML and stick it in a CGI bin directory and then  wait to get owned.  And then, well, that's not so smart.  So we start putting scripts out there.  And so that's a higher level of abstraction.  And then we have web-specific languages.  OK, things like, regrettably, PHP and things like that that  are designed for the web.  And then on top of that, we have frameworks.  Things like Django that do a lot of things for you.  So like Django, you've got your object model that handles  all your database stuff.  You've got all these connectors and everything.  So where you don't have to understand HTTP, you just  write the code that backends your website.  And you have some templates for the front end.  And it's kind of hiding things from you.  But it's making it easier for the develop apps.  As you go up these layers of abstraction, it becomes easier  for you to develop applications.  And then nowadays, it seems that most web applications are  strictly running in JavaScript in your browser than  hitting web APIs on the back end.  And so that's another way of developing, another way of  hiding some of this complexity.  When we look at containerization, it's as if we  can take any specific web application that's been  developed almost, and you can look on the Docker Hub out  there and pull down an image of it and have them talk to  each other.  And you can build a serviceable app out of these  things with just some connecting code and some  configuration files and some scripts to spin up the  containers for them and to have them talk to each other.  And so essentially, this is the next sort of meta level of  developing an application.  And so your mindset as an attacker versus a developer  after you learn Hello World for something, you learn these  things by looking at their tutorials.  You learn them by reading the most basic introductory book  that you can find on the topic.  You learn them from trainings and tutorials and videos and  things like that.  Nobody sits down with K and RC and Learn C anymore.  They're going to go through some tutorials first.  So as a developer, what you're going to learn after you  learn Hello World is what can I build with these language  constructs?  If I go through the tutorials, everything that I build is  going to look like that tutorial software with  modifications to fit my needs.  I'm going to use the same approaches and the same  practices.  And the problem with security for that is that most of the  time, tutorials sort of hand wave security for things in  favor of getting something up and running in like a couple of  screens worth of text on a website.  If you're an attacker, the first thing that you do, like  the first thing I did after I started doing Hello World in  Docker so I could build a multi-container application  is like, well, how does this work?  How does this Hello World work?  And so you start digging into, well, how to do these Docker  containers look at each other.  And so if you're an attacker or a developer, this abstraction  allows for more efficient development.  That's basically what we're talking about here.  These layers of abstraction, essentially the idea of having  these building block containers, you can have mixed  technology in them.  It doesn't matter if this one application is written in.NET.  It doesn't matter if this one's written in Python.  We'll actually look at an example in the video of an  application like this.  The way they talk to each other, they have to, at that point,  speak over common protocols like TCP IP.  They can't rely on them being PHP talking to PHP or C  talking to C. These are technologies that have to  communicate over the same sorts of networks that  organizations operate on.  And an interesting example of this is if you look at a lot  of multi-container applications used Redis, and I  might not be pronouncing it correctly, but it's essentially  when I saw it in an application, I was like, what is  this thing?  And essentially it's a key value store.  So it's essentially a variable storage container or a  service.  You can run it outside of a container.  But you connect to it, and you say this variable name, assign  it to this value.  And you have some complex data structures in there, too.  You can have some cues and stacks and things  like that in there.  And so instead of storing things in local variables in  your container applications, since your containers are  immutable, they can't maintain state if they go away and  then come back, they'll store state inside of these Redis  containers where other applications can come in and  look at that data and use it.  And so we'll look at some examples of that.  But so instead of having a program stack in C, for  example, where you'd have storage memory that pointers  are pointing to locally, you now have them on this network  based service.  And so what this means is the developers are moving up the  stack in abstraction, and as attackers are moving down,  there's a few things happening.  One, there's things higher on the stack that attackers need  to learn to work with, things like containerization and  modern web development technologies and things like  that that we might not be very interested in.  The good news, however, is that our existing techniques for  penetration testing networks of systems apply directly to  penetration testing inside of a multi-container app.  And so you're existing now, you might be a penetration  tester today, but once you start looking at web  applications, now congratulations, you're an  application security expert.  And so your current skill set becomes more lucrative over  time because you're going relatively lower in the stack  to the developers.  So think of somebody in the 70s or 80s who was writing  assembly language code, and that's what they had to do  just to write an application for a computer back then.  Well, now they're very good.  That person would be very well suited towards embedded  systems development or other lower level things.  And so when we're looking at these multi-container  applications and application internals, we want to have  things like control over execution, opportunity to  turn code against itself, things like Rop and web APIs and  CSurf and things like that.  All these sorts of things, especially in case of malware  analysis and return-oriented programming and CSurf, these  things require very specialized tools.  And sometimes you might even be internal to an application,  to exploit a monolithic binary application.  You're looking at having to attach this thing to a debugger  just to get an understanding of how to  develop an exploit for it.  The skill set for this, it's different.  So an internal application security expert, somebody who  analyzes applications for vulnerabilities, needs to have  a very strong understanding of the languages that they're  targeting and methods of exploitation.  And some of those can get quite arcane.  But a penetration tester generally works on the outside  attack surface.  They're using known vulnerabilities or relatively  easy to find vulnerabilities because they're targeting the  attack surface of an entire organization at once.  And so you might have been trained in doing some very  simple buffer overflow exploits as a penetration tester.  But if you're writing them every day in your daily work as  a penetration tester, I'm going to be very surprised.  If you're connecting to a remote service on a client and  you're throwing the A's in until it crashes type of thing,  and you're able to develop that exploit sort of blindly  without seeing the other side of it, that's a kudos to you.  But that's just not the norm in pen testing.  It's more of an application security thing.  And so there's a training gap there.  But the nice news is with this containerization is that your  existing skills of system to system pen testing apply much  easier.  So containerization allows for the design of applications  that are composed in many of these things.  And so we're democratizing post-exploitation manipulation.  So essentially, after you gain access to one of these  multi-container applications, there's lots of  vulnerabilities in these things.  So if you look on the Docker Hub, there'll be official  Docker images for different web applications.  But there'll also be a lot of Docker images that people made  one off for web applications that at some point may not have  had official Docker images.  And so somebody will take it upon themselves to create an  image for their own use, push it to their Docker Hub so that  they can use it, and by default it's available for  anybody else to go and grab and use in theirs.  But they didn't sign up to be a package maintainer.  They didn't sign up to maintain this thing over long  periods of time.  And so their use of it may have been over, or they may not  care about the vulnerabilities in it.  And so you'll see in the demo, there's a Joomla image out  there that you can grab.  And it's an old version of Joomla.  There's an official Joomla image out there.  But there's also a lot of, it isn't vulnerable.  There's also a lot of backdoor images.  And there was a news article a while back on somebody who  had crypto miners embedded in these images.  And so there's lots of things, and that's smart.  I wish I'd have thought of that.  So there's lots of vulnerability in the images  themselves that can give you that entrance into the  back end network for these things.  And so you essentially have this Docker network underneath  the horizontal line there, where your containers all work  and can talk to each other.  And then there's an outside attack surface of ports that  are forwarded to the outside world from the host.  And those ports are, and you specify those ports.  But once you're on the inside of this network, it's kind of  free for all.  They can all talk to each other no matter  what they've specified.  And so essentially, a test of an application becomes a  microcosm of an organization-wide test.  Your actual post-exploitation of this sort of thing is going  to involve identifying that you're on a Docker network,  which will be pretty simple because you're going to find  yourself on a system that doesn't have a lot of tools.  It's purpose built for that one application.  And so you might be getting a busy box shell or something.  And the video will actually show you another technique  that you can look at very quickly to see that you're on  a Docker container network.  And so your idea is once I gained access to one of these  things, let's enumerate the rest of them.  Let's find out what's on the rest of them.  Let's figure out how they talk to each other and see if we  can move around.  And this is all analogous to on a monolithic application  function hooking and breakpoints and inspection of  memory, but it's at a network scale.  And so the implementation of this is so you have access  through conventional exploits.  It's familiar territory for attackers with system or  network level of experience.  The limitations of this is you're living off the land.  You're going to have to transfer your own tools in and  just kind of statically compile them and hope for the best  that they'll work once they're on there.  It's challenging because the images are quite minimalistic.  You're going to be doing a lot of port forwarding in order to  get your tools on your system to reach in there and look at  those other hosts.  And so the demonstration video for this is a demonstration  of exploitation and post exploitation of a multi  container application.  This is a Docker multi container application that's used  frequently in public Docker trainings.  It's out there on GitHub.  The links are all in the white paper for the talk.  This is used to teach people about Docker.  And it's a voting application.  It's meant for cats versus dogs.  But I've changed it to blue team versus red team so we can  have some fun.  And we're going to attack this thing.  We're going to look at how different aspects of it, like  the voting interface is written in one language.  The back end takes the votes as another language.  The database is a Postgres database.  And then there's a Redis queue that the votes get stuffed  into and popped out of.  It's like a stack, or a queue, actually, in this case.  You have a data structure list in Redis where the votes are  being pushed onto the left-hand side and then popped off the  right-hand side as a queue.  And so we'll bump over to our video here.  If I can get my controls.  My controls are hidden under the bar there.  So this is a tour through the application itself.  The voting app at the top there is written in Python.  The result server is written with node and JavaScript.  There's a Joomla image.  In the tutorial of this, there's not really much external  attack surface.  So I dropped that vulnerable Joomla image in there just  for the purposes of demonstration.  You see in the networks there that it's showing Front Tier,  Back Tier for some of these.  Some of them are only Back Tier containers.  There's two separate Docker networks.  And it's essentially like a DMZ and then the internal  network that we're talking on.  There's a worker node that depends on Redis.  And it's simply popping votes off of the Redis queue and then  pushing them into a database.  There's the Redis queue itself on the back end and then a  database that the votes actually wind up in.  And it's good that this is a video because there's a lot  of moving parts to this that can go wrong.  It took a few takes for this.  And so we can start this thing up using Docker Swarm as  instrumentation.  And the details of all this are in the white paper.  And so I'm moving through this quickly, but it's all there.  So there's your voting interface.  There's the results served off of another port.  And there's that juicy, June-lammage there  waiting to get popped.  We can punch in votes and see the results.  And we can even change our vote.  We get a cookie that our vote is associated with, and so it  can flip and flop as time changes.  So we want to break it, right?  And luckily, this particular June-lammage, there's a  module in Metaspoit that can take care of hitting this  thing.  That's just making note of the local IP address of the  attacker.  You can see at the bottom of the screen the attacker target  designation there, the asterisk designates which  screen we're looking at at any moment.  Right now we're looking at target.  We're loading up that particular attack module.  We're setting our target up, which we identified from the  web interface.  Boring Metaspoit stuff.  Type faster, old Wesley.  Setting our local hose, because that PHP  interpreter's going to phone back to us.  And it's a quick exploit.  It just goes ahead and works.  You're sitting in a PHP interpreter right now.  You get some information about the system that you're on.  And it'll be the same kernel version for every container,  because they're all sharing in the kernel.  We can drop to a shell.  And this is showing you in the proc file system in the C  groups, you can actually, without going into what all  that means, you see a lot of references to Docker.  And so that's first indication that we're  on such a system.  Now we're doing IP information for this Docker  container that we're inside of.  And there's an internal and an external network there.  There's the 172.19 and 172.18 that are both those front  tier and back tier networks.  We can background that interpreter session.  So we have that session sitting there.  We have that persistence on that Joomla server.  We're grabbing a statically compiled version of Nmap off  GitHub in order to be able to scan that internal network.  Statically compiled because we want to upload that, which is  what we're doing right now, into a temp folder onto that  compromised container so that we can start working from there.  We have to have it statically compiled because we don't have  a lot of library support on that target.  And certainly Nmap's not already there unless you've  broken into a Cali image container.  They're fantastic.  I like working with them.  I think the actual attacker pane on this interface is running  in a Cali container.  And so here we're in mapping the 172.18 network, which is  the external network, the front tier network.  All ports on the first 10 hosts.  By these default configurations of Docker, they're  sort of incrementally assigned IP address numbers.  And so you see, and there's actually good naming for most  of them except for that Jumla server that I didn't really  give a good name for.  And also the Docker host is the first one up there.  So that gives you an idea of those three HTTP services  that we're looking at, the front end voting interface, the  results application in the Jumla.  Now the scan is going across the back tier because now we  have access to one of these containers.  Now we can start scanning the back tier network and seeing  what's there.  It's going to be a lot more interesting stuff.  It's going to be stuff that we can play around with.  And so you've got the Redis server down at the bottom.  The next one up is the Jumla instance, the Postgres  database there.  And again, the Docker host up there at the top at dot one.  And so now we've got a database and a Redis server we  can mess around with.  So now we can start manipulating local variables.  So we're back into our interpreter console, all  running on our Jumla instance.  And we're going to set up some port forwarding so that we  can interact with those services directly.  Basically, we're creating a tunnel from our external  attacker image or container or virtual machine or your local  instance of Cali, Metasploit.  And so we set a port forward here to where we can get at  that database.  And we can just use the regular old Postgres command line to  interact with it.  And the thing is, by default, with all these Docker images,  they're meant to get up and running very quickly.  So many of them are configured to just have default or no  credentials, no authentication, nothing.  Because they're all sitting on those back tier networks  where nobody can get at them directly.  And so once you are in there, you're usually good to talk  to each other unless somebody's done a lot of special  configuration, which is not in the tutorials.  And so here we're looking at the tables and looking at the  votes that are in there.  The votes take the form of an ID associated with that voter.  And also which one they voted.  A is red and B is blue, I guess.  I don't quite remember.  But then we've also done some inserts to see more votes  being put in for blue.  So blue team members can celebrate for now.  Now, as passed, Wesley just pauls their  brain farting, I guess.  Now we're back into a shell.  And we're going to, since we're just doing Netcat for the  reddest, we don't really need to set up a port for it.  We can Netcat from that host.  A lot of times you'll have Busybox, which we'll have  Netcat built in.  Here we've gone into monitor mode on red.  So once you tell that into a reddest instance, all the  commands are plain text.  And monitor allows you to see what's going on in live feed  of interactions.  And so all those pops are that worker container,  continuously looking.  Is there a new vote?  Is there a new vote?  Is there a new vote?  Because when it finds a new vote, it'll pop it and throw  it into the database.  And so it's continuously doing that.  And as I hit the votes here, you'll see things change.  And I'll move the window out of the way there so you can see  it.  So you see votes being pushed onto the right-hand side and  popped off the left-hand side.  And so now that we know how this works, we kind of got a  feel for that.  It's almost like watching an instrument and a debugger to  see how does this part of the program talk to this other  part of the program.  We can now start playing with it directly.  And so now that we're in there, I'm copying and pasting a  bunch of push commands into here so that we can push a  bunch of arbitrary votes into this thing.  And it's just as good as pushing them  from the web interface.  And so I've posted in a ton of votes there with unique IDs  of just alphabetic letters.  It doesn't verify that they're valid or associated with any  cookie or anything.  It's just sort of there.  It's a toy application.  And so now the red team's winning, of course.  And there I am rubbing it in about highlighting it.  And so the takeaway from this is that your existing  offense-oriented skills, none of that was very difficult.  This is not zero-day.  This is just regular old tactical post-exploitation  stuff that you'll learn as a pen tester if you haven't  already learned it if you're getting started.  These are things that will be in classes and stuff.  But these existing offense skills become useful at a  lower relative position in the stack of abstraction from  developers as they're using higher and  higher abstracted technologies.  And so developers are moving up.  And so the new low level moves up.  But it's important that you update yourself.  And so for us, we don't want the first multi-container  application we encounter to be on a client gig.  We don't want to have to learn it on the spot then.  We want to have some familiarization with things like  Docker and Kubernetes and other technologies for newer  development methodologies so that we are familiar with  these things when we see them.  And we don't freak out when we break into a Juma instance  and see that there are no tools in place and don't even  think to look to see if there's another back end network  that this thing can talk to.  And so it's important to update yourself and not only move  down the stack in terms of getting lower and lower and  lower levels of skills for exploitation, but work up the  stack as well.  If something seems cool and trendy, like cloud or  containerization or blockchain or anything like that,  it's really easy to turn your nose up at it, right?  Because it's trendy and you don't want to be trendy.  But ultimately, your clients are going to wind up using  these technologies and you need to understand them.  And so it represents a good entrance into application  security testing for existing penetration testers.  There's a white paper available in the conference  materials.  It has all the information about the demo and a couple of  other demos.  There's a lot of discussion in that that's not in this talk  and a lot of links to resources that I used when I was  learning about this stuff, previous work that's being  done in security of these things, and also some pointers  to some information on how to secure these applications, if  that's your thing.  I appreciate your time and I'll be down here ready for  questions and to hand out cards and trade contact  information.  Thank you.  Thank you.
DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_0_14591.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_14592_21359.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_21360_23987.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_23988_29795.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_29796_36971.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_36972_41807.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_41808_42431.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_42432_42503.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_42504_42551.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_42552_42671.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_42672_43727.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_43728_43799.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_43800_44195.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_44196_44639.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_44640_44891.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_44892_44939.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_44940_45035.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_45036_45335.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_45336_47939.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_47940_48131.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_48132_48431.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_48432_49607.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_49608_51011.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_51012_51203.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_51204_51323.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_51324_51527.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_51528_52967.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_52968_53207.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_53208_53387.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_53388_54491.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_54492_54587.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_54588_54719.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_54720_54767.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_54768_55223.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_55224_55703.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_55704_55979.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_55980_56027.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_56028_56183.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_56184_56399.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_56400_60239.png DEF_CON_26_-_Wesley_McGrew_-_An_Attacker_Looks_at_Docker_Approaching_Multi_Container_60240_60278.png