D0 N0 H4RM
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Dameff, Christian (quaddi)
Wilkerson, Jessica
Luft, Ash
Murthy, Vidya
Schmitt, Veronica
r3plicant
It is certainly a time of discovery- though the truths revealed by the COVID-19 crisis can be bitter and bleak. At a time when all attention is focused on the ERs and ICUs that make up the battle’s front lines, it is easy to cast aside old warnings to focus solely on the clinical war. But the need for safety and security only increases in the face of a pandemic- and healthcare cybersecurity is no different. From testing to ventilators, every facet of our response to COVID-19 depends on trustworthy and reliable technology. D0 No H4rm- DEF CON’s continuing conversation on healthcare returns for another up close (but not too close) and personal dialogue between hackers at the top of their fields- from the halls of the FDA to the cutting edge of medical devices security research for an all-encompassing look at what we need to focus on in the age of COVID. Moderated by physician hackers quaddi and r3plicant, this perennially packed event aims to recruit the talent, ingenuity, and vision of the DEF CON family for the challenges we face both now and after the immediate crisis passes.
 Welcome, DEF CON 28, the Do No Harm panel. This is a healthcare security conversation. And for the next 45 minutes, followed by a 45 minute Q&A, you're gonna hear from some of the world's top experts on the healthcare cybersecurity space. Welcome. Thanks, Quadi. I'm Replicant. We're going to get out of the way here in just a moment and introduce you to our panelists, but obviously a little bit different of a venue for us than the last couple of years. We hope that everyone is staying safe and healthy, and we're thinking of you and look forward to when we can hang out together in person. Without any further ado, I'm gonna let our panelists take it away and say hi. Hi, everyone. My name is Ash Left. I am a biochemist, a computer scientist, and an electrical engineer. I work at a medical device firm called Starfish in Canada, and I'm a suffer engineer there and an advocate for security and privacy. I think that means that I'm next. So I don't have a handle, unfortunately. So I am just Jessica Wilkerson. I'm a cyber policy analyst with the Food and Drug Administration, and I work on cybersecurity policy there for medical devices. Some of you may know me from my previous time with the United States Congress Energy and Commerce Committee where I worked on cybersecurity policy for them. Very nice to talk with everyone and looking forward to the panel. Thank you. Hi, this is me. Everyone should know me. The short, the sweet, the sassy, the spunky. I am leaving my company for the wildfield in Norway where I'll be having a whole host of millions to help me do medical device research. Currently doing independent research in formatronic, adding them to get more visualization on their products and their security. I'm an advocate for patient rights as well as keeping things safe, making it, keeping it simple, keeping it stupid, as I'm saying. And that's me, short and sweet. That leaves me. My name is Vidya Morthy. I work for MedCrypt, a startup in the space of bringing cybersecurity features to medical devices. I absolutely am passionate about the subject matter and think that we're at an inflection point here and there's gonna be a big amount of change as we go forward. Well, on behalf of Quadim, myself, we are just so incredibly grateful to have an amazing panel this year. And for those of us who are joining us for the first time, and this is kind of one of the silver linings, I think, about this format is that in past years, we've had folks who wanted to kind of get in on this conversation that weren't able to due to space constraints. So we're really thankful for the opportunity to have anybody who wants to come in, listen in, join with us to be able to do so. So for those who may not be familiar, what we're gonna do is we're gonna have a little conversation here between the four of these awesome panelists, and we're gonna stay out of it as much as possible, but get their insights and kind of what their thought processes are and some of the things we've been seeing in the space lately. And then part of this session will be a live Q&A followup conversation afterwards. So we don't really have a formal set of questions, like most of these kind of more rigid panels do, but I do wanna kind of start off on a little bit of a brighter note. I think some of the times that we've had this event in past years, we spend about 45 minutes talking, the term dumpster fire comes up not infrequently. And then at the end, we try and push a little optimism because there really is a lot to be sort of hopeful about in large part because of the types of things that we're able to do at places like DefCon. But I wanna kind of start off this time upfront by asking each of our panelists, what are some developments lately that have helped make you feel particularly hopeful or optimistic about where we're heading in healthcare security before we get into some of the challenges that still remain? So I'm gonna toss it back to our panel for us to get a little dose of optimism. Yeah, so I think looking at this year in particular, we've seen more and more devices going home with patients. We've seen hospitals go online in a way that they've never planned to in 10 years from now, right? And I think the notion that we can now build security into this as a need to, to let patients have confidence and them operating from their homes is something that seems to be top of mind for folks and actual common practice. So it feels like a confluence of various factors that are coming together at just the right moment. So we have this push of devices leaving the hospital, we have this push of device vendors really getting into the narrative more than ever before. And this collaboration like DEF CON brings together with researchers and devices themselves. And I just, I have a lot of hope that this year has the momentum behind it to really get this to stick and have the forces that are really gonna make this last in the future. Yeah, I could, yeah. So I echo that, but I've also seen this big push I followed back three years where you could not even have a conversation with a manufacturer, right? It was a brick wall. This year we're having conversations, this year we invited to the table, hackers are no longer the criminals that are lurking in the darkness. We've bought things to like, we haven't difficult conversations, but we're having them. And I think that's a major push forward. But except for that, we now having developers and engineers think more security wise, we've started to change the cultures of working in silos to start working as a collective. And one example that I've seen is the way that the community is jumping with the 3D printing, PPE mask making, supporting medical people with their hand. And I mean, even the CTI league that's protecting hospitals around the globe from cyber criminals. I mean, that is just a collective pull of humanity showing that if we work together, we can change the world one bit and back to the time. What are you saying, Jessica? Oh, I like the prompt. I mean, I think I started working on healthcare cybersecurity issues, I think right around 2015, 2016 timeframe. And at that point, obviously I was working on them for the United States Congress. And it was just an entirely different conversation. I mean, the things that we were talking about, like the had said about let's bring in security researchers, let's have them be part of the conversation, they have valuable experience and things to contribute. I was just such a non-starter. And you had all these manufacturers who absolutely were never going to do that. That was completely unacceptable. We are still in some cases getting asked for help by the manufacturers to come to be like, can you make this problem go away? And there were just other policy issues that have come up over the years. I work a lot on legacy device issues, I work a lot on software transparency issues. And they always started out as, this is never going to happen, we're never going to be able to do this, the industry will never accept this. And in 2020, it's completely different. I work on those issues every day, they are starting to be operationalized. Security researchers are a huge part of medical device in healthcare cybersecurity overall. We've just really made so much progress. And we've made the progress in such a way that it has its own momentum now, it's going to carry itself through. And I just think that that really can't be overstated how valuable that is. Yeah, so I agree with what everyone said. I would say that I think one of the biggest things I've noticed over the last five, 10 years is the people. So community and like the number of people sort of coming together, there seems to be lots of different groups and organizations popping up, like even the last couple of years, advocacy groups, people working together, yeah, medical device manufacturers. I mean, even when I, five years ago, started having these conversations and asking questions about security and privacy, it was harder to find other people to even have a discussion with about these things. And now it just seems like there's this momentum that's really building and the number of people are just getting who are engaged and want to make a difference and want to be a part of the conversation is growing. And I see that as a huge positive thing because realistically, like the more people that we have participating in the conversation, the more action that we're going to be able to take sooner and faster. Yeah, I think one of the big things also is we've seen physicians coming to the table and wanting to learn more about the devices they have and work with. Right, it's not just the security people driving it or policy driving. And I think the whole industry's come back into a bang and saying, how can we make this better for our patients? Because in the end, I think every physician wants to make it safer for their patients. And that's a big thing for me that I've seen this year is the physicians coming to the table saying, well, how can we learn more? How do we translate this to our patients in a way that they understand? Yeah, tremendously. Yeah, I think that, I love count friends calls. I was gonna say, I'd echo that because I think even going beyond patients and things, one of the things that my boss is really quite insistent on and you probably all know Suzanne Schwartz one way or another, is the shared responsibility of the healthcare sector where even we are the food and drug administration, we can only do so much. We have jurisdiction over the medical device. We don't have jurisdiction over hospital networks. We don't have jurisdiction over other parts of it sometimes that are certainly implicated in cybersecurity concerns. And so there's traditionally been a very fraught relationship between, for example, medical device manufacturers and healthcare delivery organizations, so hospitals and others. And what I've started to see and have personally experienced over the last couple of years is a greater and greater breakdown of the barriers between those two groups. So with some of the partnership groups that exist, I spend a lot of time working with the healthcare sector coordinating council, for example, you have hospital systems who are on the phone every day with global product security officers at medical device companies. And they're just having conversations. I mean, sometimes they're just shooting the shit and sometimes they're actually talking about work stuff, but like those relationships being there and being already established are just so meaningful. It allows the progress in the sector to be made so much faster and promise to be addressed so much quicker. And so I would echo what V said about, we're having almost new entrance into the conversation that's letting the conversation really take off. I just wanna say. I just wanna say. Sorry. Go ahead, Christian. I just wanna say these are all fantastic insights. And I love to hear the optimism around how the conversations changed because every year when we have this event, we undoubtedly have to talk about an event that's happened during the year, you know, whether or not it's wanna cry or whether or not it's a security researcher that's been shunned or threatened by a device manufacturer. And for, I think I can say this is the first year where I have not been publicly aware of such an event. And so I think that's really proof to what everyone's saying here is that at least part of the interaction with hackers has changed. Now, whether or not that'll stick or whether or not this is just an off year, I think we'll have to see. But when we have strong partners in this space, you know, hackers coming together with health organizations and device manufacturers welcoming their collaboration with the community, and that's really what makes patients safer at the end of the day. I wanted to just also say though, a lot of what was discussed is optimism around, this is a great year. It's also a horrible year. COVID, like 2020 is one I would love to forget. There are so many things about this year that have absolutely sucked. One of the things that's been voiced to me by a variety of people is, you know, how is COVID going to impact this? How are it going to impact hackers, security researchers, security professionals, and the momentum that we have made? Because there's legitimate concern, I feel, that, you know, a lot of security work, a lot of securing these types of spaces costs money, and how a lot of that money is being sucked from what would be security budgets into responding to COVID. And as a consequence, do people here think we're going to see a regression? You know, we've made 10 steps forward, or are we going to go five steps back because COVID really stopped it, stopped the momentum? Love to hear you guys' thoughts. So I'll start, sorry, Vidya. I think it's broken the perimeter, right? It's forced us to pull up our socks, right? The healthcare perimeter is now no longer within your hospital. You have patients at various areas, and everything is different. But I don't necessarily think taking 10 steps back is a bad thing, right? This year is even a time to assess, time to view the healthcare scene, the medical device scene, and if anything forces us to slow down and observe, and just take note of what's happening, right? I see hospitals crumbling up underneath a pandemic, which is something that their function is to care for patients. And we see that even the biggest hospitals are struggling, and that perhaps we just need to take a step back and rebuild. Maybe 2020 could be the year that we build things better. Instead of trying to slap a band-aid on, we do things right and we do things better from the ground up. Yeah, I love that notion of taking the time to rebuild. I mean, we've seen where healthcare facilities have already been compromised before they saw their first patient as part of trying to meet the need for COVID. One of the things I worry about is, in some sense, I think there's this urgency from a clinical perspective to get devices out to patients and kind of solve some of the medical-medical problems. Has that resulted in these devices having security measures that were intentionally not so robustly built in from the beginning just so they could get out and treat these patients? And what do you do now, right? Can you walk that back? Can you say, hey, give all those devices back to us? We don't wanna necessarily see it. So I think your point of needing to build from the beginning is absolutely heard, but I wonder if you're realistically able to call pause on the care that's already out there. I don't know. I mean, I always refer to it as the legacy problem, right? Literally, it's a sea of devices out there. And I mean, if you look at the number of devices out there every year, it just increases exponentially. I mean, the maths blows my head. I sat in it the other day and I went, well, how do we solve this? And I think one of the solutions is to do things going better forward, right? Is not to introduce new devices with the same flaws and the same vulnerabilities and adding to our problem. Cause those devices, we're knowing if you take an ICD, for example, can last in excess of a decade. If that's 600,000 devices in plant of a year, lasting 10 years, it is an ocean that you find to boil off. You can't necessarily say to someone, hey, I need to cut that to your device because it's got a flaw in. My device has got a flaw in that I'm fully aware of. But what are the options? I have to go for 10 years with this device until it's made better. But these devices were not built with a security as a functional requirement, okay? It's a clinical requirement. And I think if we start shifting and start building and designing with security in mind, we can start addressing the problem forward. In terms of the legacy problem, I actually don't have a, I don't know how to boil that ocean to be honest with you. Yeah, I like what everyone is saying. I like the idea of being able to rebuild and taking the step back to think about how we want to solve some of these problems. I think that in practice, it's really hard. Again, with the clinical, looking at the clinical perspective, like right now we're working on trying to create a massive number of respirators and ventilators in a short period of time and make them safe and make them functional. And I, you know, how much security, like if it costs more and it takes more time to add the security, it's hard to make the argument. Like what is, what's more important? You know, if people are dying and it's literally minutes, hours, makes a difference between life or death and you can get one more ventilator into a room, you know, if it took an extra two day, even day to, you know, add security, let's just say for the sake of this problem, is that worth it? Whose lives should we sacrifice to add the security in? Like it's tough. I don't know how we can sort of push for that and where draw the line in practice. It's a balancing act. Sorry, Jason. Oh yeah. No, no, no, I was just gonna say, you know, I think the way that we certainly experience the FDA is we're doing things in parallel. So I have been very blessed in my time at FDA, not to have been fully pulled into the COVID response, but I'm still 100% on cybersecurity. And essentially what that's allowed us to do is while a significant portion of the agency is all in on COVID and then figuring out what they need to do and doing some of the things exactly what you all are saying, getting devices to patients who need them. We also still have a very dedicated team at FDA looking at cybersecurity. And another thing that I was really lucky when I came into FDA, I think everybody, you know, elementary school feels like a bajillion years ago, but you know, like you like walked into elementary school and there's that one person there that was just like, you're like, we're gonna be best friends, we're gonna do great things together. For those of you who have not met Matt Hazelit, he probably is like sick of me talking about him. But anyway, Matt Hazelit is great. And he is the essentially one of the leads at FDA for doing cybersecurity reviews of devices as they come in. And so at the same time that all of this is going on with COVID, what Matt and his team within the device review office at FDA have really started doing, is they're getting tighter and tighter and tighter and tighter and looking at exactly what you all are saying. We're learning so many lessons from deployed devices about vulnerabilities that are showing up. We're learning so much about what manufacturers maybe aren't doing on the front end that are causing problems on the back end. And what Matt and his team have really been able to do is they're taking all of those things, they're taking the vulnerabilities that are showing up in post-market, they're taking the, you know, the missing pieces of the process, the gaps in the process, in the development of medical devices, and they're putting that into the pre-market review process where they're essentially saying, okay, we are learning every day how to get better at reviewing devices and they're implementing it. So I think, you know, while we're certainly seeing issues or there's, you know, there's just this, very valid urgency about getting devices out where they need to be, you know, we also still have this very robust mechanism at FDA in particular for making sure that anything that we're learning, anything that's coming up with regard to cybersecurity is actually making it back into our process. Yeah, which the one thing I would say, I think the financial, which was the original question, right? What's the financial drain gonna have an impact on these devices? I think that's exactly, there's almost a compliment there, right? If we can get the financial decisions to be informed by what the pre-market is telling us in terms of meeting certain security requirements and really having that be key criteria and decisions and not just, you know, offense to the clinicians, the clinician really likes this one brand and that's what they're going with. Having it be part of that core decision, I think absolutely will drive that change. My only decision about whether or not I use a device is how many 5G microchips I can inject into your patient. Wait, doesn't have to say AI in it? No, that's the blockchain version. That's 4G. This is being recorded, man. That is gonna be clipped. That is gonna be taken out of context. It's gonna be viral on Twitter in the next couple of days. And we're gonna hear about how doctors are part of the 5G conspiracy. So strong work on that one. Yeah, well done. What I did wanna ask, right? So I'm gonna turn it, I wanna ask question and Jay for question, seeing as we being, Graldea. You as clinicians, right? When you are doing making a decision for your patient, right? You know, security versus clinical functionality. How do you do that balancing act? Cause you guys are unfortunate as physicians to have your feet in both walls. So from your perspective, how do you balance it out? How do you work it out in your mind? Cause for us, it is security first and foremost and patient care. But we're almost more from the security perspective. I've been a patient, I've seen that side, but I'm quite keen to hear from your side what your perspective on it is. Yeah, that's a great question. And I think that COVID is really starkly forcing us to understand and choose between some of these seemingly opposite goals. In medicine, we frequently have to weigh multiple different pros and cons for any particular treatment or situation. You know, a patient may need anesthesia for a procedure, but they're also very tenuous from a cardiovascular standpoint. So, you know, what's more dangerous, not getting the procedure or putting them under anesthesia. And so oftentimes you have to say, well, we need to take both into account simultaneously. And the points that we've made about the necessity and the urgency of a situation are absolutely correct. When we're in the ICU or in the OR, dealing with these patients, you know, security is not at the forefront of our mind. It's what technology do we need to achieve the physiologic goals we have for that particular patient. When we go home and we talk to people like you and we're able to put it on our hacker and security researcher, obviously we start to be able to understand and conceptualize the consequences of some of the downstream effects, right? So it is a complete balancing act, 100% of the time. Anybody who tells you they have a perfect formula for that answer is lying to you. The thing that's been somewhat reassuring to me is we have had conversations with people that we work with in our hospitals and elsewhere, where they understand that you kind of have to shoot for both. And in some of our institutions, you know, I've had people say from the disaster and emergency management side of things, hey, we're gaming out the COVID response and going on that right now, but we wanna fold a security, a information infrastructure exercise into that as well. Understanding that some of these events, whether it's UCSF in the ransomware issue or some of the hospitals in the Czech Republic who have been hit during this period, the underlying assumption that people are now realizing is that you need a stable infrastructure and good devices to be able to treat patients and have the best outcomes in a crisis situation like this. So deficiencies on the security side only hinder your ability to achieve your main mission. And so it's less of then a shunting of resources from one to the other. And then how can we maximally benefit both of those aspects, which is never perfect, but it's a better way of thinking about it than a trade-off. Yeah, I just completely echo that, like one of my nightmare scenarios is treating a hospital full of COVID patients where the ICUs are overflowing and then getting hit with ransomware, for example. So what little bandwidth we have left to handle the surge to treat the patients as best we can goes out the window when the digital tools we use, whether or not it's the electronic health record or connected medical devices or all of the above, would be impacted by an attack. And so it's not just maybe a little bit of impact to patient care, it's gonna be huge. And one of the things we all like to talk about, I think we all know in this space, is just how dependent doctors are on connected medical technology. Doctors, nurses, technicians, modern healthcare is exceptionally connected, hyper-connected, if you will. And a lot of doctors, we've heard us say this many times, and Jeff and myself included, we haven't worked on paper charts, we've never had light boxes where we pull up a CT scan that's printed on something and put it up against the wall to read it. We've always accessed those through workstations and PAC system software. So you take what we're trained on, this digital infrastructure is connected, vulnerable infrastructure, and you add a pandemic on top of it, and then you take away the tools that we're used to using, undoubtedly patients would suffer. So that one-two punch is definitely something very concerning. To the heart of your question, which is, how do we as clinicians make that trade-off or decision about, here's a more secure device, but it has less clinical utility or functionality compared to this more secure device, et cetera. So how do we make that trade-off? Honestly, looking straight into the camera, it's really hard to even have any bearing on that decision. So a lot of the devices that we use every day in clinical practice, we don't choose. We show up to the hospital and there are monitors that are there because they made that purchasing decision five, 10 years ago. When you train to become a doctor, you may train for four or five years on a particular set of medical devices that you're gonna implant, a particular brand. You become familiar with that. And then so what do you do when you get out of practice? You use the exact same one. And so there are believable, believe it or not, there's much less time to reflect and much less ability than most people believe for us to pick which devices we use in clinical practice and to make that decision. With that being said, we're trying to educate other doctors on this. And when we do, like Jeff mentioned, they care. They want their patients to be safer. They wanna use secure medical devices. They don't want their patient's health information or even their health to be at risk. So we have sympathetic ears from the clinicians. We just lack the ability to make it easy for them. What you don't want is your doctor to have to go to, you know, 14 years to train to be a doctor and then go and have to take another year to a cybersecurity coursework or whatever to become competent. You really have to make it easy for them to make the right decision and convince them that they should tell other people in their hospital and other doctors that it's important. I think to that point, that's maybe a misperception that the expectation is that everyone in the supply chain has to become a cybersecurity expert. I think maybe that's a fatal flaw in how we think about solving this is if we need to sufficiently educate individuals by having them, you're not a medical device practitioner, but hey, I can teach you all about cybersecurity here. Like it's really hard to think that you can sufficiently educate folks that this isn't their core competency on, not to say they can't have some level of understanding what the impact is. And I think making it tangible for their work stream or function or whatever the case may be is the perfect way to do that. But I would love to hear kind of the thought around how we think about leveraging those who are experts and not necessarily trying to solve it ourselves and going at it alone. Because I think that that's probably a fatal flaw and has caused some of the historical challenge that you're talking about when you inherit a hospital full of devices that you didn't pick. And surely as well, like Jeff was saying, normally physicians are used to sort of having to balance different pros and cons of different requirements to make the best medical choice for each patient. But I think historically, maybe they weren't even aware that this is one of the things they need to consider. So even just how can we raise awareness in the consciousness that, oh, that's something that I need to consider even if I'm not an expert, maybe I can have an expert on hand or I can read a rating for a device and make a decision somehow that way. What kind of solutions are there moving forward in that direction? So I've actually spent some time working a lot with the cardiologists and their technicians in South Africa trying to just have the discussion. And I had a phone call from my cardiologist saying, we had a recall of devices. I need to explant a device, but I don't know how to explain it to my patient. I don't even sufficiently understand the engineering talk on the document. Can you translate? Can you translate? So I think the big thing is we are using our language, our linguistics, you know, instead of making it familiar to the medical practitioners. And I think there should be that portion in the hospital that does that translation. Because I can tell you, that was a very difficult conversation having with a 65-year-old lady that doesn't understand technology. I mean, the doctor didn't even sufficiently understand why he had to explant the device was just being recalled. And that's the unfortunate thing, is they are expected to make these decisions. And as he says, he has a patient coming in, his blood pressure is high. He needs to adjust the ICD and the pacemaker. What he doesn't tell him is that his wife cooked a high sodium meal, which affected his blood pressure. And a week later, he comes back because he's having issues, their peers changed significantly. So those are the challenges they deal with, you know, having to solve the clinical puzzles. And I think adding that, you know, the technical stuff on top of it, especially using terminology they don't understand, is where we've been going wrong. We should be finding ways to translate it better for them. Yeah, I think, and also I think that that's a failure overall of how we've designed not only medical devices, but how we do cybersecurity overall, right? Like so many times, if somebody gets hacked or whatever, it's their fault. Setting aside corporations who get, you know, DDoS or whatever, and then claim to have been a victim of a sophisticated cyber attack. Like there are times when people get hit by something and it really wasn't their fault. There were so many different things that they've been expected to know. And I think something a professor once said to me, stuck with me, we were talking about this whole thing about like, oh, like people are just so dumb about cyber, like why can't they just be better? He's like, yeah, and all those people who were driving Ford Pintas, like what the hell were they thinking letting their cars blow up? And, you know, I think that that's kind of the way that was sort of a light bulb moment for me and being like, oh, we have to design devices that don't blow up in people's faces. And, you know, we have to make it as easy as humanly possible to do something securely. Like doing something securely needs to be the easiest thing. And it's doing something securely is the most difficult thing. That's not the fault of the user who then didn't do the secure thing. That is the fault of the designer who made the device poorly. And so I think, you know, we've gotten away with, with frankly in a lot of cases being really lazy and being an offloading the responsibility onto other people. And I think what we have to do, especially within the healthcare sector, is all parts of it, we have to sort of reclaim the responsibility that has been ours all along and actually really do what we need to do. Yeah, I mean, the responsibility, I'm very big, the responsibility to build these things better is the manufacturers, right? They're the ones that have the control over the firmware and the hardware. We shouldn't be pointing the finger at the hospital saying, hey, your device, your network got hacked because you have vulnerable devices. Well, those should have been built better, right? Because now we're leading, we're in the industry now where we're putting band-aid on, right? The vulnerability is disclosed, now we have to run and we're always on the back foot fixing stuff. But the manufacturer should be stepping up and should be doing a better job at this. I completely agree. Because it's taking responsibility. I completely agree, oh, there's a lot that the device manufacturers can do, should do, and I think some of them have been doing. But there is also a key important thing, they can't control it once it's deployed. So if they're deploying a somewhat secure medical device on an unsecured network, or if they're turning off security controls for sake of integrating into their clinical environment, or if they're not patching critical systems around these medical devices, there's some shared responsibility there. And I think that's really important, is that we need to be having not only a strong conversation with the device manufacturers, they do the right thing from the start, but we need to be able to be giving hospitals the appropriate resources education and holding them accountable when there's something egregious. Now, I don't think they should be asked to solve the issues of insecure medical devices, because they just can't. But we've seen time and time again, some pretty horrible security practices by healthcare institutions that have nothing to do with medical device manufacturers, but yet still could have patient safety implications. Things like all the denial of service attacks we see, the ransomware attacks, the theft. I mean, it's not just critical healthcare infrastructure like hospitals. We've been seeing attacks on research. So I'm sure people have seen in the news, and I think Jeff mentioned, ransomware attacks on critical medical research infrastructure, state sponsored attacks on COVID research and vaccine trials data. So we have to have a shared responsibility. This is a whole ecosystem. It's really trying to figure out what's the best bang for our buck. And I feel like a lot of the conversations focus around medical devices, because it's a much easier thing to tackle than trying to go to a rural access hospital in South Dakota and fixing their broken network, and then being able to monitor that and continually say, how are you going to, how are we gonna secure every hospital and clinic in doctor's office in the entire United States? That's a much harder problem. But I... Kariya? Oh, that's it. I'd like to... Sorry. I just wanted to say, right, it's not just the United States problem, right? It's a global problem. You know, hate to be the one to point it, but this is a big problem everywhere. And I know everyone on here is from the US. I'm not only oddball, I'm from Canada. I don't feel so alone. But the thing is just every hospital is different, right? There's no cookie-cutter way of looking at the problem. Every hospital network is different. And I think sometimes what we forget is the patient record systems. Like, I never used to consider it. But then I sat back and thought, from the cyber criminal perspective, it is the ultimate identity theft. It's the thing that you can keep selling constantly. And it's often the thing that's least protected. Because we're focusing so much on medical devices, we've forgotten what the treasure trove that holds. And I mean, that's moving to the cloud. So, you know, Microsoft is one of the big proponents that's moving everything now to the cloud in terms of patient records and clinical systems. So I'd be interested to see how the smaller hospitals are actually going to be able to cope with having to do, you know, permitalist defenses, you know, zero trust networks, or even cloud solutions. Because I think the big hospitals like Mayo might be fine. But you have the smaller rural hospitals that I don't even know if they have IT or security on the premises. Or are they paying someone a third party to do the solution for them? Yeah, I'd like, sorry. I wanted to go back to the topic of responsibility. And I would have to echo with Christian. I think it is a shared responsibility. Obviously, I think that, you know, I work for Medical Device Manufacturer. I'm a software, a netted software engineer. I think about these things. I want to make sure that I'm doing my part to build really secure and awesome, safe products for end users. I care a lot about that. But I think it's easy to blame manufacturers. And it's often more complicated. Again, it's this balancing act. So I work for a Medical Device Manufacturer that's a consulting firm. So we don't have any of our own in-house products. We have people who come to us and we have clients. And, you know, some of them, you know, business people and some of them are other medical device manufacturers that subcontract to us. But we have a lot of people who are just doctors. And clinicians and other people who are really passionate about saving lives. And they come to us and they're like, we have this idea. We think we see this gap. We desperately want to help our patients. Please help us build this. We have this tiny, tiny budget. We only have this amount of time to do it in because, you know, we got this little budget from these people. But they say we have to get it done by this time. And, you know, we were working really, really hard to try to meet all these deadlines and actually make this little device that, you know, if we can give it to those doctors, it might save lives. And then for us to come in and try to say, like, oh, well, we have to add all of these security things. And, you know, maybe it's going to affect the budget. It's going to affect the timeline. And, you know, again, I care about these things. I'm in there. I'm advocating for this stuff. But it's hard to say, you know, if you only have this much budget and it's how do you choose? How do you tell them, OK, you shouldn't. We just aren't going to be able to make this and implement this part of security, like, where to draw the line. I think it's more complicated and harder, like, in practice. And it's hard to just say, well, the medical device manufacturer should just do it. You know, how do you make that choice? I actually want to add to that story. So I started working with Meitronik. And I'm a hard ass. Everyone knows that I say a bunch of things three years ago. Cool came out because I was unhappy with how our conversation was because we had a legal situation. And I started the first six months of my project listening and getting to know the developers and the software for engineers and the hardware engineers. And I sat back and listened and realized that these are real people that are wanting to make a real difference. Really, they are. And one of the specific engineers, Rob Mastow, had a big impact on my life because he said to me, so tell me what you want in your device. Tell me all the security you want. Get the list to me and let's discuss this. And he turned around and said to me, well, so you're getting a new device every three years, right? And I said to him, well, no, there's no fucking way that I'm doing that to myself. And he's like, well, then you can't have all of that. He says there's trade-offs for everything. So specifically with embedded, implanted devices, the more security you add, the less lifetime you'll have on your battery. Because I can tell you the one thing that that is worth gold is how long your device lasts. Because it is a horrible thing to cut out. And I don't know if you guys know, but the more you cut it out, the more scar tissue forms and the worse it gets. And I've had my device since 19. I wouldn't be here without it. So people building these things, I would be dead. I had two weeks left to live. That was the serious situation. So I think sometimes we want all the security in the world, but we don't realize these trade-offs. These devices are there to save lives. And if we try and push security too much, there's a whole lot of state of devices that will never come to market and they will never save lives. And that's one big lesson that I had to learn and I had to have humble pie. And I had to swallow my words and realize that, is people behind this wanting to make a difference with technology and science? And without that device, I wouldn't be here. I wouldn't have my kids. So technology does save lives in ways. We must just find the perfect balance to balance everything out. In your perspective, V, and folks like you, like Marie-Mau, who have these incredible security backgrounds, but also our patients, is so incredibly valuable to be able to consider these types of trade-offs in a human person in a very real way. And one of the things that Christian and I are working on is how do we understand the role of the patient as a stakeholder in these types of conversations? And is there a utility in having those types of conversations with this six-year-old man who needs to get his pacemaker revised? Or is there a trade-off between the potential risk of a vulnerability being exploited and then just putting him through that process of having it revised? And I think it highlights one of the things that Christian and I are so interested in is just developing a data and a science around this to be able to say, in other types of medical therapies, we have very clear risks. If I give you a blood transfusion, there's about a 1 in 1.5 to 2 million chance that you might have an infection that's serious as a result of that or heart or lung issues. We really don't have the ability to have conversations with patients and say, this is the clear-cut risk, because thankfully, we haven't seen many incidents of issues. And that doesn't mean that there's an absence of risk, but it makes it harder for us to have discussions with patients and say, really, this is the type of math that you have to weigh as a patient with your own values and preferences and things like that. And we've kind of done some work on touching on this idea of a cybersecurity-informed consent to say, we talk about risks, medical risks of surgeries and medications and things like that that they may be on. Should we start thinking about cybersecurity as a potential risk that should be included in this conversation a clinician has with the patient? In Jessica, I know the FDA is also really interested in this, and you've had a lot of really cool outreach types of events to be able to talk to people like V. And I think it's a challenge of moving beyond people who are security literate to people who may not have heard about these issues whatsoever for as much as we talk about them and to be able to kind of have that enter the conversation so that they can, without having a background in cybersecurity, weigh that as part of their clinical process. Yeah. I mean, I think that there's a couple things that I would bring up on this. I mean, I think, one, you had mentioned that FDA has been doing a lot of this. And I think V were actually at the October 2019 meeting, but we had the Patient Engagement Advisory Council meeting, the peak meeting. And so FDA is very much trying to make sure that the patient perspective is heard. And it's not just yearly meetings and things like that. As many of you know, Suzanne Schwartz is very approachable. You can pretty much get her email and email her, and she'll find time to talk to you. So that is a really critical part of this, because not that all of you are going around reading government white papers and things, but a couple of years ago, the FDA put out this medical device safety action plan. And so they're setting aside the flowery language that all government documents, I think, were like decreed that they must use. Essentially what it says is, look, it's exactly this issue of balancing the idea that patient safety is a twofold concern. There's the concern that we all typically think about on cybersecurity. Is this device safe and effective enough to justify its use with the patient? There's also the flip side of that, though, of what Asher was talking about and what V was talking about in that, what happens that the patient can't get the device? That is another, that's got to be a thing that gets considered too. And so, you know, FDA, when we talk about cybersecurity, when we usually do it, we're very explicitly talking about, do you have security controls? Do you have sufficient whatever to make sure that the device is cybersecurity? But a less talked about part of the work, but that is equally important, is this idea that we recognize that sometimes devices need to get to patients because of the benefit that they're going to be able to provide. And maybe we don't have as much information. Maybe we don't have as much assurance of the things that we would like to. But, you know, this is the breakthrough device program. This is some of these other investigational devices that come up. And that FDA has specific categories for and authority for them, things that you can do this. And then it's exactly this recognition that the other folks on the panel are pointing out, which is that in some cases, not having access to a device can be as big of a risk as having access to a device that isn't secure enough. And so I think that is just an important point to highlight. And that we as an agency keep in mind, but that this community, in addition, also keeps in mind as we have these kinds of conversations. All right, I want to change this. Oh, can I? I'm going to change the subject real quick, because I heard a rumor there are thousands of hackers watching this stream right now. There are literally thousands of hackers from across the globe watching. You know, very interested in what we're talking about. And I'm sure they're thinking to themselves, wow, there are a lot of issues here I might not have been aware of anymore and they want to get involved. So hackers out there wanting to get involved in this space. How does, we've heard some examples. You know, there's been a group of hackers that have tried to defend hospitals against attackers during COVID. There are hackers printing protective equipment for doctors on the front line, et cetera, nurses. But what's the next generation of this? Like when the pandemic's done, how do hackers play a role in this space? Continuing research, is it bug bounties? Should a hospital have a bug bounty? Should a device manufacturer have a bug bounty? How does HIPAA play into that? The protection of patient information. How do we really kind of bring hackers more into this space? OK, I'm going to jump in, yeah. I have a dream and I have a dream with a friend of mine, Nina. We want to establish a laboratory setup with devices, right, that people can actually do research on, that we can put on networks, that we can scan for vulnerabilities, that we can reverse engineer, that we can work with manufacturers to make better. I already see people wanting to break things, right? We want to find flaws. We want to find problems. But often, we don't want to find the solution to the problem, because that's the harder part. So I want to say that if you find a device that's vulnerable, first remember, they are human lives at stake. This is not an ego thing. Try and find the solution to the problem that you found. Write it up as you would any scientific paper. I know it's hard work. I know it's not glorious and it's not sexy. But when you hand that over to the manufacturer, they have all the data. And the more data they have, the faster that they can verify the claim, they can start acting accordingly. It's not going to be easy. Not all MDMs are going to want to be friendly, but it is changing. But the biggest thing is the more thorough your research is, the less dispute they can be, and the more we can find a way to fixing it. But it's not all about just bug bounties, because I think that for me creates a negative thing of we just finding the problems. We should be finding problems and then solving them. That's the hacker mindset. It's not just about pointing fingers and saying, this is a problem. We should be working at the solution. We should be finding new ways of doing things. So I want to see a lab full of medical devices that hackers can pull apart and we can build solutions. But not only hackers, you need people like Ash, right? Because I mean, we're good at the other side of it, but I can't build these devices. I wouldn't even know where to stop. So I think it needs to be a multidisciplinary approach. But even people like the FDA should be involved in it from a policy perspective. And everyone should just be working together. We should be building the bridges towards having synergy and having a space where we can explore this. And I think Biohack is awesome for that, because they would really do a lot of that. Sorry, everyone. Excuse my cat, who just really, just really jumped into my lap. I was trying to prevent it and failed. He just failed the side of the chair. Welcome addition to the panel. So I love the notion of being part of the solution. And I think to your point, it does require multiple voices. You hit on a point that I think is really important, right? You as someone who does a lot of security research is not totally in the space of how do I build a medical device. I think we similarly need to not have the expectation of the people building medical devices to know how to solve some of these cybersecurity issues. So I think having the willingness to partner and really helping them not make cybersecurity their number to core capability and instead offering them solutions is absolutely critical and the way to make things actually stick. So should we have solution bounties then? Oh, yeah. Oh, you should trademark that. Yes. But I just wanted to bring up just a scenario. So it's not just for medical devices. You have a medical device bug bounty, for example. If that's a good or a bad thing, we could talk hours on that. But you don't have the issue of patient health information. And so hospitals can't offer hackers to come in and poke around the periphery of their networks or to give them that type of experience. Because of a couple of things. One, their live networks are taking care of patients who would hate for something bad to happen. But then two, there's so much protected health information floating around a hospital's network. And if a security researcher or a hacker in good faith trying to do right finds something and it happens to be commingled with patient health information, then the hospital is required to report that as a hip breach. So there are some perhaps some would call barriers, perhaps some would call safeguards. But what I think it's done is it's made not just the delivery of health care itself, particularly difficult to engage with from the hacker perspective. And I think maybe we don't have to figure out that solution. Maybe that's one of the solution bounties we should offer. And how can hackers get more involved, give us your ideas. But particularly at hospitals where we've talked about they don't have a lot of in-house security expertise, if anything, but yet they're still asked to do all of that work. How can the community come together and help them out? Because that could be a lot of good right there. One very easy answer to that is just working in health care. So I would encourage anybody who's thinking about a career change, if you're looking for an area where you can make an incredible impact and have a lot of really challenging problems to solve, consider working for a hospital or a health care delivery organization or being adjacent to that in some way. We know that some of these hospitals we talked about in certain areas are critically underserved and may not even have a full-time IT security professional. So one way to get involved is just to join that effort as an actual employee of a health care organization. So I've got some specific suggestions, I think. And it's funny that you bring up HIPAA and other things like that, because I think one of the best ways that folks like these could get involved, whatever, there are actually specific programs that are being offered now within the federal government that puts you in policymakers' offices and lets you be the cybersecurity expert in the room for those people. So there's the Congress Fellowship. Some of you may have heard of it. And you may have met some of the fellows, Chris Fagoyan, who's probably a well-known name for some of you all, was a tech congress fellow. He's now works permanently for a senator and is like that senator's tech person. But there's been dozens of others. They've got great folks. It's a great program, places you with either a house office or a senate office. And so that's one of the most direct ways that you can actually influence policymaking, in a sense, is actually go be in the room when the policy is being made. So I would highly recommend that. I don't know when the application process opens for next year's class, but Travis Moore is your man, so go find Travis. The other thing I would mention is I am the cavalry. When I am the cavalry was first started, it was kind of this weird kooky thing for the Hill, for the United States Congress, who's very used to dealing with very buttoned up everyone's suit and tie professional trade associations and things like that when they were trying to get information on whatever they needed to get information on. And I am the cavalry, just kind of came up to the Hill and was just like, hi, we're a bunch of cybersecurity experts. We'd like to talk to you about this. But the thing is, there were the most effective advocacy groups in national policymaking right now. Because they were just so, everyone could tell in a city where sometimes people's motivations are a little bit shrouded, that they just wanted to be there. They were there with good faith, and they really just wanted to do the right thing. And they were going to answer whatever questions needed answering without bias. And so I know SEA works a ton with I am the cavalry. I know the Hill works a ton with I am the cavalry. And so if you want to get involved and have an opportunity to meet with the movers and shakers of the country of cybersecurity policy, I am the cavalry is another really great place to do that. I just would like to perhaps add that from another perspective, if you want to get a hold of working in a hospital or getting your hands just on some medical equipment or just having a conversation with the manufacturers. Three years ago, I spoke at Biohack Village. This was my first day of gone. This was my first ever meeting. Another group of hackers like myself. And that's how I was set on the path to do research. That was how I got access to devices. I no longer had to smuggle them through customs at an airport because I bought them on eBay or because you can't buy them in South Africa. So I would buy them in the UK. I'd go visit a friend. And I'd put them in my luggage and smell them back home so that I could work on them. Or even when they exploded my first devices, they'd give me, give me, give me. I want to have that. And my doctor's why I'm like saying to me for reasons, of course. But that's how I got my hands on. But here I walked into a place in Vegas that had devices, that had a lab, that had everything that makes a young hacker's heart do jumps. So if that tickles your fancy, then that's somewhere that you should go visit. You should go try your hand at it. You should go play with those devices, speak to the manufacturers, and go see how they implement it. And I mean, even if you're a patient, nothing precludes you from giving advice at a hospital. I mean, I've given advice from my ICU bed. There's a reason I'm not allowed a laptop in anymore. Except for that the doctor wants me to ask more. Apparently, that's the reason, but give advice. Reach out. Get involved. And if someone doesn't want to listen to you the first time, keep on trying, because it's a patient's game. It's not going to happen overnight. I've learned. I've dealt with many parties, and it's not going to happen fast because it's a big industry. Just get involved. Get your hands dirty, because that's how we change the world. Every new researcher that comes in the fold is a new mind that thinks differently. And that's, I think, how we change things one step at a time. I love that, because I think the idiosyncratic notion that it takes a specific individual or an elite person or someone who's an expert to make a change is absolutely a misconception. So the notion of just getting involved, there's no inner circle here. I think every person in this community is more than willing to share their networks, share their expertise, and share whatever corner of this problem that they have. It's a really hard problem, and we need all the talent that we can muster up right now to really make a change. And I think if we look at the diversity of roles, functions, experience, thought process, background, education, everything, any underrepresentation of that population is indicative of missing a potential attack factor. And we really have to really have to account for that and try to bring it as many folks as we can. Yeah, I think this is a call to arms, right? This is how many hackers can we round up in one go? Well, not just even hackers, right? You know, everyone, from physicians, from patients, from borders and policymakers. I think if we start breaking down the silos, which we've been functioning in, and independently working, and start this collective movement, that's how we change the world. It's not a one person thing. It's never going to be one person that's going to come up with the magic solution or the silver bullet. It takes more than one village, as I always say, to solve the problem. And I think it's time in 2020 that we as a society build together instead of apart, because we've seen what a virus can do. It's locked us in our houses. So I think it's time we take our power back, and we start moving the world into a positive future. Because I can tell you from a patient perspective, not having access to being able to go into a cardiologist office because we're afraid of COVID, because currently it's more dangerous going into a hospital. A denial of patient care is a non-acceptable, non-negotiable, not going to stand for it. And I think we should be kicking down the doors and saying, now's the time, not yesterday, now. I love it. Are there any better sentiments to go out on than that? I think it's easy to be cynical about all this, and it's easy to say we're having the same conversations year after year, and the problems are still there. But I mean, in 2015, the biohacking village didn't exist, and the idea of people from device manufacturers bringing stuff for hackers to poke and prod at would be laughed out of the room. So for people who don't believe that we can do awesome things when we work together and hang out at places like DefCon, I mean, we have evidence to the contrary. So I think that's about an hour, and I'm looking forward to seeing you guys in a little bit for a Q&A. Take care, everyone. Thank you, DefCon, for having us. A shout out to Nikita, especially, for taking another shot at us. We're going to go ahead and open it up to Q&A here in five, four, three, two, one.
 Welcome, DEF CON 28, the Do No Harm panel.  This is a healthcare security conversation.  And for the next 45 minutes, followed by a 45 minute Q&A,  you're gonna hear from some of the world's top experts  on the healthcare cybersecurity space.  Welcome.  Thanks, Quadi.  I'm Replicant.  We're going to get out of the way here in just a moment  and introduce you to our panelists,  but obviously a little bit different of a venue for us  than the last couple of years.  We hope that everyone is staying safe and healthy,  and we're thinking of you and look forward  to when we can hang out together in person.  Without any further ado,  I'm gonna let our panelists take it away and say hi.  Hi, everyone.  My name is Ash Left.  I am a biochemist, a computer scientist,  and an electrical engineer.  I work at a medical device firm called Starfish in Canada,  and I'm a suffer engineer there and an advocate  for security and privacy.  I think that means that I'm next.  So I don't have a handle, unfortunately.  So I am just Jessica Wilkerson.  I'm a cyber policy analyst  with the Food and Drug Administration,  and I work on cybersecurity policy there for medical devices.  Some of you may know me from my previous time  with the United States Congress Energy and Commerce Committee  where I worked on cybersecurity policy for them.  Very nice to talk with everyone  and looking forward to the panel.  Thank you.  Hi, this is me.  Everyone should know me.  The short, the sweet, the sassy, the spunky.  I am leaving my company for the wildfield in Norway  where I'll be having a whole host of millions  to help me do medical device research.  Currently doing independent research in formatronic,  adding them to get more visualization on their products  and their security.  I'm an advocate for patient rights  as well as keeping things safe,  making it, keeping it simple,  keeping it stupid, as I'm saying.  And that's me, short and sweet.  That leaves me.  My name is Vidya Morthy.  I work for MedCrypt,  a startup in the space of bringing cybersecurity features  to medical devices.  I absolutely am passionate about the subject matter  and think that we're at an inflection point here  and there's gonna be a big amount of change  as we go forward.  Well, on behalf of Quadim, myself,  we are just so incredibly grateful  to have an amazing panel this year.  And for those of us who are joining us for the first time,  and this is kind of one of the silver linings,  I think, about this format is that in past years,  we've had folks who wanted to kind of get in  on this conversation that weren't able to  due to space constraints.  So we're really thankful for the opportunity  to have anybody who wants to come in, listen in,  join with us to be able to do so.  So for those who may not be familiar,  what we're gonna do is we're gonna have  a little conversation here between the four  of these awesome panelists,  and we're gonna stay out of it as much as possible,  but get their insights and kind of what their thought  processes are and some of the things we've been seeing  in the space lately.  And then part of this session will be a live Q&A followup  conversation afterwards.  So we don't really have a formal set of questions,  like most of these kind of more rigid panels do,  but I do wanna kind of start off  on a little bit of a brighter note.  I think some of the times that we've had this event  in past years, we spend about 45 minutes talking,  the term dumpster fire comes up not infrequently.  And then at the end, we try and push a little optimism  because there really is a lot to be sort of hopeful  about in large part because of the types of things  that we're able to do at places like DefCon.  But I wanna kind of start off this time upfront  by asking each of our panelists,  what are some developments lately  that have helped make you feel particularly hopeful  or optimistic about where we're heading  in healthcare security before we get into some  of the challenges that still remain?  So I'm gonna toss it back to our panel for us  to get a little dose of optimism.  Yeah, so I think looking at this year in particular,  we've seen more and more devices going home with patients.  We've seen hospitals go online in a way  that they've never planned to in 10 years from now, right?  And I think the notion that we can now build security  into this as a need to, to let patients have confidence  and them operating from their homes is something  that seems to be top of mind for folks  and actual common practice.  So it feels like a confluence of various factors  that are coming together at just the right moment.  So we have this push of devices leaving the hospital,  we have this push of device vendors really getting  into the narrative more than ever before.  And this collaboration like DEF CON brings together  with researchers and devices themselves.  And I just, I have a lot of hope that this year  has the momentum behind it to really get this to stick  and have the forces that are really gonna make this last  in the future.  Yeah, I could, yeah.  So I echo that, but I've also seen this big push  I followed back three years  where you could not even have a conversation  with a manufacturer, right?  It was a brick wall.  This year we're having conversations,  this year we invited to the table,  hackers are no longer the criminals  that are lurking in the darkness.  We've bought things to like,  we haven't difficult conversations,  but we're having them.  And I think that's a major push forward.  But except for that, we now having developers  and engineers think more security wise,  we've started to change the cultures of working in silos  to start working as a collective.  And one example that I've seen is the way  that the community is jumping with the 3D printing,  PPE mask making, supporting medical people  with their hand.  And I mean, even the CTI league  that's protecting hospitals around the globe  from cyber criminals.  I mean, that is just a collective pull of humanity  showing that if we work together,  we can change the world one bit and back to the time.  What are you saying, Jessica?  Oh, I like the prompt.  I mean, I think I started working on  healthcare cybersecurity issues,  I think right around 2015, 2016 timeframe.  And at that point, obviously I was working on them  for the United States Congress.  And it was just an entirely different conversation.  I mean, the things that we were talking about,  like the had said about let's bring in security researchers,  let's have them be part of the conversation,  they have valuable experience and things to contribute.  I was just such a non-starter.  And you had all these manufacturers  who absolutely were never going to do that.  That was completely unacceptable.  We are still in some cases getting asked for help  by the manufacturers to come to be like,  can you make this problem go away?  And there were just other policy issues  that have come up over the years.  I work a lot on legacy device issues,  I work a lot on software transparency issues.  And they always started out as,  this is never going to happen,  we're never going to be able to do this,  the industry will never accept this.  And in 2020, it's completely different.  I work on those issues every day,  they are starting to be operationalized.  Security researchers are a huge part of medical device  in healthcare cybersecurity overall.  We've just really made so much progress.  And we've made the progress in such a way  that it has its own momentum now,  it's going to carry itself through.  And I just think that that really can't be overstated  how valuable that is.  Yeah, so I agree with what everyone said.  I would say that I think one of the biggest things  I've noticed over the last five, 10 years is the people.  So community and like the number of people  sort of coming together,  there seems to be lots of different groups  and organizations popping up,  like even the last couple of years,  advocacy groups, people working together,  yeah, medical device manufacturers.  I mean, even when I, five years ago,  started having these conversations  and asking questions about security and privacy,  it was harder to find other people  to even have a discussion with about these things.  And now it just seems like there's this momentum  that's really building and the number of people  are just getting who are engaged and want to make a difference  and want to be a part of the conversation is growing.  And I see that as a huge positive thing  because realistically, like the more people that we have  participating in the conversation,  the more action that we're going to be able to take sooner  and faster.  Yeah, I think one of the big things also  is we've seen physicians coming to the table  and wanting to learn more about the devices  they have and work with.  Right, it's not just the security people  driving it or policy driving.  And I think the whole industry's come back into a bang  and saying, how can we make this better for our patients?  Because in the end, I think every physician  wants to make it safer for their patients.  And that's a big thing for me that I've seen this year  is the physicians coming to the table saying,  well, how can we learn more?  How do we translate this to our patients  in a way that they understand?  Yeah, tremendously.  Yeah, I think that, I love count friends calls.  I was gonna say, I'd echo that because I think  even going beyond patients and things,  one of the things that my boss is really quite insistent on  and you probably all know Suzanne Schwartz  one way or another, is the shared responsibility  of the healthcare sector where even we are the food  and drug administration, we can only do so much.  We have jurisdiction over the medical device.  We don't have jurisdiction over hospital networks.  We don't have jurisdiction over other parts of it  sometimes that are certainly implicated  in cybersecurity concerns.  And so there's traditionally been a very  fraught relationship between, for example,  medical device manufacturers  and healthcare delivery organizations,  so hospitals and others.  And what I've started to see and have personally experienced  over the last couple of years is a greater and greater  breakdown of the barriers between those two groups.  So with some of the partnership groups that exist,  I spend a lot of time working with the healthcare sector  coordinating council, for example,  you have hospital systems who are on the phone every day  with global product security officers  at medical device companies.  And they're just having conversations.  I mean, sometimes they're just shooting the shit  and sometimes they're actually talking about work stuff,  but like those relationships being there  and being already established are just so meaningful.  It allows the progress in the sector to be made  so much faster and promise to be addressed so much quicker.  And so I would echo what V said about,  we're having almost new entrance into the conversation  that's letting the conversation really take off.  I just wanna say.  I just wanna say.  Sorry.  Go ahead, Christian.  I just wanna say these are all fantastic insights.  And I love to hear the optimism around  how the conversations changed because every year  when we have this event,  we undoubtedly have to talk about  an event that's happened during the year,  you know, whether or not it's wanna cry  or whether or not it's a security researcher  that's been shunned or threatened by a device manufacturer.  And for, I think I can say this is the first year  where I have not been publicly aware of such an event.  And so I think that's really proof  to what everyone's saying here is that at least part  of the interaction with hackers has changed.  Now, whether or not that'll stick  or whether or not this is just an off year,  I think we'll have to see.  But when we have strong partners in this space,  you know, hackers coming together with health organizations  and device manufacturers welcoming their collaboration  with the community,  and that's really what makes patients safer  at the end of the day.  I wanted to just also say though,  a lot of what was discussed is optimism around,  this is a great year.  It's also a horrible year.  COVID, like 2020 is one I would love to forget.  There are so many things about this year  that have absolutely sucked.  One of the things that's been voiced to me  by a variety of people is, you know,  how is COVID going to impact this?  How are it going to impact hackers,  security researchers, security professionals,  and the momentum that we have made?  Because there's legitimate concern, I feel,  that, you know, a lot of security work,  a lot of securing these types of spaces costs money,  and how a lot of that money is being sucked  from what would be security budgets  into responding to COVID.  And as a consequence, do people here think  we're going to see a regression?  You know, we've made 10 steps forward,  or are we going to go five steps back  because COVID really stopped it, stopped the momentum?  Love to hear you guys' thoughts.  So I'll start, sorry, Vidya.  I think it's broken the perimeter, right?  It's forced us to pull up our socks, right?  The healthcare perimeter is now no longer within your hospital.  You have patients at various areas,  and everything is different.  But I don't necessarily think  taking 10 steps back is a bad thing, right?  This year is even a time to assess,  time to view the healthcare scene,  the medical device scene,  and if anything forces us to slow down and observe,  and just take note of what's happening, right?  I see hospitals crumbling up underneath a pandemic,  which is something that their function is to care for patients.  And we see that even the biggest hospitals are struggling,  and that perhaps we just need to take a step back  and rebuild.  Maybe 2020 could be the year that we build things better.  Instead of trying to slap a band-aid on,  we do things right and we do things better  from the ground up.  Yeah, I love that notion of taking the time to rebuild.  I mean, we've seen where healthcare facilities  have already been compromised  before they saw their first patient  as part of trying to meet the need for COVID.  One of the things I worry about is, in some sense,  I think there's this urgency from a clinical perspective  to get devices out to patients  and kind of solve some of the medical-medical problems.  Has that resulted in these devices  having security measures that were intentionally  not so robustly built in from the beginning  just so they could get out and treat these patients?  And what do you do now, right?  Can you walk that back?  Can you say, hey, give all those devices back to us?  We don't wanna necessarily see it.  So I think your point of needing to build  from the beginning is absolutely heard,  but I wonder if you're realistically able  to call pause on the care that's already out there.  I don't know.  I mean, I always refer to it as the legacy problem, right?  Literally, it's a sea of devices out there.  And I mean, if you look at the number of devices  out there every year, it just increases exponentially.  I mean, the maths blows my head.  I sat in it the other day and I went,  well, how do we solve this?  And I think one of the solutions is to do things  going better forward, right?  Is not to introduce new devices with the same flaws  and the same vulnerabilities and adding to our problem.  Cause those devices, we're knowing if you take an ICD,  for example, can last in excess of a decade.  If that's 600,000 devices in plant of a year,  lasting 10 years, it is an ocean that you find to boil off.  You can't necessarily say to someone,  hey, I need to cut that to your device  because it's got a flaw in.  My device has got a flaw in that I'm fully aware of.  But what are the options?  I have to go for 10 years with this device  until it's made better.  But these devices were not built with a security  as a functional requirement, okay?  It's a clinical requirement.  And I think if we start shifting and start building  and designing with security in mind,  we can start addressing the problem forward.  In terms of the legacy problem,  I actually don't have a,  I don't know how to boil that ocean to be honest with you.  Yeah, I like what everyone is saying.  I like the idea of being able to rebuild  and taking the step back to think about  how we want to solve some of these problems.  I think that in practice, it's really hard.  Again, with the clinical, looking at the clinical perspective,  like right now we're working on  trying to create a massive number of respirators  and ventilators in a short period of time  and make them safe and make them functional.  And I, you know, how much security,  like if it costs more and it takes more time  to add the security, it's hard to make the argument.  Like what is, what's more important?  You know, if people are dying  and it's literally minutes, hours,  makes a difference between life or death  and you can get one more ventilator into a room,  you know, if it took an extra two day,  even day to, you know, add security,  let's just say for the sake of this problem,  is that worth it?  Whose lives should we sacrifice to add the security in?  Like it's tough.  I don't know how we can sort of push for that  and where draw the line in practice.  It's a balancing act.  Sorry, Jason.  Oh yeah.  No, no, no, I was just gonna say, you know,  I think the way that we certainly experience the FDA  is we're doing things in parallel.  So I have been very blessed in my time at FDA,  not to have been fully pulled into the COVID response,  but I'm still 100% on cybersecurity.  And essentially what that's allowed us to do  is while a significant portion of the agency  is all in on COVID and then figuring out what they need to do  and doing some of the things exactly what you all are saying,  getting devices to patients who need them.  We also still have a very dedicated team at FDA  looking at cybersecurity.  And another thing that I was really lucky  when I came into FDA, I think everybody, you know,  elementary school feels like a bajillion years ago,  but you know, like you like walked into elementary school  and there's that one person there  that was just like, you're like,  we're gonna be best friends,  we're gonna do great things together.  For those of you who have not met Matt Hazelit,  he probably is like sick of me talking about him.  But anyway, Matt Hazelit is great.  And he is the essentially one of the leads at FDA  for doing cybersecurity reviews of devices as they come in.  And so at the same time that all of this is going on  with COVID, what Matt and his team  within the device review office at FDA  have really started doing,  is they're getting tighter and tighter and tighter and tighter  and looking at exactly what you all are saying.  We're learning so many lessons from deployed devices  about vulnerabilities that are showing up.  We're learning so much about what manufacturers  maybe aren't doing on the front end  that are causing problems on the back end.  And what Matt and his team have really been able to do  is they're taking all of those things,  they're taking the vulnerabilities  that are showing up in post-market,  they're taking the, you know,  the missing pieces of the process,  the gaps in the process,  in the development of medical devices,  and they're putting that into the pre-market review process  where they're essentially saying,  okay, we are learning every day  how to get better at reviewing devices  and they're implementing it.  So I think, you know, while we're certainly seeing  issues or there's, you know, there's just this,  very valid urgency about getting devices out  where they need to be, you know,  we also still have this very robust mechanism at FDA  in particular for making sure that anything  that we're learning, anything that's coming up  with regard to cybersecurity  is actually making it back into our process.  Yeah, which the one thing I would say,  I think the financial, which was the original question, right?  What's the financial drain  gonna have an impact on these devices?  I think that's exactly,  there's almost a compliment there, right?  If we can get the financial decisions to be informed  by what the pre-market is telling us  in terms of meeting certain security requirements  and really having that be key criteria and decisions  and not just, you know, offense to the clinicians,  the clinician really likes this one brand  and that's what they're going with.  Having it be part of that core decision,  I think absolutely will drive that change.  My only decision about whether or not I use a device  is how many 5G microchips I can inject into your patient.  Wait, doesn't have to say AI in it?  No, that's the blockchain version.  That's 4G.  This is being recorded, man.  That is gonna be clipped.  That is gonna be taken out of context.  It's gonna be viral on Twitter in the next couple of days.  And we're gonna hear about how doctors  are part of the 5G conspiracy.  So strong work on that one.  Yeah, well done.  What I did wanna ask, right?  So I'm gonna turn it, I wanna ask question  and Jay for question, seeing as we being, Graldea.  You as clinicians, right?  When you are doing making a decision for your patient, right?  You know, security versus clinical functionality.  How do you do that balancing act?  Cause you guys are unfortunate as physicians  to have your feet in both walls.  So from your perspective, how do you balance it out?  How do you work it out in your mind?  Cause for us, it is security first and foremost  and patient care.  But we're almost more from the security perspective.  I've been a patient, I've seen that side,  but I'm quite keen to hear from your side  what your perspective on it is.  Yeah, that's a great question.  And I think that COVID is really starkly forcing us  to understand and choose between some of these  seemingly opposite goals.  In medicine, we frequently have to weigh  multiple different pros and cons  for any particular treatment or situation.  You know, a patient may need anesthesia for a procedure,  but they're also very tenuous  from a cardiovascular standpoint.  So, you know, what's more dangerous,  not getting the procedure or putting them under anesthesia.  And so oftentimes you have to say,  well, we need to take both into account simultaneously.  And the points that we've made about the necessity  and the urgency of a situation are absolutely correct.  When we're in the ICU or in the OR,  dealing with these patients, you know,  security is not at the forefront of our mind.  It's what technology do we need to achieve  the physiologic goals we have for that particular patient.  When we go home and we talk to people like you  and we're able to put it on our hacker  and security researcher,  obviously we start to be able to understand  and conceptualize the consequences  of some of the downstream effects, right?  So it is a complete balancing act, 100% of the time.  Anybody who tells you they have a perfect formula  for that answer is lying to you.  The thing that's been somewhat reassuring to me  is we have had conversations with people  that we work with in our hospitals and elsewhere,  where they understand that you kind of have to shoot for both.  And in some of our institutions, you know,  I've had people say from the disaster  and emergency management side of things,  hey, we're gaming out the COVID response  and going on that right now,  but we wanna fold a security,  a information infrastructure exercise into that as well.  Understanding that some of these events,  whether it's UCSF in the ransomware issue  or some of the hospitals in the Czech Republic  who have been hit during this period,  the underlying assumption that people are now realizing  is that you need a stable infrastructure  and good devices to be able to treat patients  and have the best outcomes in a crisis situation like this.  So deficiencies on the security side  only hinder your ability to achieve your main mission.  And so it's less of then a shunting of resources  from one to the other.  And then how can we maximally benefit  both of those aspects, which is never perfect,  but it's a better way of thinking about it  than a trade-off.  Yeah, I just completely echo that,  like one of my nightmare scenarios  is treating a hospital full of COVID patients  where the ICUs are overflowing  and then getting hit with ransomware, for example.  So what little bandwidth we have left  to handle the surge to treat the patients  as best we can goes out the window  when the digital tools we use,  whether or not it's the electronic health record  or connected medical devices or all of the above,  would be impacted by an attack.  And so it's not just maybe a little bit  of impact to patient care, it's gonna be huge.  And one of the things we all like to talk about,  I think we all know in this space,  is just how dependent doctors are  on connected medical technology.  Doctors, nurses, technicians, modern healthcare  is exceptionally connected, hyper-connected, if you will.  And a lot of doctors, we've heard us say this many times,  and Jeff and myself included, we haven't worked  on paper charts, we've never had light boxes  where we pull up a CT scan that's printed on something  and put it up against the wall to read it.  We've always accessed those through workstations  and PAC system software.  So you take what we're trained on,  this digital infrastructure is connected,  vulnerable infrastructure, and you add a pandemic  on top of it, and then you take away the tools  that we're used to using,  undoubtedly patients would suffer.  So that one-two punch is definitely something very concerning.  To the heart of your question, which is,  how do we as clinicians make that trade-off  or decision about, here's a more secure device,  but it has less clinical utility or functionality  compared to this more secure device, et cetera.  So how do we make that trade-off?  Honestly, looking straight into the camera,  it's really hard to even have any bearing on that decision.  So a lot of the devices that we use  every day in clinical practice, we don't choose.  We show up to the hospital and there are monitors  that are there because they made that purchasing decision  five, 10 years ago.  When you train to become a doctor,  you may train for four or five years  on a particular set of medical devices  that you're gonna implant, a particular brand.  You become familiar with that.  And then so what do you do when you get out of practice?  You use the exact same one.  And so there are believable, believe it or not,  there's much less time to reflect  and much less ability than most people believe  for us to pick which devices we use in clinical practice  and to make that decision.  With that being said, we're trying to educate  other doctors on this.  And when we do, like Jeff mentioned, they care.  They want their patients to be safer.  They wanna use secure medical devices.  They don't want their patient's health information  or even their health to be at risk.  So we have sympathetic ears from the clinicians.  We just lack the ability to make it easy for them.  What you don't want is your doctor to have to go to,  you know, 14 years to train to be a doctor  and then go and have to take another year  to a cybersecurity coursework or whatever  to become competent.  You really have to make it easy for them  to make the right decision and convince them  that they should tell other people in their hospital  and other doctors that it's important.  I think to that point, that's maybe a misperception  that the expectation is that everyone in the supply chain  has to become a cybersecurity expert.  I think maybe that's a fatal flaw  in how we think about solving this is  if we need to sufficiently educate individuals  by having them, you're not a medical device practitioner,  but hey, I can teach you all about cybersecurity here.  Like it's really hard to think  that you can sufficiently educate folks  that this isn't their core competency on,  not to say they can't have some level  of understanding what the impact is.  And I think making it tangible for their work stream  or function or whatever the case may be  is the perfect way to do that.  But I would love to hear kind of the thought  around how we think about leveraging those who are  experts and not necessarily trying to solve it ourselves  and going at it alone.  Because I think that that's probably a fatal flaw  and has caused some of the historical challenge  that you're talking about when you inherit a hospital full  of devices that you didn't pick.  And surely as well, like Jeff was saying,  normally physicians are used to sort of having to balance  different pros and cons of different requirements  to make the best medical choice for each patient.  But I think historically, maybe they weren't even aware  that this is one of the things they need to consider.  So even just how can we raise awareness  in the consciousness that, oh, that's something  that I need to consider even if I'm not an expert,  maybe I can have an expert on hand  or I can read a rating for a device  and make a decision somehow that way.  What kind of solutions are there  moving forward in that direction?  So I've actually spent some time working a lot  with the cardiologists and their technicians  in South Africa trying to just have the discussion.  And I had a phone call from my cardiologist saying,  we had a recall of devices.  I need to explant a device,  but I don't know how to explain it to my patient.  I don't even sufficiently understand  the engineering talk on the document.  Can you translate?  Can you translate?  So I think the big thing is we are using our language,  our linguistics, you know,  instead of making it familiar to the medical practitioners.  And I think there should be that portion in the hospital  that does that translation.  Because I can tell you,  that was a very difficult conversation having  with a 65-year-old lady that doesn't understand technology.  I mean, the doctor didn't even sufficiently understand  why he had to explant the device was just being recalled.  And that's the unfortunate thing,  is they are expected to make these decisions.  And as he says, he has a patient coming in,  his blood pressure is high.  He needs to adjust the ICD and the pacemaker.  What he doesn't tell him is that his wife  cooked a high sodium meal,  which affected his blood pressure.  And a week later, he comes back  because he's having issues, their peers changed significantly.  So those are the challenges they deal with,  you know, having to solve the clinical puzzles.  And I think adding that, you know,  the technical stuff on top of it,  especially using terminology they don't understand,  is where we've been going wrong.  We should be finding ways to translate it better for them.  Yeah, I think, and also I think that that's a failure overall  of how we've designed not only medical devices,  but how we do cybersecurity overall, right?  Like so many times, if somebody gets hacked  or whatever, it's their fault.  Setting aside corporations who get, you know,  DDoS or whatever, and then claim to have been a victim  of a sophisticated cyber attack.  Like there are times when people get hit by something  and it really wasn't their fault.  There were so many different things that they've been expected  to know.  And I think something a professor once said to me,  stuck with me, we were talking about this whole thing about  like, oh, like people are just so dumb about cyber,  like why can't they just be better?  He's like, yeah, and all those people  who were driving Ford Pintas, like what the hell  were they thinking letting their cars blow up?  And, you know, I think that that's kind of the way  that was sort of a light bulb moment for me  and being like, oh, we have to design devices  that don't blow up in people's faces.  And, you know, we have to make it as easy as humanly possible  to do something securely.  Like doing something securely needs to be the easiest thing.  And it's doing something securely  is the most difficult thing.  That's not the fault of the user  who then didn't do the secure thing.  That is the fault of the designer  who made the device poorly.  And so I think, you know, we've gotten away with,  with frankly in a lot of cases being really lazy  and being an offloading the responsibility onto other people.  And I think what we have to do,  especially within the healthcare sector,  is all parts of it, we have to sort of reclaim  the responsibility that has been ours all along  and actually really do what we need to do.  Yeah, I mean, the responsibility, I'm very big,  the responsibility to build these things better  is the manufacturers, right?  They're the ones that have the control  over the firmware and the hardware.  We shouldn't be pointing the finger at the hospital saying,  hey, your device, your network got hacked  because you have vulnerable devices.  Well, those should have been built better, right?  Because now we're leading, we're in the industry now  where we're putting band-aid on, right?  The vulnerability is disclosed, now we have to run  and we're always on the back foot fixing stuff.  But the manufacturer should be stepping up  and should be doing a better job at this.  I completely agree.  Because it's taking responsibility.  I completely agree, oh, there's a lot  that the device manufacturers can do, should do,  and I think some of them have been doing.  But there is also a key important thing,  they can't control it once it's deployed.  So if they're deploying a somewhat secure medical device  on an unsecured network,  or if they're turning off security controls  for sake of integrating into their clinical environment,  or if they're not patching critical systems  around these medical devices,  there's some shared responsibility there.  And I think that's really important,  is that we need to be having not only a strong conversation  with the device manufacturers,  they do the right thing from the start,  but we need to be able to be giving hospitals  the appropriate resources education  and holding them accountable  when there's something egregious.  Now, I don't think they should be asked to solve  the issues of insecure medical devices,  because they just can't.  But we've seen time and time again,  some pretty horrible security practices  by healthcare institutions  that have nothing to do with medical device manufacturers,  but yet still could have patient safety implications.  Things like all the denial of service attacks we see,  the ransomware attacks, the theft.  I mean, it's not just critical healthcare infrastructure  like hospitals.  We've been seeing attacks on research.  So I'm sure people have seen in the news,  and I think Jeff mentioned,  ransomware attacks on critical medical research infrastructure,  state sponsored attacks on COVID research  and vaccine trials data.  So we have to have a shared responsibility.  This is a whole ecosystem.  It's really trying to figure out  what's the best bang for our buck.  And I feel like a lot of the conversations  focus around medical devices,  because it's a much easier thing to tackle  than trying to go to a rural access hospital in South Dakota  and fixing their broken network,  and then being able to monitor that and continually say,  how are you going to,  how are we gonna secure every hospital and clinic  in doctor's office in the entire United States?  That's a much harder problem.  But I...  Kariya?  Oh, that's it.  I'd like to...  Sorry.  I just wanted to say, right,  it's not just the United States problem, right?  It's a global problem.  You know, hate to be the one to point it,  but this is a big problem everywhere.  And I know everyone on here is from the US.  I'm not only oddball, I'm from Canada.  I don't feel so alone.  But the thing is just every hospital is different, right?  There's no cookie-cutter way of looking at the problem.  Every hospital network is different.  And I think sometimes what we forget  is the patient record systems.  Like, I never used to consider it.  But then I sat back and thought,  from the cyber criminal perspective,  it is the ultimate identity theft.  It's the thing that you can keep selling constantly.  And it's often the thing that's least protected.  Because we're focusing so much on medical devices,  we've forgotten what the treasure trove that holds.  And I mean, that's moving to the cloud.  So, you know, Microsoft is one of the big proponents  that's moving everything now to the cloud  in terms of patient records and clinical systems.  So I'd be interested to see how the smaller hospitals  are actually going to be able to cope  with having to do, you know, permitalist defenses,  you know, zero trust networks, or even cloud solutions.  Because I think the big hospitals like Mayo might be fine.  But you have the smaller rural hospitals  that I don't even know if they have IT or security  on the premises.  Or are they paying someone a third party  to do the solution for them?  Yeah, I'd like, sorry.  I wanted to go back to the topic of responsibility.  And I would have to echo with Christian.  I think it is a shared responsibility.  Obviously, I think that, you know,  I work for Medical Device Manufacturer.  I'm a software, a netted software engineer.  I think about these things.  I want to make sure that I'm doing my part  to build really secure and awesome, safe products  for end users.  I care a lot about that.  But I think it's easy to blame manufacturers.  And it's often more complicated.  Again, it's this balancing act.  So I work for a Medical Device Manufacturer  that's a consulting firm.  So we don't have any of our own in-house products.  We have people who come to us and we have clients.  And, you know, some of them, you know, business people  and some of them are other medical device manufacturers  that subcontract to us.  But we have a lot of people who are just doctors.  And clinicians and other people who  are really passionate about saving lives.  And they come to us and they're like, we have this idea.  We think we see this gap.  We desperately want to help our patients.  Please help us build this.  We have this tiny, tiny budget.  We only have this amount of time to do it in because, you know,  we got this little budget from these people.  But they say we have to get it done by this time.  And, you know, we were working really, really hard  to try to meet all these deadlines  and actually make this little device that, you know,  if we can give it to those doctors, it might save lives.  And then for us to come in and try to say, like, oh, well,  we have to add all of these security things.  And, you know, maybe it's going to affect the budget.  It's going to affect the timeline.  And, you know, again, I care about these things.  I'm in there.  I'm advocating for this stuff.  But it's hard to say, you know, if you only have this much  budget and it's how do you choose?  How do you tell them, OK, you shouldn't.  We just aren't going to be able to make this  and implement this part of security, like,  where to draw the line.  I think it's more complicated and harder, like, in practice.  And it's hard to just say, well, the medical device  manufacturer should just do it.  You know, how do you make that choice?  I actually want to add to that story.  So I started working with Meitronik.  And I'm a hard ass. Everyone knows that I say a bunch of things  three years ago.  Cool came out because I was unhappy with how our conversation  was because we had a legal situation.  And I started the first six months of my project listening  and getting to know the developers and the software  for engineers and the hardware engineers.  And I sat back and listened and realized that these are real  people that are wanting to make a real difference.  Really, they are.  And one of the specific engineers, Rob Mastow,  had a big impact on my life because he said to me,  so tell me what you want in your device.  Tell me all the security you want.  Get the list to me and let's discuss this.  And he turned around and said to me, well,  so you're getting a new device every three years, right?  And I said to him, well, no, there's no fucking way  that I'm doing that to myself.  And he's like, well, then you can't have all of that.  He says there's trade-offs for everything.  So specifically with embedded, implanted devices,  the more security you add, the less lifetime  you'll have on your battery.  Because I can tell you the one thing that that is worth gold  is how long your device lasts.  Because it is a horrible thing to cut out.  And I don't know if you guys know, but the more you cut it out,  the more scar tissue forms and the worse it gets.  And I've had my device since 19.  I wouldn't be here without it.  So people building these things, I would be dead.  I had two weeks left to live.  That was the serious situation.  So I think sometimes we want all the security in the world,  but we don't realize these trade-offs.  These devices are there to save lives.  And if we try and push security too much,  there's a whole lot of state of devices  that will never come to market and they will never save lives.  And that's one big lesson that I had to learn  and I had to have humble pie.  And I had to swallow my words and realize that,  is people behind this wanting to make a difference  with technology and science?  And without that device, I wouldn't be here.  I wouldn't have my kids.  So technology does save lives in ways.  We must just find the perfect balance  to balance everything out.  In your perspective, V, and folks like you,  like Marie-Mau, who have these incredible security  backgrounds, but also our patients,  is so incredibly valuable to be able to consider  these types of trade-offs in a human person  in a very real way.  And one of the things that Christian and I are working on  is how do we understand the role of the patient  as a stakeholder in these types of conversations?  And is there a utility in having those types of conversations  with this six-year-old man who needs to get his pacemaker  revised?  Or is there a trade-off between the potential risk  of a vulnerability being exploited  and then just putting him through that process  of having it revised?  And I think it highlights one of the things  that Christian and I are so interested in  is just developing a data and a science around this  to be able to say, in other types of medical therapies,  we have very clear risks.  If I give you a blood transfusion,  there's about a 1 in 1.5 to 2 million chance  that you might have an infection that's  serious as a result of that or heart or lung issues.  We really don't have the ability to have conversations  with patients and say, this is the clear-cut risk,  because thankfully, we haven't seen many incidents of issues.  And that doesn't mean that there's an absence of risk,  but it makes it harder for us to have discussions with patients  and say, really, this is the type of math  that you have to weigh as a patient with your own values  and preferences and things like that.  And we've kind of done some work on touching on this idea  of a cybersecurity-informed consent  to say, we talk about risks, medical risks of surgeries  and medications and things like that that they may be on.  Should we start thinking about cybersecurity  as a potential risk that should be included  in this conversation a clinician has with the patient?  In Jessica, I know the FDA is also really interested in this,  and you've had a lot of really cool outreach types of events  to be able to talk to people like V.  And I think it's a challenge of moving beyond people who  are security literate to people who may not  have heard about these issues whatsoever for as much as we  talk about them and to be able to kind of have  that enter the conversation so that they can,  without having a background in cybersecurity,  weigh that as part of their clinical process.  Yeah.  I mean, I think that there's a couple things that I would  bring up on this.  I mean, I think, one, you had mentioned  that FDA has been doing a lot of this.  And I think V were actually at the October 2019 meeting,  but we had the Patient Engagement Advisory  Council meeting, the peak meeting.  And so FDA is very much trying to make sure  that the patient perspective is heard.  And it's not just yearly meetings and things like that.  As many of you know, Suzanne Schwartz is very approachable.  You can pretty much get her email and email her,  and she'll find time to talk to you.  So that is a really critical part of this,  because not that all of you are going around reading  government white papers and things,  but a couple of years ago, the FDA put out this medical device  safety action plan.  And so they're setting aside the flowery language  that all government documents, I think,  were like decreed that they must use.  Essentially what it says is, look,  it's exactly this issue of balancing  the idea that patient safety is a twofold concern.  There's the concern that we all typically think about  on cybersecurity.  Is this device safe and effective enough  to justify its use with the patient?  There's also the flip side of that, though,  of what Asher was talking about and what  V was talking about in that, what happens  that the patient can't get the device?  That is another, that's got to be a thing that gets considered  too.  And so, you know, FDA, when we talk about cybersecurity,  when we usually do it, we're very explicitly talking about,  do you have security controls?  Do you have sufficient whatever to make sure  that the device is cybersecurity?  But a less talked about part of the work,  but that is equally important, is this idea  that we recognize that sometimes devices  need to get to patients because of the benefit  that they're going to be able to provide.  And maybe we don't have as much information.  Maybe we don't have as much assurance of the things  that we would like to.  But, you know, this is the breakthrough device program.  This is some of these other investigational devices  that come up.  And that FDA has specific categories for and authority  for them, things that you can do this.  And then it's exactly this recognition  that the other folks on the panel are pointing out,  which is that in some cases, not having access to a device  can be as big of a risk as having access to a device that  isn't secure enough.  And so I think that is just an important point to highlight.  And that we as an agency keep in mind,  but that this community, in addition, also  keeps in mind as we have these kinds of conversations.  All right, I want to change this.  Oh, can I?  I'm going to change the subject real quick,  because I heard a rumor there are thousands of hackers  watching this stream right now.  There are literally thousands of hackers  from across the globe watching.  You know, very interested in what we're talking about.  And I'm sure they're thinking to themselves,  wow, there are a lot of issues here  I might not have been aware of anymore  and they want to get involved.  So hackers out there wanting to get involved in this space.  How does, we've heard some examples.  You know, there's been a group of hackers  that have tried to defend hospitals against attackers  during COVID.  There are hackers printing protective equipment  for doctors on the front line, et cetera, nurses.  But what's the next generation of this?  Like when the pandemic's done, how  do hackers play a role in this space?  Continuing research, is it bug bounties?  Should a hospital have a bug bounty?  Should a device manufacturer have a bug bounty?  How does HIPAA play into that?  The protection of patient information.  How do we really kind of bring hackers more into this space?  OK, I'm going to jump in, yeah.  I have a dream and I have a dream with a friend of mine,  Nina.  We want to establish a laboratory setup with devices,  right, that people can actually do research on,  that we can put on networks, that we can scan for vulnerabilities,  that we can reverse engineer, that we can work with manufacturers  to make better.  I already see people wanting to break things, right?  We want to find flaws.  We want to find problems.  But often, we don't want to find the solution to the problem,  because that's the harder part.  So I want to say that if you find a device that's vulnerable,  first remember, they are human lives at stake.  This is not an ego thing.  Try and find the solution to the problem that you found.  Write it up as you would any scientific paper.  I know it's hard work.  I know it's not glorious and it's not sexy.  But when you hand that over to the manufacturer,  they have all the data.  And the more data they have, the faster  that they can verify the claim, they  can start acting accordingly.  It's not going to be easy.  Not all MDMs are going to want to be friendly,  but it is changing.  But the biggest thing is the more thorough your research is,  the less dispute they can be, and the more  we can find a way to fixing it.  But it's not all about just bug bounties,  because I think that for me creates a negative thing of we  just finding the problems.  We should be finding problems and then solving them.  That's the hacker mindset.  It's not just about pointing fingers and saying,  this is a problem.  We should be working at the solution.  We should be finding new ways of doing things.  So I want to see a lab full of medical devices  that hackers can pull apart and we can build solutions.  But not only hackers, you need people like Ash, right?  Because I mean, we're good at the other side of it,  but I can't build these devices.  I wouldn't even know where to stop.  So I think it needs to be a multidisciplinary approach.  But even people like the FDA should  be involved in it from a policy perspective.  And everyone should just be working together.  We should be building the bridges towards having synergy  and having a space where we can explore this.  And I think Biohack is awesome for that,  because they would really do a lot of that.  Sorry, everyone.  Excuse my cat, who just really, just really  jumped into my lap.  I was trying to prevent it and failed.  He just failed the side of the chair.  Welcome addition to the panel.  So I love the notion of being part of the solution.  And I think to your point, it does require multiple voices.  You hit on a point that I think is really important, right?  You as someone who does a lot of security research  is not totally in the space of how  do I build a medical device.  I think we similarly need to not have the expectation  of the people building medical devices  to know how to solve some of these cybersecurity issues.  So I think having the willingness to partner  and really helping them not make cybersecurity their number  to core capability and instead offering them solutions  is absolutely critical and the way to make things actually  stick.  So should we have solution bounties then?  Oh, yeah.  Oh, you should trademark that.  Yes.  But I just wanted to bring up just a scenario.  So it's not just for medical devices.  You have a medical device bug bounty, for example.  If that's a good or a bad thing, we could talk hours on that.  But you don't have the issue of patient health information.  And so hospitals can't offer hackers to come in  and poke around the periphery of their networks  or to give them that type of experience.  Because of a couple of things.  One, their live networks are taking care of patients  who would hate for something bad to happen.  But then two, there's so much protected health information  floating around a hospital's network.  And if a security researcher or a hacker in good faith  trying to do right finds something  and it happens to be commingled with patient health  information, then the hospital is  required to report that as a hip breach.  So there are some perhaps some would call barriers,  perhaps some would call safeguards.  But what I think it's done is it's made not just  the delivery of health care itself, particularly difficult  to engage with from the hacker perspective.  And I think maybe we don't have to figure out that solution.  Maybe that's one of the solution bounties  we should offer.  And how can hackers get more involved, give us your ideas.  But particularly at hospitals where we've  talked about they don't have a lot of in-house security  expertise, if anything, but yet they're still asked  to do all of that work.  How can the community come together and help them out?  Because that could be a lot of good right there.  One very easy answer to that is just working in health care.  So I would encourage anybody who's  thinking about a career change, if you're looking for an area  where you can make an incredible impact  and have a lot of really challenging problems to solve,  consider working for a hospital or a health care delivery  organization or being adjacent to that in some way.  We know that some of these hospitals  we talked about in certain areas are critically underserved  and may not even have a full-time IT security  professional.  So one way to get involved is just  to join that effort as an actual employee  of a health care organization.  So I've got some specific suggestions, I think.  And it's funny that you bring up HIPAA and other things  like that, because I think one of the best ways that folks  like these could get involved, whatever,  there are actually specific programs  that are being offered now within the federal government  that puts you in policymakers' offices  and lets you be the cybersecurity expert in the room  for those people.  So there's the Congress Fellowship.  Some of you may have heard of it.  And you may have met some of the fellows, Chris Fagoyan,  who's probably a well-known name for some of you all,  was a tech congress fellow.  He's now works permanently for a senator  and is like that senator's tech person.  But there's been dozens of others.  They've got great folks.  It's a great program, places you with either a house  office or a senate office.  And so that's one of the most direct ways  that you can actually influence policymaking, in a sense,  is actually go be in the room when the policy is being made.  So I would highly recommend that.  I don't know when the application process opens  for next year's class, but Travis Moore is your man,  so go find Travis.  The other thing I would mention is I am the cavalry.  When I am the cavalry was first started,  it was kind of this weird kooky thing for the Hill,  for the United States Congress, who's  very used to dealing with very buttoned up everyone's suit  and tie professional trade associations and things  like that when they were trying to get information on whatever  they needed to get information on.  And I am the cavalry, just kind of came up to the Hill  and was just like, hi, we're a bunch of cybersecurity experts.  We'd like to talk to you about this.  But the thing is, there were the most effective advocacy  groups in national policymaking right now.  Because they were just so, everyone  could tell in a city where sometimes people's motivations  are a little bit shrouded, that they just  wanted to be there.  They were there with good faith, and they really just  wanted to do the right thing.  And they were going to answer whatever questions  needed answering without bias.  And so I know SEA works a ton with I am the cavalry.  I know the Hill works a ton with I am the cavalry.  And so if you want to get involved  and have an opportunity to meet with the movers  and shakers of the country of cybersecurity policy,  I am the cavalry is another really great place to do that.  I just would like to perhaps add that from another perspective,  if you want to get a hold of working in a hospital  or getting your hands just on some medical equipment  or just having a conversation with the manufacturers.  Three years ago, I spoke at Biohack Village.  This was my first day of gone.  This was my first ever meeting.  Another group of hackers like myself.  And that's how I was set on the path to do research.  That was how I got access to devices.  I no longer had to smuggle them through customs  at an airport because I bought them on eBay or because you  can't buy them in South Africa.  So I would buy them in the UK.  I'd go visit a friend.  And I'd put them in my luggage and smell them back home  so that I could work on them.  Or even when they exploded my first devices,  they'd give me, give me, give me.  I want to have that.  And my doctor's why I'm like saying to me for reasons,  of course.  But that's how I got my hands on.  But here I walked into a place in Vegas that had devices,  that had a lab, that had everything that makes  a young hacker's heart do jumps.  So if that tickles your fancy, then  that's somewhere that you should go visit.  You should go try your hand at it.  You should go play with those devices,  speak to the manufacturers, and go  see how they implement it.  And I mean, even if you're a patient,  nothing precludes you from giving advice at a hospital.  I mean, I've given advice from my ICU bed.  There's a reason I'm not allowed a laptop in anymore.  Except for that the doctor wants me to ask more.  Apparently, that's the reason, but give advice.  Reach out.  Get involved.  And if someone doesn't want to listen to you the first time,  keep on trying, because it's a patient's game.  It's not going to happen overnight.  I've learned.  I've dealt with many parties, and it's not going to happen fast  because it's a big industry.  Just get involved.  Get your hands dirty, because that's how we change the world.  Every new researcher that comes in the fold  is a new mind that thinks differently.  And that's, I think, how we change things one step at a time.  I love that, because I think the idiosyncratic notion  that it takes a specific individual or an elite person  or someone who's an expert to make a change  is absolutely a misconception.  So the notion of just getting involved,  there's no inner circle here.  I think every person in this community  is more than willing to share their networks,  share their expertise, and share whatever corner  of this problem that they have.  It's a really hard problem, and we  need all the talent that we can muster up right now  to really make a change.  And I think if we look at the diversity of roles, functions,  experience, thought process, background, education,  everything, any underrepresentation of that population  is indicative of missing a potential attack factor.  And we really have to really have to account for that  and try to bring it as many folks as we can.  Yeah, I think this is a call to arms, right?  This is how many hackers can we round up in one go?  Well, not just even hackers, right?  You know, everyone, from physicians, from patients,  from borders and policymakers.  I think if we start breaking down the silos, which we've  been functioning in, and independently working,  and start this collective movement,  that's how we change the world.  It's not a one person thing.  It's never going to be one person that's  going to come up with the magic solution or the silver bullet.  It takes more than one village, as I always say,  to solve the problem.  And I think it's time in 2020 that we as a society  build together instead of apart, because we've  seen what a virus can do.  It's locked us in our houses.  So I think it's time we take our power back,  and we start moving the world into a positive future.  Because I can tell you from a patient perspective,  not having access to being able to go into a cardiologist  office because we're afraid of COVID,  because currently it's more dangerous going into a hospital.  A denial of patient care is a non-acceptable, non-negotiable,  not going to stand for it.  And I think we should be kicking down the doors  and saying, now's the time, not yesterday, now.  I love it.  Are there any better sentiments to go out on than that?  I think it's easy to be cynical about all this,  and it's easy to say we're having the same conversations  year after year, and the problems are still there.  But I mean, in 2015, the biohacking village  didn't exist, and the idea of people from device manufacturers  bringing stuff for hackers to poke and prod at  would be laughed out of the room.  So for people who don't believe that we can do awesome things  when we work together and hang out at places like DefCon,  I mean, we have evidence to the contrary.  So I think that's about an hour, and I'm  looking forward to seeing you guys in a little bit for a Q&A.  Take care, everyone.  Thank you, DefCon, for having us.  A shout out to Nikita, especially,  for taking another shot at us.  We're going to go ahead and open it up to Q&A here in five,  four, three, two, one.
DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_0_2375.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_2376_3095.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_3096_5339.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_5340_5423.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_5424_7079.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_7080_7259.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_7260_7811.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_7812_14435.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_14436_14567.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_14568_15179.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_15180_15263.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_15264_15467.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_15468_15923.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_15924_16115.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_16116_17195.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_17196_22403.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_22404_25427.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_25428_25799.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_25800_27311.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_27312_27371.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_27372_27875.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_27876_28355.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_28356_33383.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_33384_33431.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_33432_43043.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_43044_45191.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_45192_45419.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_45420_45611.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_45612_45947.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_45948_47939.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_47940_47999.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_48000_48887.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_48888_48947.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_48948_49019.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_49020_51323.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_51324_52595.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_52596_52643.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_52644_55751.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_55752_56075.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_56076_56351.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_56352_56531.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_56532_58631.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_58632_58679.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_58680_58955.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_58956_59003.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_59004_59099.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_59100_59759.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_59760_59807.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_59808_60143.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_60144_60203.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_60204_60551.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_60552_61031.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_61032_61931.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_61932_62279.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_62280_62327.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_62328_62471.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_62472_66443.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_66444_66515.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_66516_66635.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_66636_66719.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_66720_70175.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_70176_73379.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_73380_73595.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_73596_79259.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_79260_79547.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_79548_79715.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_79716_80315.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_80316_80363.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_80364_81011.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_81012_81119.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_81120_81191.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_81192_83987.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_83988_85031.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_85032_88487.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_88488_88547.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_88548_88715.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_88716_92027.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_92028_93059.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_93060_96743.png DEF_CON_Safe_Mode_-_Christian__quaddi__Dameff_MD_a_fAU7V3pvj1Q_96744_96790.png