Integrate all the Things!
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Long, Mike
Hardware, firmware, software, cloud, big data - help! How do the best companies manage the complexity of software development in the world of the IoT? This presentation will show the challenges of developing for the Internet of Things, and provide examples of how real companies have managed the complexity with continuous delivery, devops and test automation.
 Hi, everyone. My name is Mike. I'm from Pragma, a continuous delivery consultant. This is Internet of Things Day, so I want to talk about integrating all the things. My background is in embedded systems, large-scale embedded systems for the oil and gas industry. So I spent more or less 10 years building these big industrial things that you use to find and drill for oil and gas. But since then, I've been focusing more and more on making teams develop products faster and the tools and practices around that. And that's what I want to talk about, the intersection of these two things, how to make products that are physical, integrated, Internet of Things devices, and how to do it efficiently. And look at some of the, so I would say the ecosystem around all this. So our mission, should we choose to accept it, I want to show that there is some real challenges, obviously, in developing reliable IoT products. There's been a lot of talk about security today. And I think this is a big concern we all have. And I think one aspect of security is one aspect of the kind of quality we'd like to build into these products. So we're going to explore the techniques that we can apply to the problem and look at case studies for customers, for companies that are already working on this stuff. I'm, like I said, an embedded software person with a flavor for DevOps these days. I'm also a trainer. I do give courses in Git and Jenkins and TDD. And I'm a certified Docker trainer for what that's worth. And I also run a business here in Oslo that helps companies adopt continuous delivery. So how many people here consider themselves embedded software engineers? Okay, there's a few. So for me, when, I don't know if you guys are the same, but when I first heard the phrase IoT, I kind of rolled my eyes. Actually, I felt like I've heard this all before. We had this 10 years ago when it was called M2M. And it also feels like, in a way, DevOps to me does. It's a cute phrase that's fun to say. But it just gives a name to something that's been around for a very long time. Whatever way you think about it, IoT has always been there. We've always made embedded devices with internet connectivity as long as there has been internet connectivity. On the other hand, it is a really fun space. And despite the fact that it's got a funny name, IoT is actually a ballooning industry. There's more and more things that we're connecting to the internet. This is why we have IPv6 in the first place. So we're going to have an explosion of smart devices. And this is fun time to be a technologist. So I'll try and not to be a grumpy old man about IoT. And I'll just embrace it today. So I don't want to describe IoT, but briefly I just want to look at the attributes about IoT, the IoT, and how they were, these will play an important part to how we can build these systems. So the most important thing about IoT is that they're the T, actually, the things, they're physical things that happen to be part of our world. There's hardware and software co-design and they interact with our environment. They are either sensing and monitoring the environment or they're actually actuating on the environment that we work in. They're also distributed in nature, connected via networks. And often, although they can often work in coordination, they're usually independent actors focused on one specific need. And the idea with these IoT things is to add some intelligence to otherwise dumb parts of our world. It's not just fridges. So this is another way to describe what IoT is, intelligent, distributed, physical systems. Now, for those of us that come from a software background, actually all those words kind of terrify us because we know that distributed systems are hard. We know that making intelligent systems is hard. We know that making physical systems is hard. So that's what I want to talk about today. This is a hard thing about hard things. What do we do when we're faced with these very difficult problems? There is complexity in this problem. There's complexity in distributed computing, of course. This has been known for a long time. The guys at Sun, microsystems always knew this. This was their domain, actually. Their problem space was the network computer. And there's a list of fallacies that everybody working with distributed computers or designing distributed systems kind of fall prey to. We design these systems thinking that the network is going to be reliable, latency is going to be zero, bandwidth is going to be infinite, blah, blah, blah. We just can't think of all the ways that these things can go wrong in reality. But that's not, and that's part of IoT, right? They're distributed systems, but they're not just distributed, they're physical systems. So there's a lot of fallacies that people making embedded systems, hardware and physical systems fall prey to as well, such as the hardware always works. The environment never changes. The resources are endless. We're never going to run out of memory. The sensor is always going to be calibrated. The data we're reading from our sensor across three time zones is definitely live. It's not just returning the same thing over and over again. We're assuming all kinds of things. The clock is in sync. The data it's giving us is actually at the right time that we believe it to be. And then, of course, we have the more life cycle issues. How do we keep this thing on? Is the power reliable? What happens when we lose power? Is there a safe time to lose power? And what happens when it comes back on again? How do we keep these systems up to date? And actually today, most of the talks that we've gone, that I've been to, I've been security focused in some way. And Hans Alpou was in the rifle talk. Right. This was a really cool talk about how to hack a Linux powered rifle. The interesting thing is when the flaw was discovered and it was published, the manufacturer said that they would send out an update on a USB stick to the however many thousand of people that have bought the product. That's not a really scalable way to update your system. We need systems to be up to date. And as we connect them to the network and we bring them into our home, we bring them into our body, we let them drive our cars. It's of paramount importance that we keep them up to date. And then of course, there's the fallacy of intelligent systems. There is the idea that we'll always, I mean, we're smart enough to figure out how to do this thing. And that's never true. Complex systems never work. It's been known ever since we've had systems, actually. Complex systems never work. The first time you do anything, it will never work. John Goll has written a lot about this, but so has Jerry Weinberg. There's a lot of good information. Obviously, the first thing, the first time you do something, it never works. But that's why you should start small. All the best systems start from small simple things that work. This sounds pretty familiar, right? And this is the way we've approached software in the last 15 years as well. We don't want to design a big complex system. We want to start with something small. It's right to make it bigger and bigger, more and more features, do more and more stuff. This is how we figured out how to actually make software. We realized that we couldn't do complex things. We couldn't design it up front. For me, there's two very important principles in NAHGEL manifesto that come into play here. The one is that obviously, we want to start early, give value continuously. And that working software is the primary measure of progress. So without these, those two are actually another way of saying this. How do you start from a small system and grow it into something that's complex and still have it work? And it's these two principles that are the enabler for that. And for me, coming from a continuous delivery point of view and continuous integration point of view, these are the two principles that we try and solve with continuous delivery. So I'm not going to go through, this is just a fond cartoon, but I'm not going to go through all of this, but the point is that there is a system in place that we've decided in software to make sure that if you like continuous delivery is the technical implementation of those agile principles, how do we make sure that we can continuous delivery deliver valuable software that works? How do we make sure that the software already works? Well, we have continuous integration, which is the gray box. We say that once the software builds and it runs any kind of simple test, then it's good enough to share with my colleagues. That means I'm not going to break it for anyone else. But that's not to say that after that point, it's good enough to ship, it's good enough to put in your car, it's good enough to put in your pacemaker. After that, you need what's called the continuous delivery pipeline, where any other steps in your process that need to happen before the code can be considered potentially shipable has to happen. And obviously, I would promote as much of this to be automated, but some of this stuff can take a really long time. For instance, with the trucks at Volvo, they have to go through many, many steps before you could say that this is actually good enough to go into a truck. But still, there is a pipeline, there is a process, and there is a way for always working software to be ready for the customer. That's all it is. Continuous delivery is just the implementation of those principles. So I want to go back to the things and the Internet of Things, because for me, that's the most interesting part. The Internet isn't interesting, it's just a network. But the things that we attach to it are actually the important thing. I'm going to contrast that by looking at a project that was actually my very first project after graduating, and look at how we made software then and how we make software now and see the differences. So I was young, I got a job after university to join a company that was making products for the oil and gas industry. And my first project was to make a rotary steerable drilling tool, which is basically a long tool that has some pads on it on the side and then a drill bit. And then when you're drilling for oil, what you do is you put this down into the well bore and you spin it around. You have a long pipe all the way up to the top, to the rig, and you spin it. Spin it really fast, like 200 times a minute. And while this thing is going around, you want to spin it. You want to steer it. So you've got gyros, you've got accelerometers, you've got shock measurements, all these things. And in real time, you want to try and drill a path towards where the driller wants the drill bit to go. So this was quite a project for someone straight out of university who never programmed C before. I was determined to make a go of it and I wanted to learn as much as I could. So I wrote down all the steps that were necessary in this professional world of software development. So I found out the first step you do when starting a software project is you take a copy of the code base for the existing product. So we're going to make a new tool. Well, there's an old tool already. Let's just copy the code base for that product. We'll start from there. And then what you do is you spend a few months documenting its current algorithms. This is important because nobody knows how this code works. The person that actually wrote the control systems long gone. And it was done four or five years ago. So it's going to take some time to figure this out. You might need to have a systems engineer around for the math. Then what you want to do is be very careful about the changes you make to this source code because you know what kind of sort of works. So any changes you make, surround them by FDF so that you can always switch them off again if it doesn't work out. Then the most important and the longest part of this development process is testing. So what you want to do is you want to test it on your own hardware on your desk. You want to put it into a tool and test it down in the shop. You want to put mud through it and power up and watch it steer in an automated flow loop. And you want to go to Oklahoma, you want to go to Siberia, you want to test it on the rigs. Take as much time as you like for testing. And then finally when you're happy with the results, everybody's happy. You commit that code as a new project in the version control system. And you hope someone's going to take care of merging it into the previous project at some point in the future. Then you compile it on your laptop and give it to manufacturing. That's how we do professional software development. At least that's what I learned. I knew it was professional because we had a software development process and I followed it to the letter and it was part of the product development process and part of the project governance approach. So we were ISO 9000 certified. So we were very professional. I mean, I am being ironic though, but this project was actually considered to be very successful. It was very successful. The product worked, it met its commercial goals. And that was largely due to this step four where it was lots and lots and lots of testing. So there's two sides to this. This can work in that kind of context. But if we move towards the modern times, this type of software development doesn't work. And I want to contrast this kind of project to the way we do embedded systems now. So ideally, instead of having this as your professional, as your software development process, you would have something more like this. As a developer, what you do is you fetch the latest source code, which is up to date with whatever is released, what your customers are using. You branch, you implement your feature, you test it on your machine. When you're happy with it, you think it's good enough, you appreciate it to the central repository. The continuous integration system kicks off. It starts building your system, building your code, doing any unit tests you have, maybe any quick smoke tests you have. And if they're good, your code should get merged into the master branch. Then that should kick off a continuous delivery pipeline, which will do the static analysis, program your code onto a device, verify the device and the hardware and the software work together in an environment. Maybe you have a lab setup. You might even do canary deployment, where you deploy this to some customers and check that they're still happy and there's no complaints before you send it to all the customers. And then you want to get some real-time production feedback for those things. Now, I don't think there's anything too controversial here. This is the way most software is developed in the world. This is the way we should develop. This is if you go to any newspaper or broadcast or this is anybody with a website, really, this is their approach. So let's take a look at the difference between a project like this and a project like this. So back then, embedded systems were versions like this. You just put them in a new folder and give it a different name. If you are really, really conscious that you might make a backup every week and give it a name of the date. That was how you did product versioning. But now we have distributed version control systems. We have branching strategies that actually make sense and work so that we can always work with working code as a team. Now, I don't want to talk about branching strategies, but I want to talk about this. If you're working in embedded systems, this is a great book. It tells a practical approach to large scale agile development. It doesn't sound very interesting. I know. But it tells the story of the laser jet business from Hewlett Packard. The embedded systems for those, there was 800 people spread across five different continents and they had really tough legacy software issues. The book tells the journey of them transitioning to an agile way of working. They had a different branch for every product and they were trying to maintain patches and branches across all these hundreds of different laser jet models. They had, I think, almost 100 people whose job it was just to merge and patch and build their software. You can imagine how slow and how unresponsive the system eventually becomes. One of the key learning points is just this. Actually having a release-trained branching model, branching strategy can hugely improve your development speed. Of course, now we have better ways of versioning as well. We don't just take the subversion number or the git sha. We have a way to put some semantic meaning on our versions as well from the products we built. We can tell consumers whether this is going to break, if it's a breaking change, if it's just a new feature, if it's a bug fix. You can figure out how you want to change your dependencies. Of course, we can stamp this information right into the products. This is important. Actually, Novella is a company that I work with. What they want to do is they want to, from one of their radars, be able to find out exactly what version of software is in that radar. From all the different components, all the different binaries, what versions they were and which git sha they were related to, which build information where I can find them in the artifact management system, you can put that right into the binary. You can ask the radar itself all this information. It's just simple stuff. You make a version.h and a build.h like we've always done. You have the system generated for you. Also, one of the things that have really changed between back when I started and now is that product families are really different. When I was starting systems, you had the old systems, which were maintained by what was called the sustaining organization and the new products, which were made by the engineering department. It was really interesting. Today, we talk about DevOps, about bringing operations and developers together. In a lot of engineering organizations, it's even worse than that. There's a support function and a development function, and they don't even talk to each other. The people that are making the stuff are not the people that are supporting the stuff, so they never get that feedback from the customers. That was very much the case. I think it still is in some places. Now, we don't have the old one and the new one. We have the family. We want to sell products. We want customization. Like Volvo, every single truck they make is different, but they have to use the same stuff. They have to use the same software, the same tools, the same tests. That only works if you can reuse. Of course, back then, I was building an ID. I don't think anyone does this now, but this was how we cut a release. We opened our laptop and we pressed build. I think that this blog post from Jeff Atwood really changed the way we approached this. It's crazy. It was only six or seven years ago where he said you should have a script for your build instead of using the ID, and it was considered like news. I don't think anyone would release software like this now. The thing that was missing, of course, is the traceability to figure out how was this thing made? Should I be worried? Where did it come from? How do I make it again? How do I reproduce this? Can I get the same results? What we do is we put this in, we create a Jenkins system, a server that can do all this. The idea is that, while at least the server we should know a little bit about, if it's a developer laptop, we don't know so much. We start putting all our configuration in this Jenkins user interface. I don't think that's a very good solution either because who knows, anyone can change that. Nobody can track it. The traceability and reproducibility of using Jenkins isn't really there. It has a lot of other benefits, but you won't solve those problems with using Jenkins. What you need to do is instead define your builds as code in some way. Find some kind of text format that you can version alongside your software so that you can track exactly how something was built in the past. You want to check out a version of the code from a year ago. You want to know exactly how the build server was set up then. This is job DSL. There's a lot of different ways to do this, but this is one way. You don't even have to stop there. You can go way beyond that and look at the whole ecosystem of tools you use and figure out what would be a traceability ecosystem for building these products. How could I use Artifactory and configuration as code tools like Docker and Vagrant and Puppet to actually be able to reproduce reproducible development environments from code that is version controlled. Maybe this seems like going a bit too far, but for a lot of, especially safety critical systems, this is the only way you can really control how something was built, which header files were included in the build, what was the library path when something was built. These are all important considerations and you can only do that when you have control over the entire environment. This is something also, for me, interesting about Docker because I was at DockerCon and there was maybe 100 presentations there this year and they were all nearly from web companies. All the places that I'm implementing, well, say 99% of the places that I'm implementing Docker solutions, they're embedded safety critical places where they want to be able to say, I want to rebuild this ASIC. I want to know exactly that this same ASIC is going to be built or I'm going to put some code that controls the breaking system of our truck. I want to be able to know exactly how it was built. There's a lot of different uses for Docker, not just for creating microservice websites. Then of course, if you have this configuration code, you can have Phoenix builds. You can basically spin up and spin down build systems as you please. They're no longer special dedicated build servers. You can use the cloud to rip them up and tear them down and create new ones as you please and the developers can have access to the same environments. One of the fun things about that rotary steerable drilling tool that I was telling you about is that the only way to update the software was to send a zip file to basically a workshop, a mechanical workshop, and people would maybe take the tool on a helicopter from the rig to the shop and the mechanics would open it up. Then they would attach a JTAG to it and flash the thing. That was the only way to update it. Now, we have a lot of systems where they just get updated automatically over the air. The latest version of the software is always available. Now, this is a really hard thing to get right with the IoT devices because there's so much that can go wrong when an update occurs. We get used to over the air software updates with phones and so on, but actually to implement this on an IoT device, if you make it yourself, it's very, very difficult. It can take a year of development to be able to have a reliable update system that you know that no matter what kind of failure mode, you won't break the device. Now, that's all well and good. This is how we build the software. I think the only way that we can actually have this continuous delivery of working software, the actualization of that is only if we take testing seriously. I see a lot of companies that really focus on automation, but it's always around build. Their testing is very, very weak. The testing is the only thing that makes sure what you're delivering is actually any good. There's no sense in delivering crap really fast. Anyone can do that. So there's a lot of discussion like how does testing fit into IoT embedded, all these kind of product kind of devices, like even if you're making cars, what does testing look like? This doesn't really fit. We don't really think about UIs and services. Obviously, we have units, but we do have a testing pyramid, but it's not quite the same. Of course, we need this idea of a system test, a full level system test, integration end to end, full test. This is the only way you make sure everything is connected. In IoT devices, this usually requires a lot of programming actually. You need to spend a lot of time to automate this because unlike unit tests where you just use a test framework for your language and start writing tests, system tests for embedded and IoT devices actually require a lot more. They're usually very custom things. You need to be able to control the power to the device, turn it on, turn it off. You need to be able to program it with new code, like access its flash. If it is a device that engages with the environment in some way, changes the environment in some way, you need to control those actuators through some kind of API. And you also need to be able to sense the results. One system I worked on was a steering system for marine seismic acquisition. What that was is a big cable you drag through the sea and you steer it around. They want to make sure it's at the right depth and it's in the right place. How do you test something like that? You can send it lots of commands and you hope that what it's doing is correct. But what you need to do is sense what it's doing physically in the world. You want to see that actually the blade is at this angle. So you need to have some kind of watch guardian over these systems to see if they are interacting with the environment that they interact correctly. We don't have services, but we do have components. We can test things in isolation and we should aim to test things in isolation as much as we can. The idea of microservices is just as equally applicable to IoT and is to figure out what is the way to isolate the change and find an abound in context for these devices. And of course, unit tests are great. We should have them everywhere and make development super fast. I get a lot of pushback actually from management when I show this slide because they say, well, if I'm writing all these end-to-end system tests, why do I need the unit tests? And if I'm writing all these unit tests, why do I need the end-to-end system tests? I didn't really have a good answer for that until I read this book, Growing Object Oriented Software Guided by Tests, which is really good. It doesn't really say much about IoT, but what it does tell you is different types of tests give you different types of feedback. And the important thing to know is unit tests give you a lot of feedback about your internal quality. They tell you that the code is easy to change, is probably well understood, that it works how the developer expected. It doesn't tell you anything about your external, about your, where are we? It doesn't tell you anything about your external quality. How much this, together as a system, meets the requirements of the customer. And the same is true for the end-to-end tests. The end-to-end tests tell you very little about how easy the code is to change, how safe it is to change, but it tells you an awful lot about whether the system as a whole meets the needs. Clear? No. So with IoT, it adds an extra level to this because what is a system test in the context of IoT? These IoT systems are distributed in a sense. When you say end-to-end, if your intelligent light bulb connects to a cloud service which connects to your phone so that you can control the hue or the color of this light bulb, how do you do an end-to-end system test? Do you include the whole internet? Do you include deploying a web service to do this? This is a really hard problem to solve. And you have to solve it because it's the complex nature of the interaction of all those things together that will produce your failure modes. Another important thing is that there has to be access to hardware. The developers and the testers and the customers, they all need access to this hardware so that you can make sure that what you're running in the lab is the same thing that the customers are using and is the same thing that you have on your laptop. So the goal is, I wrote this slide yesterday, the goal is to go from one-year product cycles to one-hour product cycles, the difference between using continuous delivery and the old way of doing software. And I thought that that was quite a gimmicky sales pitch thing to say. But then when Peter talked about Volvo, he talked about going from, I think, 46 weeks to 10 minutes. So it is actually, this was maybe a little bit underambitious. When you can get development cycles from one year to one hour, it really changes the way you can deliver product to customers and change your plans. Another interesting aspect of this is the Novella story. Now, Novella make this radar trip, it's very clever, it can do all kinds of things. It can detect your breathing, it can monitor your breathing, it can know whether you're sleeping, it can tell us how many people are in this room, whether they're dead or alive, for instance, small details like that. And they take continuous delivery very seriously as well. But they have a problem. When they make a radar that's in right signal processing that's supposed to monitor whether a baby is breathing or not, how do you test that? Actually, one of the first months I was there, I actually called up some friends that were on mama permission and they came with their babies and we put the radar on them. That was fun because we had to tell these mothers we're going to put radars on your babies. Is that okay? Surprisingly, they said yes. That's some trust. But that's not reproducible, right? Even if we collect all that data and run it over and over again, that's not a good test. It's maybe a good acceptance test, it's maybe a good product demonstration, but it's no way to run your development process because between the time that test is happening and the time you deliver your product, you might have made a thousand or two thousand commits all changing the behavior of your code. So the guys at Navelda were very clever and what they realized was that if they took a ball bearing and rotated it at a certain frequency to the radar, it looks just like the chest cavity of a baby. You get basically the same information. And because LEGO is more or less invisible to radar, it doesn't have a very high reflection, this was a great way to, this was a great solution. What they did was they set up a lab with all this LEGO. There's people at Navelda that get paid to play with LEGO and they build all these different kinds of systems to produce like a baby or whatever it is they happen to be testing for. If you want to test elderly care, you can just make a bigger ball bearing, rotate it slower, I don't know. But they have lots of fun robots there. And they took it even further as well. Continuous delivery wasn't just on the software, it was on the ASICs that they're producing, the hardware chips that they're taping out and silicon labs, right? They make a change to the RTL, the definition, if you like the source code for their ASIC. They do some firmware builds, they do some system testing, they do synthesis, does this ASIC still synthesize, do all the tools work, can we do root and place? And then we do the final system test and if that's good, then we know we have something still on track. And so you can go really far with this stuff, even outside of the realm of software. And this is where hardware, software, co-design can become very interesting. Like I said, in marine seismic, we have very similar systems, right? Systems could fail in all kinds of strange ways. When you have 100,000 sensors and all kinds of physical equipment that you're dragging in the sea, you know, a shark could come and take a bite out of it thinking it was some food, all kinds of fun stuff. But one of the most interesting things, like in terms of IoT, I think that is applicable, is the idea of FMEA. Does anyone know what FMEA is? So it's failure mode effects analysis, I believe. Thank you. And the idea is what you do is you sit down and you look at the system and you look at the whole picture of the system about all the things, ways it interacts with the world, all the different ways it could fail. And then you say, well, if that failed, what would happen? And if that failed, what would happen? And if that failed, what would happen? And you get a list of all the outcomes from different failures and try to look for complex failure modes. And then from that, you have to figure out, okay, well, how do we mitigate those different kinds of failures? And it's a very systematic approach. We say, okay, well, these ones, we can test them, actually. We can add test systems around this. These ones, we can add tests in production so that if something happens while the software is running, we can recover from this, we can try and do that, we can set this kind of state in the software. So if you are building a complex IoT distributed system, I recommend you take a look at FMEA. That's about really it. I just want to finish with one story, so I think every talk so far today has had a slide with Tesla in it, so I didn't want to be left out. And this is the story of the hill start. So there is an idea of what is the purpose of being able to go from one year to one hour in your delivery cycle. And the purpose is to respond to your customers. Now, a couple of years ago, Tesla, they have the product forums on their website, and one of their customers wrote in and said, I'm really annoyed when I'm at the hill and there's a traffic light and it's a red light, I'm sitting there with my foot on the brake, I take my foot off the brake and put my foot on the accelerator when the light goes green, and the car slips back a little bit before it starts going forward. This is ridiculous. This is a luxury car. This should not happen. Other cars have got a hill start feature. Why doesn't the Tesla? So I guess someone in product marketing agreed with this guy, and the product teams started building the feature. They tested it, I hope. And then the software was ready to be released. It went over the air. The Tesla's automatically upgrade over the air. They're firmware. And then the customer, one day, got into his car in the morning, switched it on and said, the dashboard on the Tesla said, good morning. Your software has been updated overnight. We've run through the system test. You're good to go. By the way, here's a list of the new features that you have now, and one of them would be hill start. And if you were that customer complaining on the customer forums, you would feel like static, right? And that's the purpose. Now, I think in the last talk, somebody mentioned about the Tesla hack. Some students have managed to hack Tesla. And this is the flip side to being able to be responsive to change and be responsive to the environment is that when the researchers found this flaw, they went to Tesla. And the researchers praised Tesla for the way they handled the problem. And Tesla gave them a reward. They fixed the software. They released it automatically over the net, and everybody got this new fixed software. So that security flaw was fixed for everyone. Now, other companies also find security flaws, and they don't fare so well. Like the Fiat Chrysler had to recall 1.4 million vehicles because the hackers found vulnerabilities. And it's like the sniper, the rifle sighting, when that was hacked, they wanted to send USB sticks to the customers. That's not a solution to the problem. So this continuous delivery can actually help both in the security side and in the market penetration side of the story. That's all. Does anyone have any questions? Okay, well, thank you.
 Hi, everyone. My name is Mike. I'm from Pragma, a continuous delivery consultant. This is  Internet of Things Day, so I want to talk about integrating all the things. My background  is in embedded systems, large-scale embedded systems for the oil and gas industry. So I  spent more or less 10 years building these big industrial things that you use to find  and drill for oil and gas. But since then, I've been focusing more and more on making  teams develop products faster and the tools and practices around that. And that's what  I want to talk about, the intersection of these two things, how to make products that  are physical, integrated, Internet of Things devices, and how to do it efficiently. And  look at some of the, so I would say the ecosystem around all this. So our mission, should we  choose to accept it, I want to show that there is some real challenges, obviously, in developing  reliable IoT products. There's been a lot of talk about security today. And I think  this is a big concern we all have. And I think one aspect of security is one aspect of the  kind of quality we'd like to build into these products. So we're going to explore the techniques  that we can apply to the problem and look at case studies for customers, for companies  that are already working on this stuff. I'm, like I said, an embedded software person with  a flavor for DevOps these days. I'm also a trainer. I do give courses in Git and Jenkins  and TDD. And I'm a certified Docker trainer for what that's worth. And I also run a business  here in Oslo that helps companies adopt continuous delivery. So how many people here consider  themselves embedded software engineers? Okay, there's a few. So for me, when, I don't know  if you guys are the same, but when I first heard the phrase IoT, I kind of rolled my  eyes. Actually, I felt like I've heard this all before. We had this 10 years ago when  it was called M2M. And it also feels like, in a way, DevOps to me does. It's a cute phrase  that's fun to say. But it just gives a name to something that's been around for a very  long time. Whatever way you think about it, IoT has always been there. We've always made  embedded devices with internet connectivity as long as there has been internet connectivity.  On the other hand, it is a really fun space. And despite the fact that it's got a funny  name, IoT is actually a ballooning industry. There's more and more things that we're connecting  to the internet. This is why we have IPv6 in the first place. So we're going to have  an explosion of smart devices. And this is fun time to be a technologist. So I'll try  and not to be a grumpy old man about IoT. And I'll just embrace it today. So I don't  want to describe IoT, but briefly I just want to look at the attributes about IoT, the IoT,  and how they were, these will play an important part to how we can build these systems. So  the most important thing about IoT is that they're the T, actually, the things, they're  physical things that happen to be part of our world. There's hardware and software co-design  and they interact with our environment. They are either sensing and monitoring the environment  or they're actually actuating on the environment that we work in. They're also distributed  in nature, connected via networks. And often, although they can often work in coordination,  they're usually independent actors focused on one specific need. And the idea with these  IoT things is to add some intelligence to otherwise dumb parts of our world. It's not  just fridges. So this is another way to describe what IoT is, intelligent, distributed, physical  systems. Now, for those of us that come from a software background, actually all those  words kind of terrify us because we know that distributed systems are hard. We know that  making intelligent systems is hard. We know that making physical systems is hard. So that's  what I want to talk about today. This is a hard thing about hard things. What do we do  when we're faced with these very difficult problems? There is complexity in this problem.  There's complexity in distributed computing, of course. This has been known for a long  time. The guys at Sun, microsystems always knew this. This was their domain, actually.  Their problem space was the network computer. And there's a list of fallacies that everybody  working with distributed computers or designing distributed systems kind of fall prey to.  We design these systems thinking that the network is going to be reliable, latency is  going to be zero, bandwidth is going to be infinite, blah, blah, blah. We just can't  think of all the ways that these things can go wrong in reality. But that's not, and that's  part of IoT, right? They're distributed systems, but they're not just distributed, they're  physical systems. So there's a lot of fallacies that people making embedded systems, hardware  and physical systems fall prey to as well, such as the hardware always works. The environment  never changes. The resources are endless. We're never going to run out of memory. The  sensor is always going to be calibrated. The data we're reading from our sensor across  three time zones is definitely live. It's not just returning the same thing over and  over again. We're assuming all kinds of things. The clock is in sync. The data it's giving  us is actually at the right time that we believe it to be. And then, of course, we have the  more life cycle issues. How do we keep this thing on? Is the power reliable? What happens  when we lose power? Is there a safe time to lose power? And what happens when it comes  back on again? How do we keep these systems up to date? And actually today, most of the  talks that we've gone, that I've been to, I've been security focused in some way. And  Hans Alpou was in the rifle talk. Right. This was a really cool talk about how to hack  a Linux powered rifle. The interesting thing is when the flaw was discovered and it was  published, the manufacturer said that they would send out an update on a USB stick to  the however many thousand of people that have bought the product. That's not a really scalable  way to update your system. We need systems to be up to date. And as we connect them to  the network and we bring them into our home, we bring them into our body, we let them drive  our cars. It's of paramount importance that we keep them up to date. And then of course,  there's the fallacy of intelligent systems. There is the idea that we'll always, I mean,  we're smart enough to figure out how to do this thing. And that's never true. Complex  systems never work. It's been known ever since we've had systems, actually. Complex systems  never work. The first time you do anything, it will never work. John Goll has written  a lot about this, but so has Jerry Weinberg. There's a lot of good information. Obviously,  the first thing, the first time you do something, it never works. But that's why you should  start small. All the best systems start from small simple things that work. This sounds  pretty familiar, right? And this is the way we've approached software in the last 15  years as well. We don't want to design a big complex system. We want to start with something  small. It's right to make it bigger and bigger, more and more features, do more and more stuff.  This is how we figured out how to actually make software. We realized that we couldn't  do complex things. We couldn't design it up front. For me, there's two very important  principles in NAHGEL manifesto that come into play here. The one is that obviously, we want  to start early, give value continuously. And that working software is the primary measure  of progress. So without these, those two are actually another way of saying this. How do  you start from a small system and grow it into something that's complex and still have  it work? And it's these two principles that are the enabler for that. And for me, coming  from a continuous delivery point of view and continuous integration point of view, these  are the two principles that we try and solve with continuous delivery. So I'm not going  to go through, this is just a fond cartoon, but I'm not going to go through all of this,  but the point is that there is a system in place that we've decided in software to make  sure that if you like continuous delivery is the technical implementation of those agile  principles, how do we make sure that we can continuous delivery deliver valuable software  that works? How do we make sure that the software already works? Well, we have continuous integration,  which is the gray box. We say that once the software builds and it runs any kind of simple  test, then it's good enough to share with my colleagues. That means I'm not going to  break it for anyone else. But that's not to say that after that point, it's good enough  to ship, it's good enough to put in your car, it's good enough to put in your pacemaker.  After that, you need what's called the continuous delivery pipeline, where any other steps in  your process that need to happen before the code can be considered potentially shipable  has to happen. And obviously, I would promote as much of this to be automated, but some  of this stuff can take a really long time. For instance, with the trucks at Volvo, they  have to go through many, many steps before you could say that this is actually good enough  to go into a truck. But still, there is a pipeline, there is a process, and there is  a way for always working software to be ready for the customer. That's all it is. Continuous  delivery is just the implementation of those principles. So I want to go back to the things  and the Internet of Things, because for me, that's the most interesting part. The Internet  isn't interesting, it's just a network. But the things that we attach to it are actually  the important thing. I'm going to contrast that by looking at a project that was actually  my very first project after graduating, and look at how we made software then and how  we make software now and see the differences. So I was young, I got a job after university  to join a company that was making products for the oil and gas industry. And my first  project was to make a rotary steerable drilling tool, which is basically a long tool that  has some pads on it on the side and then a drill bit. And then when you're drilling  for oil, what you do is you put this down into the well bore and you spin it around.  You have a long pipe all the way up to the top, to the rig, and you spin it. Spin it  really fast, like 200 times a minute. And while this thing is going around, you want  to spin it. You want to steer it. So you've got gyros, you've got accelerometers, you've  got shock measurements, all these things. And in real time, you want to try and drill  a path towards where the driller wants the drill bit to go. So this was quite a project  for someone straight out of university who never programmed C before. I was determined  to make a go of it and I wanted to learn as much as I could. So I wrote down all the steps  that were necessary in this professional world of software development. So I found out the  first step you do when starting a software project is you take a copy of the code base  for the existing product. So we're going to make a new tool. Well, there's an old tool  already. Let's just copy the code base for that product. We'll start from there. And  then what you do is you spend a few months documenting its current algorithms. This is  important because nobody knows how this code works. The person that actually wrote the  control systems long gone. And it was done four or five years ago. So it's going to take  some time to figure this out. You might need to have a systems engineer around for the  math. Then what you want to do is be very careful about the changes you make to this  source code because you know what kind of sort of works. So any changes you make, surround  them by FDF so that you can always switch them off again if it doesn't work out. Then  the most important and the longest part of this development process is testing. So what  you want to do is you want to test it on your own hardware on your desk. You want to put  it into a tool and test it down in the shop. You want to put mud through it and power up  and watch it steer in an automated flow loop. And you want to go to Oklahoma, you want to  go to Siberia, you want to test it on the rigs. Take as much time as you like for testing.  And then finally when you're happy with the results, everybody's happy. You commit that  code as a new project in the version control system. And you hope someone's going to take  care of merging it into the previous project at some point in the future. Then you compile  it on your laptop and give it to manufacturing. That's how we do professional software development.  At least that's what I learned. I knew it was professional because we had a software  development process and I followed it to the letter and it was part of the product development  process and part of the project governance approach. So we were ISO 9000 certified. So  we were very professional. I mean, I am being ironic though, but this project was actually  considered to be very successful. It was very successful. The product worked, it met its  commercial goals. And that was largely due to this step four where it was lots and lots  and lots of testing. So there's two sides to this. This can work in that kind of context.  But if we move towards the modern times, this type of software development doesn't work.  And I want to contrast this kind of project to the way we do embedded systems now. So  ideally, instead of having this as your professional, as your software development process, you  would have something more like this. As a developer, what you do is you fetch the latest  source code, which is up to date with whatever is released, what your customers are using.  You branch, you implement your feature, you test it on your machine. When you're happy  with it, you think it's good enough, you appreciate it to the central repository. The continuous  integration system kicks off. It starts building your system, building your code, doing any  unit tests you have, maybe any quick smoke tests you have. And if they're good, your  code should get merged into the master branch. Then that should kick off a continuous delivery  pipeline, which will do the static analysis, program your code onto a device, verify the  device and the hardware and the software work together in an environment. Maybe you have  a lab setup. You might even do canary deployment, where you deploy this to some customers and  check that they're still happy and there's no complaints before you send it to all the  customers. And then you want to get some real-time production feedback for those things.  Now, I don't think there's anything too controversial here. This is the way most software is developed  in the world. This is the way we should develop. This is if you go to any newspaper or broadcast  or this is anybody with a website, really, this is their approach. So let's take a look  at the difference between a project like this and a project like this. So back then, embedded  systems were versions like this. You just put them in a new folder and give it a different  name. If you are really, really conscious that you might make a backup every week and  give it a name of the date. That was how you did product versioning. But now we have distributed  version control systems. We have branching strategies that actually make sense and work  so that we can always work with working code as a team. Now, I don't want to talk about  branching strategies, but I want to talk about this. If you're working in embedded systems,  this is a great book. It tells a practical approach to large scale agile development.  It doesn't sound very interesting. I know. But it tells the story of the laser jet business  from Hewlett Packard. The embedded systems for those, there was 800 people spread across  five different continents and they had really tough legacy software issues. The book tells  the journey of them transitioning to an agile way of working. They had a different branch  for every product and they were trying to maintain patches and branches across all these  hundreds of different laser jet models. They had, I think, almost 100 people whose job  it was just to merge and patch and build their software. You can imagine how slow and how  unresponsive the system eventually becomes. One of the key learning points is just this.  Actually having a release-trained branching model, branching strategy can hugely improve  your development speed. Of course, now we have better ways of versioning as well. We  don't just take the subversion number or the git sha. We have a way to put some semantic  meaning on our versions as well from the products we built. We can tell consumers whether this  is going to break, if it's a breaking change, if it's just a new feature, if it's a bug  fix. You can figure out how you want to change your dependencies. Of course, we can stamp  this information right into the products. This is important. Actually, Novella is a  company that I work with. What they want to do is they want to, from one of their radars,  be able to find out exactly what version of software is in that radar. From all the different  components, all the different binaries, what versions they were and which git sha they  were related to, which build information where I can find them in the artifact management  system, you can put that right into the binary. You can ask the radar itself all this information.  It's just simple stuff. You make a version.h and a build.h like we've always done. You  have the system generated for you. Also, one of the things that have really changed between  back when I started and now is that product families are really different. When I was  starting systems, you had the old systems, which were maintained by what was called the  sustaining organization and the new products, which were made by the engineering department.  It was really interesting. Today, we talk about DevOps, about bringing operations and  developers together. In a lot of engineering organizations, it's even worse than that.  There's a support function and a development function, and they don't even talk to each  other. The people that are making the stuff are not the people that are supporting the  stuff, so they never get that feedback from the customers. That was very much the case.  I think it still is in some places. Now, we don't have the old one and the new one. We  have the family. We want to sell products. We want customization. Like Volvo, every single  truck they make is different, but they have to use the same stuff. They have to use the  same software, the same tools, the same tests. That only works if you can reuse. Of course,  back then, I was building an ID. I don't think anyone does this now, but this was how we  cut a release. We opened our laptop and we pressed build. I think that this blog post  from Jeff Atwood really changed the way we approached this. It's crazy. It was only six  or seven years ago where he said you should have a script for your build instead of using  the ID, and it was considered like news. I don't think anyone would release software  like this now. The thing that was missing, of course, is the traceability to figure out  how was this thing made? Should I be worried? Where did it come from? How do I make it again?  How do I reproduce this? Can I get the same results? What we do is we put this in, we  create a Jenkins system, a server that can do all this. The idea is that, while at least  the server we should know a little bit about, if it's a developer laptop, we don't know  so much. We start putting all our configuration in this Jenkins user interface. I don't think  that's a very good solution either because who knows, anyone can change that. Nobody can  track it. The traceability and reproducibility of using Jenkins isn't really there. It has  a lot of other benefits, but you won't solve those problems with using Jenkins. What you  need to do is instead define your builds as code in some way. Find some kind of text  format that you can version alongside your software so that you can track exactly how  something was built in the past. You want to check out a version of the code from a  year ago. You want to know exactly how the build server was set up then. This is job  DSL. There's a lot of different ways to do this, but this is one way. You don't even  have to stop there. You can go way beyond that and look at the whole ecosystem of tools  you use and figure out what would be a traceability ecosystem for building these products. How  could I use Artifactory and configuration as code tools like Docker and Vagrant and Puppet  to actually be able to reproduce reproducible development environments from code that is  version controlled. Maybe this seems like going a bit too far, but for a lot of, especially  safety critical systems, this is the only way you can really control how something was  built, which header files were included in the build, what was the library path when  something was built. These are all important considerations and you can only do that when  you have control over the entire environment. This is something also, for me, interesting  about Docker because I was at DockerCon and there was maybe 100 presentations there this  year and they were all nearly from web companies. All the places that I'm implementing, well,  say 99% of the places that I'm implementing Docker solutions, they're embedded safety critical  places where they want to be able to say, I want to rebuild this ASIC. I want to know  exactly that this same ASIC is going to be built or I'm going to put some code that controls  the breaking system of our truck. I want to be able to know exactly how it was built.  There's a lot of different uses for Docker, not just for creating microservice websites.  Then of course, if you have this configuration code, you can have Phoenix builds. You can  basically spin up and spin down build systems as you please. They're no longer special dedicated  build servers. You can use the cloud to rip them up and tear them down and create new  ones as you please and the developers can have access to the same environments.  One of the fun things about that rotary steerable drilling tool that I was telling you about  is that the only way to update the software was to send a zip file to basically a workshop,  a mechanical workshop, and people would maybe take the tool on a helicopter from the rig  to the shop and the mechanics would open it up. Then they would attach a JTAG to it and  flash the thing. That was the only way to update it. Now, we have a lot of systems where  they just get updated automatically over the air. The latest version of the software is  always available. Now, this is a really hard thing to get right with the IoT devices because  there's so much that can go wrong when an update occurs. We get used to over the air  software updates with phones and so on, but actually to implement this on an IoT device,  if you make it yourself, it's very, very difficult. It can take a year of development to be able  to have a reliable update system that you know that no matter what kind of failure mode,  you won't break the device. Now, that's all well and good. This is how we build the software.  I think the only way that we can actually have this continuous delivery of working software,  the actualization of that is only if we take testing seriously. I see a lot of companies  that really focus on automation, but it's always around build. Their testing is very,  very weak. The testing is the only thing that makes sure what you're delivering is actually  any good. There's no sense in delivering crap really fast. Anyone can do that.  So there's a lot of discussion like how does testing fit into IoT embedded, all these kind  of product kind of devices, like even if you're making cars, what does testing look like? This  doesn't really fit. We don't really think about UIs and services. Obviously, we have  units, but we do have a testing pyramid, but it's not quite the same. Of course, we need  this idea of a system test, a full level system test, integration end to end, full test. This  is the only way you make sure everything is connected. In IoT devices, this usually requires  a lot of programming actually. You need to spend a lot of time to automate this because  unlike unit tests where you just use a test framework for your language and start writing  tests, system tests for embedded and IoT devices actually require a lot more. They're usually  very custom things. You need to be able to control the power to the device, turn it on,  turn it off. You need to be able to program it with new code, like access its flash. If  it is a device that engages with the environment in some way, changes the environment in some  way, you need to control those actuators through some kind of API. And you also need to be  able to sense the results. One system I worked on was a steering system for marine seismic  acquisition. What that was is a big cable you drag through the sea and you steer it around.  They want to make sure it's at the right depth and it's in the right place. How do you test  something like that? You can send it lots of commands and you hope that what it's doing  is correct. But what you need to do is sense what it's doing physically in the world. You  want to see that actually the blade is at this angle. So you need to have some kind  of watch guardian over these systems to see if they are interacting with the environment  that they interact correctly. We don't have services, but we do have components. We can  test things in isolation and we should aim to test things in isolation as much as we  can. The idea of microservices is just as equally applicable to IoT and is to figure  out what is the way to isolate the change and find an abound in context for these devices.  And of course, unit tests are great. We should have them everywhere and make development super  fast. I get a lot of pushback actually from management when I show this slide because  they say, well, if I'm writing all these end-to-end system tests, why do I need the unit tests?  And if I'm writing all these unit tests, why do I need the end-to-end system tests? I didn't  really have a good answer for that until I read this book, Growing Object Oriented Software  Guided by Tests, which is really good. It doesn't really say much about IoT, but what  it does tell you is different types of tests give you different types of feedback. And  the important thing to know is unit tests give you a lot of feedback about your internal  quality. They tell you that the code is easy to change, is probably well understood, that  it works how the developer expected. It doesn't tell you anything about your external, about  your, where are we? It doesn't tell you anything about your external quality. How much this,  together as a system, meets the requirements of the customer. And the same is true for  the end-to-end tests. The end-to-end tests tell you very little about how easy the code  is to change, how safe it is to change, but it tells you an awful lot about whether the  system as a whole meets the needs. Clear? No. So with IoT, it adds an extra level to  this because what is a system test in the context of IoT? These IoT systems are distributed  in a sense. When you say end-to-end, if your intelligent light bulb connects to a cloud  service which connects to your phone so that you can control the hue or the color of this  light bulb, how do you do an end-to-end system test? Do you include the whole internet? Do  you include deploying a web service to do this? This is a really hard problem to solve.  And you have to solve it because it's the complex nature of the interaction of all those  things together that will produce your failure modes. Another important thing is that there  has to be access to hardware. The developers and the testers and the customers, they all  need access to this hardware so that you can make sure that what you're running in the  lab is the same thing that the customers are using and is the same thing that you have  on your laptop. So the goal is, I wrote this slide yesterday, the goal is to go from one-year  product cycles to one-hour product cycles, the difference between using continuous delivery  and the old way of doing software. And I thought that that was quite a gimmicky sales pitch  thing to say. But then when Peter talked about Volvo, he talked about going from, I think,  46 weeks to 10 minutes. So it is actually, this was maybe a little bit underambitious.  When you can get development cycles from one year to one hour, it really changes the way  you can deliver product to customers and change your plans. Another interesting aspect of  this is the Novella story. Now, Novella make this radar trip, it's very clever, it can  do all kinds of things. It can detect your breathing, it can monitor your breathing,  it can know whether you're sleeping, it can tell us how many people are in this room,  whether they're dead or alive, for instance, small details like that. And they take continuous  delivery very seriously as well. But they have a problem. When they make a radar that's  in right signal processing that's supposed to monitor whether a baby is breathing or  not, how do you test that? Actually, one of the first months I was there, I actually called  up some friends that were on mama permission and they came with their babies and we put  the radar on them. That was fun because we had to tell these mothers we're going to put  radars on your babies. Is that okay? Surprisingly, they said yes. That's some trust. But that's  not reproducible, right? Even if we collect all that data and run it over and over again,  that's not a good test. It's maybe a good acceptance test, it's maybe a good product  demonstration, but it's no way to run your development process because between the time  that test is happening and the time you deliver your product, you might have made a thousand  or two thousand commits all changing the behavior of your code. So the guys at Navelda were  very clever and what they realized was that if they took a ball bearing and rotated it  at a certain frequency to the radar, it looks just like the chest cavity of a baby. You  get basically the same information. And because LEGO is more or less invisible to radar, it  doesn't have a very high reflection, this was a great way to, this was a great solution.  What they did was they set up a lab with all this LEGO. There's people at Navelda that  get paid to play with LEGO and they build all these different kinds of systems to produce  like a baby or whatever it is they happen to be testing for. If you want to test elderly  care, you can just make a bigger ball bearing, rotate it slower, I don't know. But they have  lots of fun robots there. And they took it even further as well. Continuous delivery  wasn't just on the software, it was on the ASICs that they're producing, the hardware  chips that they're taping out and silicon labs, right? They make a change to the RTL,  the definition, if you like the source code for their ASIC. They do some firmware builds,  they do some system testing, they do synthesis, does this ASIC still synthesize, do all the  tools work, can we do root and place? And then we do the final system test and if that's  good, then we know we have something still on track. And so you can go really far with  this stuff, even outside of the realm of software. And this is where hardware, software, co-design  can become very interesting. Like I said, in marine seismic, we have very similar systems,  right? Systems could fail in all kinds of strange ways. When you have 100,000 sensors  and all kinds of physical equipment that you're dragging in the sea, you know, a shark could  come and take a bite out of it thinking it was some food, all kinds of fun stuff. But  one of the most interesting things, like in terms of IoT, I think that is applicable,  is the idea of FMEA. Does anyone know what FMEA is? So it's failure mode effects analysis,  I believe. Thank you. And the idea is what you do is you sit down and you look at the  system and you look at the whole picture of the system about all the things, ways it interacts  with the world, all the different ways it could fail. And then you say, well, if that  failed, what would happen? And if that failed, what would happen? And if that failed, what  would happen? And you get a list of all the outcomes from different failures and try to  look for complex failure modes. And then from that, you have to figure out, okay, well,  how do we mitigate those different kinds of failures? And it's a very systematic approach.  We say, okay, well, these ones, we can test them, actually. We can add test systems around  this. These ones, we can add tests in production so that if something happens while the software  is running, we can recover from this, we can try and do that, we can set this kind of state  in the software. So if you are building a complex IoT distributed system, I recommend  you take a look at FMEA. That's about really it. I just want to finish with one story,  so I think every talk so far today has had a slide with Tesla in it, so I didn't want  to be left out. And this is the story of the hill start. So there is an idea of what is  the purpose of being able to go from one year to one hour in your delivery cycle. And the  purpose is to respond to your customers. Now, a couple of years ago, Tesla, they have the  product forums on their website, and one of their customers wrote in and said, I'm really  annoyed when I'm at the hill and there's a traffic light and it's a red light, I'm  sitting there with my foot on the brake, I take my foot off the brake and put my foot  on the accelerator when the light goes green, and the car slips back a little bit before  it starts going forward. This is ridiculous. This is a luxury car. This should not happen.  Other cars have got a hill start feature. Why doesn't the Tesla? So I guess someone in  product marketing agreed with this guy, and the product teams started building the feature.  They tested it, I hope. And then the software was ready to be released. It went over the  air. The Tesla's automatically upgrade over the air. They're firmware. And then the customer,  one day, got into his car in the morning, switched it on and said, the dashboard on  the Tesla said, good morning. Your software has been updated overnight. We've run through  the system test. You're good to go. By the way, here's a list of the new features that  you have now, and one of them would be hill start. And if you were that customer complaining  on the customer forums, you would feel like static, right? And that's the purpose. Now,  I think in the last talk, somebody mentioned about the Tesla hack. Some students have managed  to hack Tesla. And this is the flip side to being able to be responsive to change and  be responsive to the environment is that when the researchers found this flaw, they went  to Tesla. And the researchers praised Tesla for the way they handled the problem. And  Tesla gave them a reward. They fixed the software. They released it automatically over the net,  and everybody got this new fixed software. So that security flaw was fixed for everyone.  Now, other companies also find security flaws, and they don't fare so well. Like the Fiat  Chrysler had to recall 1.4 million vehicles because the hackers found vulnerabilities.  And it's like the sniper, the rifle sighting, when that was hacked, they wanted to send  USB sticks to the customers. That's not a solution to the problem. So this continuous  delivery can actually help both in the security side and in the market penetration side of  the story. That's all. Does anyone have any questions? Okay, well, thank you.
Integrate_all_the_Things__-_Mike_Long_0_107.png Integrate_all_the_Things__-_Mike_Long_108_179.png Integrate_all_the_Things__-_Mike_Long_180_275.png Integrate_all_the_Things__-_Mike_Long_276_407.png Integrate_all_the_Things__-_Mike_Long_408_455.png Integrate_all_the_Things__-_Mike_Long_456_503.png Integrate_all_the_Things__-_Mike_Long_504_1415.png Integrate_all_the_Things__-_Mike_Long_1416_2207.png Integrate_all_the_Things__-_Mike_Long_2208_3887.png Integrate_all_the_Things__-_Mike_Long_3888_5519.png Integrate_all_the_Things__-_Mike_Long_5520_6191.png Integrate_all_the_Things__-_Mike_Long_6192_7055.png Integrate_all_the_Things__-_Mike_Long_7056_8159.png Integrate_all_the_Things__-_Mike_Long_8160_10883.png Integrate_all_the_Things__-_Mike_Long_10884_11267.png Integrate_all_the_Things__-_Mike_Long_11268_11915.png Integrate_all_the_Things__-_Mike_Long_11916_11963.png Integrate_all_the_Things__-_Mike_Long_11964_13079.png Integrate_all_the_Things__-_Mike_Long_13080_13259.png Integrate_all_the_Things__-_Mike_Long_13260_13655.png Integrate_all_the_Things__-_Mike_Long_13656_15911.png Integrate_all_the_Things__-_Mike_Long_15912_20771.png Integrate_all_the_Things__-_Mike_Long_20772_24923.png Integrate_all_the_Things__-_Mike_Long_24924_24971.png Integrate_all_the_Things__-_Mike_Long_24972_25055.png Integrate_all_the_Things__-_Mike_Long_25056_25415.png Integrate_all_the_Things__-_Mike_Long_25416_28307.png Integrate_all_the_Things__-_Mike_Long_28308_29339.png Integrate_all_the_Things__-_Mike_Long_29340_29495.png Integrate_all_the_Things__-_Mike_Long_29496_29567.png Integrate_all_the_Things__-_Mike_Long_29568_30899.png Integrate_all_the_Things__-_Mike_Long_30900_31463.png Integrate_all_the_Things__-_Mike_Long_31464_32375.png Integrate_all_the_Things__-_Mike_Long_32376_32735.png Integrate_all_the_Things__-_Mike_Long_32736_33671.png Integrate_all_the_Things__-_Mike_Long_33672_34403.png Integrate_all_the_Things__-_Mike_Long_34404_36827.png Integrate_all_the_Things__-_Mike_Long_36828_37415.png Integrate_all_the_Things__-_Mike_Long_37416_38291.png Integrate_all_the_Things__-_Mike_Long_38292_39359.png Integrate_all_the_Things__-_Mike_Long_39360_40211.png Integrate_all_the_Things__-_Mike_Long_40212_44207.png Integrate_all_the_Things__-_Mike_Long_44208_44291.png Integrate_all_the_Things__-_Mike_Long_44292_44639.png Integrate_all_the_Things__-_Mike_Long_44640_47855.png Integrate_all_the_Things__-_Mike_Long_47856_48371.png Integrate_all_the_Things__-_Mike_Long_48372_49487.png Integrate_all_the_Things__-_Mike_Long_49488_50135.png Integrate_all_the_Things__-_Mike_Long_50136_52883.png Integrate_all_the_Things__-_Mike_Long_52884_54071.png Integrate_all_the_Things__-_Mike_Long_54072_56387.png Integrate_all_the_Things__-_Mike_Long_56388_56531.png Integrate_all_the_Things__-_Mike_Long_56532_60923.png Integrate_all_the_Things__-_Mike_Long_60924_61151.png Integrate_all_the_Things__-_Mike_Long_61152_61199.png Integrate_all_the_Things__-_Mike_Long_61200_61367.png Integrate_all_the_Things__-_Mike_Long_61368_61379.png