DevOps: More than Marketing
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Turnbull, James
DevOps is a new concept that has emerged in the Development and Operations community. It calls for closer relations between developers and operations people with a view toward improving the quality of the services we provide. I'll introduce the tenants of DevOps: * Simplicity * Relationships * Process, automation, tooling * Continuous improvement I'll also focus on automation and tooling and especially some of the more interesting open source projects (and people!) that make DevOps possible: configuration management tools, monitoring, metrics management and visualisation. Finally, I'll cover some of the critiques, criticisms and FUD throw at DevOps and why the concepts are more than just marketing.
 Thank you very much. Thank you very much. Thank you. Good afternoon. Thanks for coming along. It's a very, very big room. It's a lot bigger room than I'm used to speaking in. It's not a little bit about me. I've been involved in the open source community for about 20 years now. Since I was a teenager and discovered this Linux thing, it was a bit cooler than that Windows thing. I work at puppet labs. I run the operations and professional services group there. I'm an author. I've written five technical books. Mostly about Linux. This had been an open source set of things. I'm a big geek. My wife would say very much a big geek. I'm known to buy every single gadget that appears on the market. I usually buy two of them and break them fairly quickly. I should warn you, I do talk funny. I'm from Australia. Pretty much every other part of the English speaking world doesn't believe we speak English. If you find anything you can't understand or find that I'm talking very quickly, wave your hands frantically in the air and go, please slow down. Okay. Not a good sign. I'm also a winchig bastard. I've spent a lot of years in operations working in some big shops. I'm well known for complaining about things. I don't like things that are broken. I'm a firm believer in perfection. I don't think you can achieve perfection, but you can get a bloody well of a long way towards it. So what's DevOps all about? This is a topic that comes up a fair bit. I don't think there's a clear answer to it yet. DevOps is about cooperation between development teams and operations teams. Particularly important as to why Belgium is so important for DevOps is because the gentleman who coined the phrase DevOps, Patrick Dubois, is obviously from Belgium. Chris and I'm going to stuff his surname up, Bay Art, also heavily involved in the original DevOps movement. The original DevOps thinking emerged out of a bunch of people in Belgium sitting around drinking beer, which may have something to do with it, while they felt the love for one another. Generally, the whole movement emerged out of a bunch of northern European sysadmins. So what comes up when you start to look at DevOps? I did a little bit of a keyword cloud, and I'm afraid the top bit's cut off there. There's some of the phrases that come up when you look at various blog posts around DevOps. I look at these sort of things and I go, well, what does this become? Is this some kind of buzzword bingo where after a few phrases, we'll get to sit around together and go, we can talk about DevOps and cooperation and ITIL and we can talk about cross-functional teams, and then somebody can sit in their meeting and go, bingo, we've managed to parrot out all of the appropriate corporate phrases and phraseology. Is it a pop culture movement? There will be a bunch of fly-by-night guys who are just trying to con you into buying something. I've seen a couple of products so far that claim they're DevOps compliant. I'm not sure who they were certified by, but if they buy me beer, I'm happy to. Or bourbon even better. I don't think we're a pop culture movement. I also don't think that we're going away in a hurry, but I do have some caveats before I talk a bit further. It's very early days. DevOps is probably about two years old in terms of its thinking, and it's probably only about six months old in terms of people actually, I guess, more theoretically considering all of the possibilities that DevOps implies. Largely speaking, the first conversations about DevOps happened about two years ago. I think last year they had a DevOps dinner at FOSDEM, and I believe there was about 10 or 15 people there. Last night they had the same thing again, and there were 60 people. So it's early days. No one has all the answers. I don't think anyone's suggesting that it's a fixed movement. We certainly don't have a manifesto or a style guide that says, this is how you recognize it's DevOps, it's painted green, and it's got little stripes. That's not the case. Nothing's fixed in stone. I'm certainly not advocating a position that there's some sort of doctrinal approach that we need to take. And at the moment, for me, this is all about outreach. It's trying to get a message out about what DevOps is and about the community around DevOps. And you may notice a little footer on my thing, all opinions of my own, maybe subject to change without alcohol. I think a lot of the DevOps movement has been about discussion, about people going, well, DevOps is a bit of this, it's a bit of that, and let's see how this all fits together. For me, I like lists and DevOps for me is about four things. Simplicity, relationships, process, and continuous improvement. And I'm going to talk a little bit about each of those areas and why they're sort of important to me and why I think they're an important part of DevOps and why I think that makes DevOps useful to both operations and development communities. Firstly, simplicity. So we all work in, you know, if you work with IT, then IT is complex. You know, there's simple things, some of the simple things, you know, that in an IT shop, you know, how you build your application stack, what particular operating system you choose can cause a massive amount of debate. I'm not even going to talk about things like, if I sat down and said to you, Vim is the best editor in the world, then all the EMACs people would be stoning me. And it's not an uncommon phenomenon in the IT world to have huge, complex applications linked together based on choices that you have to inherit from somewhere, legacy applications. And in order to manage those environments, simplicity is the key. So you need to choose things that are simple and easy to manage. Those things need to be repeatable and reusable. It's particularly in environments where you're managing thousands of hosts. You cannot do something that is a one-off thing anymore. You can no longer get away with saying, all of my systems are unique snowflakes and they're all different and they need a different set of rules. You simply would do not have the bandwidth or the time to manage a thousand systems that you are considering to be unique snowflakes. And simplicity implies easy to communicate. Whatever message you need to communicate about your systems, whether it be their state, whether it be their configuration, whether it be, you know, how to recover them or back them up or secure them, it needs to be something that's simple and easy to communicate. The key for me around this is that if you're sitting in, if you're sitting in the end, at the end of the day, you're sitting in a room full of people trying to fix a bug or doing an outage. You want to have the best possible conversation with somebody where the end result is that you solve the problem as fast as possible and you solve the problem in such a way that it never reappears again. And if your environment isn't simple and repeatable, then you're going to have a lot of struggle with that. The next thing for me is relationships. So we all work in an industry that's well known for the fact that we're sort of, supposedly misanthropic, we're supposedly this bastard operator from hell types, we've just had been types, and we're the ones, you pick up the phone and call them and you go, they're going to blame me for breaking something or the dev team is going to actually, you know, he's going to cry out and say, here's the guys that, you know, they can't install our latest version of the application and they hate us and, you know, we cause them all their problems. But every single thing that we do in the IT world is built around a relationship with other people. Too many of the conversations I've had over the years in IT have consisted of, you know, what I like to call the blame storm. So the blame storm is where instead of ascertaining what the actual problem is, the first thing you do is work out who can I blame for the problem and how can I make sure it's not my fault. And there's a famous American phrase, cover your ass. And that's an expression that gets quite used in enterprise IT shops. You're attempting to rather than solve the problem, you're attempting to be the person that is not at fault. The only way to change that mentality, to change that conflict between teams, is to build relationships. And there's a lot of reading you can do about how to build relationships and a lot of it's touchy feely stuff. I now work in America where everyone talks about their feelings all the time. I'm Australian, we don't. We just get drunk and hit things. But generally speaking, it's about engagement and it's about engaging early and often. So if you imagine the life cycle of a project, life cycle project we can see solve, you know, if you're working a business and the business says, we would like to build a product or a service and that product or service is going to talk to a new bunch of customers. And the marketing team, you know, come up with a marketing message and they come up with a business comes up with all the things they want to do. They throw over a sort of, you know, maybe a few scratchings on a cocktail napkin that the marketing guys came up with between strippers and then handed over to a bunch of developers. And the developers go, we can't build this, it's impossible. That's the first reaction almost every time. They next go, okay, we can build this, but it's going to look a bit like this. They then build the application, you know, there's a whole bunch of people sitting around and they go, this is really cool, we should use React here. That no SQL stuff is really cool. And then someone says, oh, wow, we better make sure that it runs on this and we better make sure it runs on the latest version of Ruby. And my God, we can use this really cool library I found and let's install all of these gems and vendor them in here. And all of a sudden, and I'm picking on Ruby people because I'm a Ruby person, but it's true of almost every environment. The next step that happens is that the team, you know, puts together this amazing demo and they demo it for marketing people. Some people say we hate it. They hire a couple of UI people and fix that. And then the application people say we're done. Hand it over, let's toss it over the fence. And the people are handed over to operations people. The operations people take them on look at it and go, oh, my God, how are we going to make this work? It doesn't run on our standard operating system, it requires all these libraries we don't have. How do we back it up? My God, it's syncing data between three data centers across four continents and how do we secure it? The application architecture is completely different from our existing application architecture. And they model through, they spend six weeks trying to work out all of the bits and pieces, they jury rig things, they write scripts, they build systems that, you know, especially when you need systems to stick application, they run it up, first day goes under some load and it falls over. So there's then, you know, we've all sat in that meeting where the system's fallen over, you know, there's 10,000 concurrent connections and the developers say it should support 10,000 concurrent connections, you guys haven't provisioned enough systems, you guys haven't worked out how to handle this problem. And everybody sits around and basically it's an exercise in trying to ascertain who's fault it was, rather than fixing the problem and trying to ascertain, you know, how do we get here and how do we get out of the hole? The key to getting out of that hole is instead of right back at the very start where the business has a particular need, the business says we want to create something. The people they should have gone and have a conversation with are both developers and operations. And if those people are in the same room having that conversation, then all of these little things which are called in IT world, non-functional requirements. So non-functional requirements are little things like security and backing things up. And they're equally as, ironically called non-functional, but equally as important to the process of building an application. So if we start from the beginning where both sides actually have a conversation, actually get involved and talk to one another, then we actually have an opportunity to develop applications and products where everybody gets what they want out of it. So how do we do that? To start to do that, both sides need to have des tentes. And my French accent is terrible. It sounds much better when you actually have a French accent. But des tentes is about both sides need to understand that they're all in the same problem. We're all in the same problem space. We all have the same issues we have to deal with. We all have the same bosses, the customers. We all work for the same people. We all get paid the same way. We need to actually get stuck into a conversation with one another where we actually don't let our prejudices sort of color the conversation. And you have to have that conversation. You can't send someone an email. I find the classic example of someone is wrong on the internet. And the reason someone is usually wrong on the internet is because someone has written an email and someone else has read the email and gone, I interpret it like this, or I speak English as a second language and I've read this email in a particular way. Or I miss the smiley face. You suck at the end of that. And next thing you know, somebody is having a conversation that consists of we're actually aggressively pursuing a point rather than actually having a dialogue. And I can't underestimate you the number of times that I've solved simple problems by getting off my ass, walking halfway across a room, or getting onto a video conference and actually talking to the person involved. And building those relationships is a key part of any of the steps that we need to talk about and DevOps. So the next important thing for me is process. So I spent about watching the IT industry change dramatically over the last sort of 20 years. When I first started, the one system we worked on was a massive big mainframe, partitioned into production and development. There was a DRP mainframe and there was pretty much, we had a system and it did stuff. There was no client server. There were dumb terminals. I started to cut my teeth on 3470 green screens, 3070 green screens. That's changed dramatically. We went from a world where all of a sudden client server appeared. You were no longer managing one big system. You were managing thousands of small systems. Throw in virtualization, throw in the cloud if you're happy with the term the cloud. And all of a sudden individual sysadmins have gone from saying a 1 to 50 relationship with machines to a 1 to 5,000 relationship with machines. That's simply not scalable. I don't know how many hours people spend doing sysadmins stuff back in the day, but certainly these days you literally, I don't know sysadmins who isn't overworked. I don't know sysadmins in a big shop who isn't constantly looking at new technologies, constantly having to look at new challenges, new products, and having to deal with projects in, you know, far more projects than they used to handling. The only way to handle that is automate. I cannot under emphasize the importance of automation. The only way you're going to scale up to those environments is to actually manage to do all of the things that you do should be done at the click of a button. The other thing is built-in redundancy and the expectation of failure. For me, the classic example of this is that the business comes along and they say, we want 99.999 availability. And it needs to be redundant across all of our data centers, and if anything goes wrong, it needs to be up within an hour. That's whatever OSLA happens to be. And then they say, you know, we want this magnificent Rolls-Royce solution, but here's a budget that buys you a Morris minor. And that's something that we all have to deal with. So my assumption is that instead of starting from the position of let's build the best possible, let's build the most wonderful architecture in the world, let's build a thing, let's make the assumption that let's build from a concept of this is going to fail at some point. So have the expectation of build around, build your process, build your process, and your tools around the assumption that things are going to fail, and you need to build redundancy around them. The key part of doing that is to test things. So don't do your testing at the last moment of your operation. Assume that at any point in time, you should go to pull the plug and say, this is broken, this is gone, what's happened? How do I resolve this problem? The next, the key part of that is understanding that if this is in a failure state, how did it get there and how do I reverse it? So you have the processes and the automation to say, if it fails, this will automatically make it better again. The other thing too is that remember that processes and the various processes and steps across both themes, both development and operations, is all part of the same life cycle. So all part of the experience of running an IT environment, so if the development team is doing testing in their development environment, if they're deploying applications, then the overall, it's part of an overall life cycle that ends with that application going to production. So any process you build should be something that should be transparent and open to everybody. The key around that is tooling. And this is where the open source community has really come to the fore in the DevOps sort of scene. And the reason that it's so prevalent in the open source community is that there are tools like Puppet and CF Engine and Chef in the configuration management space, Control Tier, Rundeck, M Collective, monitoring tools, security testing, a number of people I think believe Lindsay Holmwood who came up with QCumban Agios, talked to you last year about that. These are all perceived as ops tools. And the reason that I put the ops in the question mark is that the value you get from having these tools is not limited to operations teams. Development teams have a key stake in building these tools. And the important part, for example, that we cite as an example of why use Puppet is that an application developer can build a Puppet manifest or a Chef cookbook or a CF Engine recipe, and they can actually deliver that recipe from LifeCycle to its end. So they've cut their code, they've built their associated Puppet module or their Chef cookbook, and it ships all the way through the LifeCycle. So it gets the application development team handed over to a testing team, handed over to a production operations team who actually implement that live. And it means that you actually have that co-operation, have those things like, you know, we are building the infrastructure, we're building, this is how the DNS looks, this is how the backups work, this is how you secure things all the way through that LifeCycle. Deployment and orchestration, you know, application teams, particularly, for example, big Java shops, there's a very clear way they like to deploy and orchestrate applications. There's an important opportunity for developers and operations people to work together with those tools and actually understand what the particular requirements the development team has and what the requirements the operation team have and why those are important. The other thing is things like monitoring and testing, they're not solely limited to one group or another. So the sort of data that you make available with your monitoring environment is a key piece of data that most of your application developers need to understand the troubleshoot problems. So if you actually go and ask them questions about what they want to monitor, what they want to see, then result in something that you both work with and both use. So excuse me. The other piece of this is that lots of things developers do that are useful to operations people. So smart sysadmins for a long time have been version controlling things, everything ranging from DNS configurations to whatever happens to be that changes state you want to keep track of. The last couple of years you've seen lots more operations people understand tools like Git and Bazaar. The reason they've done that is because there's some useful things developers do about branching, about the way that you can actually have multiple people collaborate on tools and collaborate on configuration and collaboration on code that's useful to both operations people. Following up from that there's also things like Agile and XP and concepts like continuous deployment that have emerged out of the development community that are useful things for operations people to know as well. That's not to say that those concepts are perfect, not to say those concepts are things that you should immediately adopt in your operations team. John Allspar from who was at Flickr and is now at Etsy talks a number of times about how Flickr used to deploy into production 10 times a day. A lot of people look at that and go those web ops guys are fucking nuts. There's no way a stable environment is going to be healthy if you deploy 10 times a day into that environment. That may not suit your particular environment, but the ability to say go from having a six-week deployment cycle and there's a number of banks and financial institutions and insurance companies that literally go weeks and weeks between deployments because the processes of deploying an application is so complex, so risky and has such a potential to cause disaster that they're incredibly cautious and risk averse. There's a lot of concepts that come out of that Agile and continuous deployment things that can be adopted to cut down that time period. Application architecture and orchestration. Operations people are classically bad at documentation. I think the only time the network diagram tends to be up to date is usually after the post-mortem for an outage. There's a lot to be said for the way applications people build architecture, the way they document architecture that operations people can learn a lot from. Another thing is testing methodology. Operations people, this is something that's new to a lot of operations people, for most operations people, the concept of testing things is monitoring. Is the SMTP port open? Is the DNS service running? What's the load on this particular machine? Applications people think about testing in a whole different way. They think about testing in terms of, is the application functional? Does the result that I input run through this method and output, does that result mean something? A lot of what someone like Lindsay Holmwood has to say, for example, about Cucumber Nagios, is about understanding that testing things like is the SMTP T service running is not that important. The reason it's not important is because it doesn't tell you anything about the function of the application. It doesn't tell you anything about what the business value of that application does. The SMTP port can be open, but if it's not sending email, and that email is not shaped the right way, whatever letters, marketing blast that Oracle is sending out to everybody, if it's not shaped that way, then the service is valueless. You need to take your testing up to another layer, which is test the business functionality. From an operations perspective, you can learn the lessons that application developers have, which is that you test the function and you return the result that you expect the customer to see. In the process of testing that function, in order to send a message by that SMTP, wow, the SMTP port has to be open. You actually get all of the base-level reporting, and I'm not saying you certainly turn off the monitoring, but all of the base-level reporting, you get that as well, but you also get to understand what the customer is expecting to see, what the application is supposed to deliver actually gets delivered. The last piece I want to talk about is continuous improvement. So in the IT industry, as I said before, we all have customers. The business is our customer. It doesn't matter what sort of organization you work in. You have somebody who consumes the result of what you produce. Those people don't stop asking for things. When I first started out, as a CIS admin, I think we fixed a couple of problems, and I was really chuffed. I was really pleased. I fixed this problem. It's awesome. The customers have been complaining for months that this thing didn't do what they thought it was doing, and it was generating a report. They wanted the report by 7.30 in the morning instead of 8 o'clock. I was really chuffed. They did a bit of fiddling around, and I tweaked some SQL and the report generated much faster. I sat back and I went, I can have a cup of coffee and a cigarette. I've done a really good job. About 9 o'clock, I got an email going, yeah, we've got the report. It looks like this, but now we want it at 7 because we've actually ascertained that it would be really useful if this particular operations team in this time zone had it. I'm like, fuck you guys. Where's my thanks? You did a really good job. Where's my pat on the back? It never comes because customers are never happy with the end result. They want the next best thing. And fair enough too because they pay your salary and you work for them. I get a lot of stick for this because I'm a firm believer in saying that we all deliver a service and the people that pay for that service are entitled to get a decent service out of us. Obviously, if they pay like crap, then that's a whole different thing. But those customers don't stand still. They have requirements that keep changing. You cannot sit back and go, wow, we've done something really cool here. That's the last time we have to worry about this because they will come back next week and say, well, you've delivered me this. I'd like a little piece more. Products don't stand still either. So businesses change. The organizations like banks, I spent a lot of time working in banks, 20 years ago, the concept of online banking, completely non-existent. Nowadays, I ask most customers to say, with the notable exception of the United States, writing a check is a foreign concept to most banking customers. As a result, things become far more complex that don't stand still and you need to keep on top of delivering things that are new, that are changed, and that have unexpected sets of requirements. Technology definitely doesn't stand still either. Things like virtualization have come for a root shock for a lot of shops. Sysadmins have gone from saying, we can manage a 50 systems like this to managing that 5,000 system model. And your team doesn't stand still either. People get bored and there's nothing worse than a bunch of people who are bored and disinterested by their jobs. If you provide people with the right set of challenges, the right sort of things to learn, then you move forward quite rapidly and you're able to improve on things, move forward, and look at things in that sort of continuous improvement light. Then it's an opportunity for both people to learn and people to develop. The strike off and strike hard and be aggressive is that you can't sit still in an environment like this. You can't sit still and wait for things to happen to you. So the continuous improvement piece needs to be, you know, think about what would go wrong here. Think about the failure you're trying to resolve. Think about the issues you need to deal with and deal with those first. The more problems you solve up front, the more time you spend putting into making things better is the... And my boss, Luke Kenees, has a phrase, the faster you get to the pub. And I'm a firm believer that if you address things aggressively and quickly and actually think about the problems first, you're able to get to the pub much faster. The big thing about DevOps for me is that it's a cultural change. It's not a series of technologies and tools and processes alone. A cultural change is hard. People don't like thinking about things differently. They don't like looking at things in a new way. The not invented here syndrome is very strong in a lot of organizations. People hate change. Even little changes. People don't like little changes in their life. We like routine. We like things to be normal. We like things to be consistent. And as a result, when you make change, people don't like you. So I've worked in a lot of environments where we're introducing changes, like DevOps, where the coordination between teams, the first thing that comes out is, why do I have to talk to him? That's the guy that wrote that code that really sucked, and I woke up at 4 o'clock in the morning with a pager. And as a result, people will respond to you in a fairly aggressive way when you make changes like that. That fear of change is largely irrational. It's largely a, I don't know where this is leading. I'm largely, I'm going to be concerned about my future, where this is leading, what happens to me. The best way to deal with that fear of change is to listen to people, actually have a conversation with them about what they're worried about, what's going on, provide concrete examples of how things help, how things change. I think the classic example for a lot of people is presenting to them, this solution will actually result in improved availability. It will result in a better SLA, actually demonstrate how the change will work with them. The other thing is you can't just stand up and make pronouncements. So, as a, you have to actually be in the guts of things. Actually solving problems is actually part of the problem, part of the solution rather than the problem. The worst thing in the world for any IT shop, the classic example is the development team who don't get woken up at 4 o'clock in the morning, who are able to come in at 10 o'clock or whatever happens to be, and they get that, they see the ticket and they go, oh, something's broken. Whereas the development team have been awake for the last five, the operation team have been awake for the last five hours trying to resolve that issue. If you want to make things like culture change in that space, give developers pages, make them on call, make them responsible for service levels and availability that they wouldn't otherwise be. So, I also wanted to talk a little bit about some of the responses to the DevOps movement. And I'm very fond of the classic, Danger Will Robinson with Robbie the Robot, talking about what, you know, some whatever unknown danger is around. It's also my favorite Thomas Paine quote. When we're planning for posterity, we ought to remember that virtue is not hereditary. So, when I first started out in the IT industry, the guys who taught me about ITOps were all in their 50s. They'd all come from that mainframe world. They were very serious guys who were like, availability is the key, big iron is the only way to do it. And the virtues they taught have not passed on. There's a lot of people who've come into the operations and development world with a very different perception of how availability, how operations works. You need to actually provide that concrete set of planning in order for people to continue to understand, to understand what's important. So, some of the classic responses to DevOps that have been, is good systems, CIS admin's have been doing this since they came down from the trees. And for me, this is really something that I look at in fairly bit horror. There was a very small percentage of the CIS admin community in the development community who look at this stuff and go, we've always done this. And those people sort of go, shrug, as a result of that, I'm not going to make any changes outside of my world. So, that other part, they completely ignore the 85% of the community that has no idea that there are better way of doing things, no idea that automation will make you free, that there are ways of, different ways of looking at things. The next response is that, you know, my organization is different. That can't work here. As I said earlier, that whole unique snowflake world, that's gone. There's no way that you can look at an environment and say, we're going to manage everything differently. We're going to be different from everyone else. That thinking is something that will become, you know, that comes from that whole bastard operated from hell view of the world. And that's no longer something that's feasible in a modern environment. The next one is it's all about one particular group or the other. So, I concentrate on operations. That's the area that I work in. And the response here is that operations people are going to have to become like developers. The developer, we're espousing that development methodology is the key, the way forward. And for me, that's simply not true. The thing for us is that both sides need to learn from one another. So, for us, for me, it's understanding that both sides need to actually collaborate. Both sides need to learn from one another. And yes, someone actually posted you're a bunch of elitist European wankers, which I was a bit horrified by because I'd love an EU passport and I didn't have one. So, the other danger is that where DevOps ends up. And for me, as I said earlier, there's been a couple of products that have said they're DevOps certified. The new hotness is DevOps and therefore my product does DevOps. I've also seen Forrester and Gartner both have DevOps events. I know, speaking to Patrick and Chris, neither of them were invited to that. And it's always fascinating when analysts, some of you may understand the analyst market is very soft-serving. The analysts get a question from customers saying, I've heard about this DevOps thing, what's it all about? And they all go around and ask all their questions of their other customers and then feed the result back and it becomes a self-fulfilling loop. The key thing for me is that around DevOps and understanding how to avoid that for DevOps is to make it more than marketing speak. So, actually put this thing stuff in practice in organizations to actually solve problems using DevOps concepts, to solve problems by actually making operations and develop work better together. The other thing is it's very easy to pay lip service to this. It's very easy to say, we're going to have a role called a DevOps person and we're going to have a DevOps team. It's not a role. It's not a thing you can build a team around. It's about a different way of looking at IT and IT operations. It's also a concept that's very easy to cause disenchantment amongst people. Mostly because people are uncertain about those changes. They're uncertain about how the world will change for them in that sort of DevOps environment. It's very easy for people to lose faith in the organization that they work for because they don't feel that they're going to get anything out of this. And that whole disenfranchisement of people is something that I think every organization struggles with. And it's something that I think DevOps has a real opportunity to change that frustrated mentality that a lot of CIS admins have and the frustrated mentality that a lot of developers have dealing with those CIS admins. And that change is around that whole learning, understanding from each other and understanding what's going on. I've found the last few years that I spend a lot of time cutting code where I didn't used to. I used to spend a lot of time writing batch scripts. I now find myself cutting Ruby code, learning things, learning new things. And I get a lot more pleasure out of my role than I did previously. And the last one, anti-disestablishment-terrorism. I just wanted to put in because I like anti-disestablishment-terrorism. So I think I've spoken very quickly and I think rather poorly, unfortunately. But I did want to ask if anyone had any questions before we go on. When you look at the role of development and operations in your environments, what are some of the tangible things that change about their roles? Like I know for agile principles, for example, I mean things like between agile and testing tend to blur the lines. What about in DevOps? Does it change the... Does it change some of the activities? Does it blur some of the lines between development and operations? So I think classically, the whole... I think configuration management is the prime example there. So we used to spend a lot of time doing things like creating users, installing packages, managing services, monitoring things. Those are not cool things. Those are boring tasks. The thing about that DevOps allows is to say, let's take all of that stuff, let's make it automated, let's make it something that we don't have to worry about. Because the really cool thing I'd like to do is learning how to scale this particular application. Or I'd like to learn how the no-SQL stuff that the developers have introduced works. I think the tangible outcome for a lot of organizations is that you get away with saying that all the boring shit, the stuff that has occupied your time, fixing things, installing things, changing basic configuration things that is fairly sort of boring and monotonous, you can actually work on the projects that are actually interesting. The other thing is that being for a lot of organizations, having that conversation with development teams means that you're actually in the guts of the development process. That for me is a far more interesting activity than sitting around being an operator and watching the meters and running... If you run a whole bunch of people through and, for example, I have a couple of organizations that I've worked for that have embedded operations people into the developers' scrum. They can't necessarily cut the same code, but they're part of the actual development of the application. By the time the application gets to production, those guys are like, this is really cool. And they become evangelists for the applications developed. Not only do they understand the application better, they understand the metrics and the basic concept the developers put into the thing, but they actually care about it. I think that's a really important... I've seen too many applications that get thrown over the fence, and you go, I don't really understand what this does, I don't understand why it breaks all the time, I don't understand why it performs like shit. But if you're actually involved from day one, you can go, well, we made a compromise here. We actually decided to do it like this instead of this because of this particular reason. And you go, we can make it better like this, this, this, but here's the caveats we had. I think a lot of that sort of knowledge is not something that gets passed on otherwise. When will your big and explicit book about Puppet will be finished finally? So I'm hugely jet lagged, and that question has been asked 17 times since I landed on Belgian soil. March or April, maybe? No. Unfortunately, yeah, I think I spent about 20-hour days at the moment, and it's almost done. Almost. Two chapters left. And I had the help of a very good colleague who was pushing along. Hello. Obviously, one of the keys behind DevOps is a better collaboration between the development and the IT operations, guys. Coming from the IT operations field yourself, what would you expect from a development team for you to relate better to it? What could the developers do to ease your way towards this collaboration? So I talked about embedding operations people into Scrum or into Yakanban or whatever developers do. Do it the other way around as well. So drag development people into incidents and outages. If you have an operation stand up, drag the developers in. Make them understand the sort of problems you have. A lot of developers operate on the principle that we've built something that's really cool, that's really hot, and then we piss off down the pub and the operations people have to manage it. If they actually understand that there are implications and consequences to their choices, that if they choose a particular application architecture, it's going to have an impact on the operations team at the end of the day. The number of developers I have a conversation with who go, oh, you didn't tell us that you don't do things like this or it's hard to package this or this set of Java libraries are like this. If you actually have teams involved on both sides from day one of a project, there's a far greater appreciation for how that works. Developers also, developers are like, you have a particular operations problem. We spend to be fairly metric centric people, operations people. We're interested in metrics, we're interested in measuring things. Developers are also interested in that sort of thing, and they also have a set of things they'd like to know. When something goes wrong, something's performing badly, they have that desire for that information. If you have a conversation with a developer that says, let me tell you about the metrics I have, you can tell me about how your application is instrumented, and then I can tell you what I'd like to have added to that instrumentation, you can tell me what data you'd like from my end. Those sort of conversations are really powerful, and I found that a lot of developers are sort of stunned by the depth to which operations people use metrics, and they've sort of appreciated our perception of that. So I guess there's some conversations around that that are sort of really healthy, and it promotes that sort of cooperation. So this sounds really interesting, and it sounds like what I've done actually in some of the companies I've worked with. It works really well, but does this really scale at all? Does this scale to a bigger team, and bigger teams of operations and development people? Because that's one of the main problems when the teams become big enough, it's really hard for them to mangle. And the other thing is mostly if you've done this, but I have this very weird observation that development people think, for example, that CPU memory and hard drives are infinite, and most of their resources are infinite, and this is one of the main issues with deploying their software, so if you've managed to teach anyone that, that's not exactly right. I think the challenge, and you can see a lot of pushback around DevOps and the fact that it works best in small teams, for example. People say it's much easier to do this if you haven't got, you're not an enterprise and you haven't got 500 people. I think that's, it's true, it's easy to have relationships in small teams. I think the larger teams, you do things like, you move whole groups of people through different teams and different organizations. You shuffle people around, you keep people actually learning new things, you don't silo things. I think that the key mistake people make is you build silos, you build a group of people who are all the Java developers, and you build another group of people that are all of the WebOps people, and then you build another two, that is not only an unhealthy way of building teams, but it's a way that doesn't promote that sort of cooperation and coordination. He's cancelled his question. Hi. I have a quick question over here. When would be the exact perfect time to start doing DevOps when you're doing a startup? When you're doing a beginning of a startup and you found it yourself, you don't have that much capital. I can't see why, for example, a whole bunch of things are useful, but the whole configuration management and all that might just be a drag in the beginning, because you really need to move quickly forward in order to get capital and first clients and angel and whatever. So how would you approach that one? I think Greenfields is far easier. If you have a Greenfields environment, it's far easier to introduce new concepts. People don't have that it's not invented here, or we've always done it this way, mentality in a Greenfields sort of environment. I think though there are real advantages to looking at something like DevOps in legacy environments. So legacy environments, we've all got them. We've all got lots of organizations have their collection of REL4 machines that have been living there forever and they can't change or upgrade them. They're a bunch of Windows NT machines that they've ring fenced off from the rest of the environment. Those things cost lots of money to maintain, because they're all sort of, that becomes sort of boutique or no one wants to change anything. There's real opportunities in that sort of environment to actually, if you automate stuff in those environments to save a lot of money and to become a lot more efficient, but it's a lot harder to do because making those changes, systems like that, tend to be sort of very fossilized. There tend to be sort of things that have the organizations become risk adverse about. We can't change that no one who works here understand how that works anymore. But as a result, the opportunities there are quite large, but the risks are quite high. But Greenfields is certainly easier. If you're a startup from day one, the classic example is Agile. If you start from day one using Agile, it's far easier to go from a legacy development process to an Agile than going from a legacy to an Agile process. If you start from scratch there. I'm in operation myself. And in the area, in this not uncommon situation where all the applications, all the applications that we are deploying, we are not developed in-house. So is DevOps still relevant? And how do you trigger this cultural change about automating things? Because we can only automate some baseline, basic things like SSH configuration, VM provisioning, and very basic stuff. All the rest is too diverse and too complex to do automatic deployment. So I deal a lot with customers that have outsourced stuff to India and China. And it's really interesting. Outsourcing in a lot of cases gets a very bad name. It's the lowest bidder who's outsourced, you outsource your stuff to, and they have no interest in quality or changing things. I think that's the wrong attitude to take. The opportunity is there for you to infect those organizations. If they understand, and there's a couple of people here, I know there's one person here from India who works in a big Indian IT company who actually can tell you that those guys are interested in learning new things. They want to work on cool technology, and if you have an opportunity to infect them with a whole different idea about how to do things, I think those things will be picked up. And there's big cultural barriers in any outsourcing arrangement, but I think those are also opportunities. And it's also about demonstrating value. So if you can demonstrate that it's cheaper for the organization to do it a particular way, then they're going to jump on that because they want to deliver the service and make a bigger profit. If you can deliver, you get something out of that, you get better availability, you get a better quality of service, you get a better SLA, and they get to do it cheaper or faster or whatever it is. So it's a green-win situation for both organizations. Hi. I luckily started two weeks ago, and I'm responsible for implementing application monitoring in a quite very big corporate Belgian environment, which already says a lot for people to know about that. And I'm facing the fact that the operation people have a solar system to be able to participate in that, but on the other hand, they're saying, don't flood us with all your application-level events and all the business process metrics that you would like to see on the dashboard for the business. Do you have any comment on that, or do you just want to show some empathy? So I'm not sure I understood the question, Tarlot, totally, but I think, again, it's about demonstrating what... So if you demonstrate that... Was the question, if I'm paraphrased, they're not interested in looking at the business metrics, they're not interested in looking at the application metrics? Well, the developers and the business are, but the iOS people, they want to limit the number of events that will go through their monitoring systems. So what I'm trying to promote is to help them change their mind in such a way that they accept that monitoring should go a bit higher level. I think it is about demonstrating the value of it, so demonstrating that monitoring the CPU, the memory, and the disk is not what's important about the application. What's important about the application is what business results it delivers. You still want to know about the other stuff, but if all of those metrics are perfect, if all of those dashboards are perfect, but the response time over here and the application doesn't do what it's supposed to do, then it's not working. So you need to demonstrate that if they have an underpicture of that, they're necessarily the same metrics that all the dashboards and the business people look at, but a view of that and how it links to those other metrics that they care about, infrastructure metrics, that's a way that you can say, this is the picture your customers get, this is the picture that they're seeing, and they pay your salaries, they pay all of us, and we have to work together and solve in that particular problem, or give that particular view. I think it's probably... I think there's time for one more question. Any more questions here? I don't see myself. Thanks very much.
 Thank you very much.  Thank you very much.  Thank you.  Good afternoon. Thanks for coming along.  It's a very, very big room. It's a lot bigger room than I'm used to speaking in.  It's not a little bit about me. I've been involved in the open source community for about 20 years now.  Since I was a teenager and discovered this Linux thing, it was a bit cooler than that Windows thing.  I work at puppet labs. I run the operations and professional services group there.  I'm an author. I've written five technical books.  Mostly about Linux. This had been an open source set of things.  I'm a big geek. My wife would say very much a big geek.  I'm known to buy every single gadget that appears on the market.  I usually buy two of them and break them fairly quickly.  I should warn you, I do talk funny. I'm from Australia.  Pretty much every other part of the English speaking world doesn't believe we speak English.  If you find anything you can't understand or find that I'm talking very quickly,  wave your hands frantically in the air and go, please slow down.  Okay. Not a good sign.  I'm also a winchig bastard.  I've spent a lot of years in operations working in some big shops.  I'm well known for complaining about things.  I don't like things that are broken. I'm a firm believer in perfection.  I don't think you can achieve perfection, but you can get a bloody well of a long way towards it.  So what's DevOps all about?  This is a topic that comes up a fair bit.  I don't think there's a clear answer to it yet.  DevOps is about cooperation between development teams and operations teams.  Particularly important as to why Belgium is so important for DevOps  is because the gentleman who coined the phrase DevOps, Patrick Dubois,  is obviously from Belgium. Chris and I'm going to stuff his surname up, Bay Art,  also heavily involved in the original DevOps movement.  The original DevOps thinking emerged out of a bunch of people in Belgium  sitting around drinking beer, which may have something to do with it,  while they felt the love for one another.  Generally, the whole movement emerged out of a bunch of northern European sysadmins.  So what comes up when you start to look at DevOps?  I did a little bit of a keyword cloud, and I'm afraid the top bit's cut off there.  There's some of the phrases that come up when you look at various blog posts around DevOps.  I look at these sort of things and I go, well, what does this become?  Is this some kind of buzzword bingo where after a few phrases,  we'll get to sit around together and go, we can talk about DevOps and cooperation  and ITIL and we can talk about cross-functional teams,  and then somebody can sit in their meeting and go, bingo,  we've managed to parrot out all of the appropriate corporate phrases and phraseology.  Is it a pop culture movement?  There will be a bunch of fly-by-night guys who are just trying to con you into buying something.  I've seen a couple of products so far that claim they're DevOps compliant.  I'm not sure who they were certified by, but if they buy me beer, I'm happy to.  Or bourbon even better.  I don't think we're a pop culture movement.  I also don't think that we're going away in a hurry,  but I do have some caveats before I talk a bit further.  It's very early days.  DevOps is probably about two years old in terms of its thinking,  and it's probably only about six months old in terms of people actually,  I guess, more theoretically considering all of the possibilities that DevOps implies.  Largely speaking, the first conversations about DevOps happened about two years ago.  I think last year they had a DevOps dinner at FOSDEM,  and I believe there was about 10 or 15 people there.  Last night they had the same thing again, and there were 60 people.  So it's early days.  No one has all the answers.  I don't think anyone's suggesting that it's a fixed movement.  We certainly don't have a manifesto or a style guide that says,  this is how you recognize it's DevOps, it's painted green, and it's got little stripes.  That's not the case.  Nothing's fixed in stone.  I'm certainly not advocating a position that there's some sort of doctrinal approach  that we need to take.  And at the moment, for me, this is all about outreach.  It's trying to get a message out about what DevOps is  and about the community around DevOps.  And you may notice a little footer on my thing, all opinions of my own,  maybe subject to change without alcohol.  I think a lot of the DevOps movement has been about discussion,  about people going, well, DevOps is a bit of this, it's a bit of that,  and let's see how this all fits together.  For me, I like lists and DevOps for me is about four things.  Simplicity, relationships, process, and continuous improvement.  And I'm going to talk a little bit about each of those areas  and why they're sort of important to me and why I think they're an important part of DevOps  and why I think that makes DevOps useful to both operations and development communities.  Firstly, simplicity.  So we all work in, you know, if you work with IT, then IT is complex.  You know, there's simple things, some of the simple things, you know,  that in an IT shop, you know, how you build your application stack,  what particular operating system you choose can cause a massive amount of debate.  I'm not even going to talk about things like,  if I sat down and said to you, Vim is the best editor in the world,  then all the EMACs people would be stoning me.  And it's not an uncommon phenomenon in the IT world to have huge,  complex applications linked together based on choices  that you have to inherit from somewhere, legacy applications.  And in order to manage those environments, simplicity is the key.  So you need to choose things that are simple and easy to manage.  Those things need to be repeatable and reusable.  It's particularly in environments where you're managing thousands of hosts.  You cannot do something that is a one-off thing anymore.  You can no longer get away with saying, all of my systems are unique snowflakes  and they're all different and they need a different set of rules.  You simply would do not have the bandwidth or the time to manage a thousand systems  that you are considering to be unique snowflakes.  And simplicity implies easy to communicate.  Whatever message you need to communicate about your systems,  whether it be their state, whether it be their configuration,  whether it be, you know, how to recover them or back them up or secure them,  it needs to be something that's simple and easy to communicate.  The key for me around this is that if you're sitting in,  if you're sitting in the end, at the end of the day,  you're sitting in a room full of people trying to fix a bug or doing an outage.  You want to have the best possible conversation with somebody  where the end result is that you solve the problem as fast as possible  and you solve the problem in such a way that it never reappears again.  And if your environment isn't simple and repeatable,  then you're going to have a lot of struggle with that.  The next thing for me is relationships.  So we all work in an industry that's well known for the fact that we're sort of,  supposedly misanthropic, we're supposedly this bastard operator from hell types,  we've just had been types, and we're the ones, you pick up the phone and call them and you go,  they're going to blame me for breaking something or the dev team is going to actually,  you know, he's going to cry out and say,  here's the guys that, you know, they can't install our latest version of the application  and they hate us and, you know, we cause them all their problems.  But every single thing that we do in the IT world is built around a relationship with other people.  Too many of the conversations I've had over the years in IT have consisted of, you know,  what I like to call the blame storm.  So the blame storm is where instead of ascertaining what the actual problem is,  the first thing you do is work out who can I blame for the problem  and how can I make sure it's not my fault.  And there's a famous American phrase, cover your ass.  And that's an expression that gets quite used in enterprise IT shops.  You're attempting to rather than solve the problem,  you're attempting to be the person that is not at fault.  The only way to change that mentality, to change that conflict between teams,  is to build relationships.  And there's a lot of reading you can do about how to build relationships  and a lot of it's touchy feely stuff.  I now work in America where everyone talks about their feelings all the time.  I'm Australian, we don't. We just get drunk and hit things.  But generally speaking, it's about engagement and it's about engaging early and often.  So if you imagine the life cycle of a project,  life cycle project we can see solve, you know,  if you're working a business and the business says,  we would like to build a product or a service  and that product or service is going to talk to a new bunch of customers.  And the marketing team, you know, come up with a marketing message  and they come up with a business comes up with all the things they want to do.  They throw over a sort of, you know, maybe a few scratchings on a cocktail napkin  that the marketing guys came up with between strippers  and then handed over to a bunch of developers.  And the developers go, we can't build this, it's impossible.  That's the first reaction almost every time.  They next go, okay, we can build this, but it's going to look a bit like this.  They then build the application, you know,  there's a whole bunch of people sitting around and they go,  this is really cool, we should use React here.  That no SQL stuff is really cool.  And then someone says, oh, wow, we better make sure that it runs on this  and we better make sure it runs on the latest version of Ruby.  And my God, we can use this really cool library I found  and let's install all of these gems and vendor them in here.  And all of a sudden, and I'm picking on Ruby people because I'm a Ruby person,  but it's true of almost every environment.  The next step that happens is that the team, you know,  puts together this amazing demo and they demo it for marketing people.  Some people say we hate it.  They hire a couple of UI people and fix that.  And then the application people say we're done.  Hand it over, let's toss it over the fence.  And the people are handed over to operations people.  The operations people take them on look at it and go, oh, my God,  how are we going to make this work?  It doesn't run on our standard operating system,  it requires all these libraries we don't have.  How do we back it up?  My God, it's syncing data between three data centers across four continents  and how do we secure it?  The application architecture is completely different from our existing application architecture.  And they model through, they spend six weeks trying to work out all of the bits and pieces,  they jury rig things, they write scripts,  they build systems that, you know, especially when you need systems to stick application,  they run it up, first day goes under some load and it falls over.  So there's then, you know, we've all sat in that meeting where the system's fallen over,  you know, there's 10,000 concurrent connections and the developers say  it should support 10,000 concurrent connections, you guys haven't provisioned enough systems,  you guys haven't worked out how to handle this problem.  And everybody sits around and basically it's an exercise in trying to ascertain who's fault it was,  rather than fixing the problem and trying to ascertain, you know,  how do we get here and how do we get out of the hole?  The key to getting out of that hole is instead of right back at the very start  where the business has a particular need, the business says we want to create something.  The people they should have gone and have a conversation with are both developers and operations.  And if those people are in the same room having that conversation,  then all of these little things which are called in IT world, non-functional requirements.  So non-functional requirements are little things like security and backing things up.  And they're equally as, ironically called non-functional,  but equally as important to the process of building an application.  So if we start from the beginning where both sides actually have a conversation,  actually get involved and talk to one another,  then we actually have an opportunity to develop applications and products  where everybody gets what they want out of it.  So how do we do that?  To start to do that, both sides need to have des tentes.  And my French accent is terrible.  It sounds much better when you actually have a French accent.  But des tentes is about both sides need to understand that they're all in the same problem.  We're all in the same problem space.  We all have the same issues we have to deal with.  We all have the same bosses, the customers.  We all work for the same people.  We all get paid the same way.  We need to actually get stuck into a conversation with one another  where we actually don't let our prejudices sort of color the conversation.  And you have to have that conversation.  You can't send someone an email.  I find the classic example of someone is wrong on the internet.  And the reason someone is usually wrong on the internet is because someone has written an email  and someone else has read the email and gone,  I interpret it like this, or I speak English as a second language  and I've read this email in a particular way.  Or I miss the smiley face. You suck at the end of that.  And next thing you know, somebody is having a conversation that consists of  we're actually aggressively pursuing a point rather than actually having a dialogue.  And I can't underestimate you the number of times that I've solved simple problems  by getting off my ass, walking halfway across a room,  or getting onto a video conference and actually talking to the person involved.  And building those relationships is a key part of any of the steps that we need to talk about and DevOps.  So the next important thing for me is process.  So I spent about watching the IT industry change dramatically over the last sort of 20 years.  When I first started, the one system we worked on was a massive big mainframe,  partitioned into production and development.  There was a DRP mainframe and there was pretty much, we had a system and it did stuff.  There was no client server. There were dumb terminals.  I started to cut my teeth on 3470 green screens, 3070 green screens.  That's changed dramatically.  We went from a world where all of a sudden client server appeared.  You were no longer managing one big system. You were managing thousands of small systems.  Throw in virtualization, throw in the cloud if you're happy with the term the cloud.  And all of a sudden individual sysadmins have gone from saying a 1 to 50 relationship with machines  to a 1 to 5,000 relationship with machines.  That's simply not scalable.  I don't know how many hours people spend doing sysadmins stuff back in the day,  but certainly these days you literally, I don't know sysadmins who isn't overworked.  I don't know sysadmins in a big shop who isn't constantly looking at new technologies,  constantly having to look at new challenges, new products,  and having to deal with projects in, you know, far more projects than they used to handling.  The only way to handle that is automate.  I cannot under emphasize the importance of automation.  The only way you're going to scale up to those environments is to actually manage to  do all of the things that you do should be done at the click of a button.  The other thing is built-in redundancy and the expectation of failure.  For me, the classic example of this is that the business comes along and they say,  we want 99.999 availability.  And it needs to be redundant across all of our data centers,  and if anything goes wrong, it needs to be up within an hour.  That's whatever OSLA happens to be.  And then they say, you know, we want this magnificent Rolls-Royce solution,  but here's a budget that buys you a Morris minor.  And that's something that we all have to deal with.  So my assumption is that instead of starting from the position of let's build the best possible,  let's build the most wonderful architecture in the world, let's build a thing,  let's make the assumption that let's build from a concept of this is going to fail at some point.  So have the expectation of build around, build your process, build your process,  and your tools around the assumption that things are going to fail,  and you need to build redundancy around them.  The key part of doing that is to test things.  So don't do your testing at the last moment of your operation.  Assume that at any point in time, you should go to pull the plug and say,  this is broken, this is gone, what's happened?  How do I resolve this problem?  The next, the key part of that is understanding that if this is in a failure state,  how did it get there and how do I reverse it?  So you have the processes and the automation to say,  if it fails, this will automatically make it better again.  The other thing too is that remember that processes and the various processes and steps  across both themes, both development and operations, is all part of the same life cycle.  So all part of the experience of running an IT environment,  so if the development team is doing testing in their development environment,  if they're deploying applications, then the overall,  it's part of an overall life cycle that ends with that application going to production.  So any process you build should be something that should be transparent and open to everybody.  The key around that is tooling.  And this is where the open source community has really come to the fore in the DevOps sort of scene.  And the reason that it's so prevalent in the open source community is that there are tools like  Puppet and CF Engine and Chef in the configuration management space,  Control Tier, Rundeck, M Collective, monitoring tools, security testing,  a number of people I think believe Lindsay Holmwood who came up with QCumban Agios,  talked to you last year about that.  These are all perceived as ops tools.  And the reason that I put the ops in the question mark is that the value you get from having these tools  is not limited to operations teams.  Development teams have a key stake in building these tools.  And the important part, for example, that we cite as an example of why use Puppet is that  an application developer can build a Puppet manifest or a Chef cookbook or a CF Engine recipe,  and they can actually deliver that recipe from LifeCycle to its end.  So they've cut their code, they've built their associated Puppet module or their Chef cookbook,  and it ships all the way through the LifeCycle.  So it gets the application development team handed over to a testing team,  handed over to a production operations team who actually implement that live.  And it means that you actually have that co-operation, have those things like,  you know, we are building the infrastructure, we're building, this is how the DNS looks,  this is how the backups work, this is how you secure things all the way through that LifeCycle.  Deployment and orchestration, you know, application teams, particularly,  for example, big Java shops, there's a very clear way they like to deploy and orchestrate applications.  There's an important opportunity for developers and operations people to work together with those tools  and actually understand what the particular requirements the development team has  and what the requirements the operation team have and why those are important.  The other thing is things like monitoring and testing, they're not solely limited to one group or another.  So the sort of data that you make available with your monitoring environment  is a key piece of data that most of your application developers need to understand the troubleshoot problems.  So if you actually go and ask them questions about what they want to monitor,  what they want to see, then result in something that you both work with and both use.  So excuse me.  The other piece of this is that lots of things developers do that are useful to operations people.  So smart sysadmins for a long time have been version controlling things,  everything ranging from DNS configurations to whatever happens to be that changes state you want to keep track of.  The last couple of years you've seen lots more operations people understand tools like Git and Bazaar.  The reason they've done that is because there's some useful things developers do about branching,  about the way that you can actually have multiple people collaborate on tools  and collaborate on configuration and collaboration on code that's useful to both operations people.  Following up from that there's also things like Agile and XP  and concepts like continuous deployment that have emerged out of the development community  that are useful things for operations people to know as well.  That's not to say that those concepts are perfect,  not to say those concepts are things that you should immediately adopt in your operations team.  John Allspar from who was at Flickr and is now at Etsy  talks a number of times about how Flickr used to deploy into production 10 times a day.  A lot of people look at that and go those web ops guys are fucking nuts.  There's no way a stable environment is going to be healthy if you deploy 10 times a day into that environment.  That may not suit your particular environment, but the ability to say go from having a six-week deployment cycle  and there's a number of banks and financial institutions and insurance companies  that literally go weeks and weeks between deployments  because the processes of deploying an application is so complex, so risky  and has such a potential to cause disaster that they're incredibly cautious and risk averse.  There's a lot of concepts that come out of that Agile and continuous deployment things  that can be adopted to cut down that time period.  Application architecture and orchestration.  Operations people are classically bad at documentation.  I think the only time the network diagram tends to be up to date  is usually after the post-mortem for an outage.  There's a lot to be said for the way applications people build architecture,  the way they document architecture that operations people can learn a lot from.  Another thing is testing methodology.  Operations people, this is something that's new to a lot of operations people,  for most operations people, the concept of testing things is monitoring.  Is the SMTP port open?  Is the DNS service running?  What's the load on this particular machine?  Applications people think about testing in a whole different way.  They think about testing in terms of, is the application functional?  Does the result that I input run through this method and output,  does that result mean something?  A lot of what someone like Lindsay Holmwood has to say, for example, about Cucumber Nagios,  is about understanding that testing things like is the SMTP T service running  is not that important.  The reason it's not important is because it doesn't tell you anything  about the function of the application.  It doesn't tell you anything about what the business value of that application does.  The SMTP port can be open, but if it's not sending email,  and that email is not shaped the right way,  whatever letters, marketing blast that Oracle is sending out to everybody,  if it's not shaped that way, then the service is valueless.  You need to take your testing up to another layer,  which is test the business functionality.  From an operations perspective, you can learn the lessons that application developers have,  which is that you test the function and you return the result that you expect the customer to see.  In the process of testing that function, in order to send a message by that SMTP,  wow, the SMTP port has to be open.  You actually get all of the base-level reporting,  and I'm not saying you certainly turn off the monitoring,  but all of the base-level reporting, you get that as well,  but you also get to understand what the customer is expecting to see,  what the application is supposed to deliver actually gets delivered.  The last piece I want to talk about is continuous improvement.  So in the IT industry, as I said before, we all have customers.  The business is our customer.  It doesn't matter what sort of organization you work in.  You have somebody who consumes the result of what you produce.  Those people don't stop asking for things.  When I first started out, as a CIS admin, I think we fixed a couple of problems,  and I was really chuffed. I was really pleased.  I fixed this problem. It's awesome.  The customers have been complaining for months that this thing didn't do what they thought it was doing,  and it was generating a report.  They wanted the report by 7.30 in the morning instead of 8 o'clock.  I was really chuffed. They did a bit of fiddling around,  and I tweaked some SQL and the report generated much faster.  I sat back and I went, I can have a cup of coffee and a cigarette.  I've done a really good job.  About 9 o'clock, I got an email going,  yeah, we've got the report. It looks like this, but now we want it at 7  because we've actually ascertained that it would be really useful  if this particular operations team in this time zone had it.  I'm like, fuck you guys. Where's my thanks?  You did a really good job. Where's my pat on the back?  It never comes because customers are never happy with the end result.  They want the next best thing.  And fair enough too because they pay your salary and you work for them.  I get a lot of stick for this because I'm a firm believer in saying  that we all deliver a service and the people that pay for that service  are entitled to get a decent service out of us.  Obviously, if they pay like crap, then that's a whole different thing.  But those customers don't stand still.  They have requirements that keep changing.  You cannot sit back and go, wow, we've done something really cool here.  That's the last time we have to worry about this  because they will come back next week and say,  well, you've delivered me this. I'd like a little piece more.  Products don't stand still either.  So businesses change.  The organizations like banks, I spent a lot of time working in banks,  20 years ago, the concept of online banking, completely non-existent.  Nowadays, I ask most customers to say, with the notable exception of the United States,  writing a check is a foreign concept to most banking customers.  As a result, things become far more complex that don't stand still  and you need to keep on top of delivering things that are new, that are changed,  and that have unexpected sets of requirements.  Technology definitely doesn't stand still either.  Things like virtualization have come for a root shock for a lot of shops.  Sysadmins have gone from saying, we can manage a 50 systems like this  to managing that 5,000 system model.  And your team doesn't stand still either.  People get bored and there's nothing worse than a bunch of people  who are bored and disinterested by their jobs.  If you provide people with the right set of challenges,  the right sort of things to learn, then you move forward quite rapidly  and you're able to improve on things, move forward,  and look at things in that sort of continuous improvement light.  Then it's an opportunity for both people to learn and people to develop.  The strike off and strike hard and be aggressive is that you can't sit still  in an environment like this. You can't sit still and wait for things to happen to you.  So the continuous improvement piece needs to be, you know,  think about what would go wrong here.  Think about the failure you're trying to resolve.  Think about the issues you need to deal with and deal with those first.  The more problems you solve up front,  the more time you spend putting into making things better is the...  And my boss, Luke Kenees, has a phrase,  the faster you get to the pub.  And I'm a firm believer that if you address things aggressively and quickly  and actually think about the problems first,  you're able to get to the pub much faster.  The big thing about DevOps for me is that it's a cultural change.  It's not a series of technologies and tools and processes alone.  A cultural change is hard.  People don't like thinking about things differently.  They don't like looking at things in a new way.  The not invented here syndrome is very strong in a lot of organizations.  People hate change.  Even little changes. People don't like little changes in their life.  We like routine. We like things to be normal.  We like things to be consistent.  And as a result, when you make change, people don't like you.  So I've worked in a lot of environments where we're introducing changes,  like DevOps, where the coordination between teams,  the first thing that comes out is, why do I have to talk to him?  That's the guy that wrote that code that really sucked,  and I woke up at 4 o'clock in the morning with a pager.  And as a result, people will respond to you in a fairly aggressive way  when you make changes like that.  That fear of change is largely irrational.  It's largely a, I don't know where this is leading.  I'm largely, I'm going to be concerned about my future,  where this is leading, what happens to me.  The best way to deal with that fear of change is to listen to people,  actually have a conversation with them about what they're worried about,  what's going on, provide concrete examples of how things help, how things change.  I think the classic example for a lot of people is presenting to them,  this solution will actually result in improved availability.  It will result in a better SLA, actually demonstrate how the change will work with them.  The other thing is you can't just stand up and make pronouncements.  So, as a, you have to actually be in the guts of things.  Actually solving problems is actually part of the problem,  part of the solution rather than the problem.  The worst thing in the world for any IT shop,  the classic example is the development team who don't get woken up at 4 o'clock in the morning,  who are able to come in at 10 o'clock or whatever happens to be,  and they get that, they see the ticket and they go, oh, something's broken.  Whereas the development team have been awake for the last five,  the operation team have been awake for the last five hours trying to resolve that issue.  If you want to make things like culture change in that space,  give developers pages, make them on call,  make them responsible for service levels and availability that they wouldn't otherwise be.  So, I also wanted to talk a little bit about some of the responses to the DevOps movement.  And I'm very fond of the classic,  Danger Will Robinson with Robbie the Robot,  talking about what, you know, some whatever unknown danger is around.  It's also my favorite Thomas Paine quote.  When we're planning for posterity,  we ought to remember that virtue is not hereditary.  So, when I first started out in the IT industry,  the guys who taught me about ITOps were all in their 50s.  They'd all come from that mainframe world.  They were very serious guys who were like,  availability is the key, big iron is the only way to do it.  And the virtues they taught have not passed on.  There's a lot of people who've come into the operations and development world  with a very different perception of how availability, how operations works.  You need to actually provide that concrete set of planning  in order for people to continue to understand, to understand what's important.  So, some of the classic responses to DevOps that have been,  is good systems, CIS admin's have been doing this since they came down from the trees.  And for me, this is really something that I look at in fairly bit horror.  There was a very small percentage of the CIS admin community  in the development community who look at this stuff and go,  we've always done this.  And those people sort of go, shrug, as a result of that,  I'm not going to make any changes outside of my world.  So, that other part, they completely ignore the 85% of the community  that has no idea that there are better way of doing things,  no idea that automation will make you free,  that there are ways of, different ways of looking at things.  The next response is that, you know, my organization is different.  That can't work here.  As I said earlier, that whole unique snowflake world, that's gone.  There's no way that you can look at an environment and say,  we're going to manage everything differently.  We're going to be different from everyone else.  That thinking is something that will become, you know,  that comes from that whole bastard operated from hell view of the world.  And that's no longer something that's feasible in a modern environment.  The next one is it's all about one particular group or the other.  So, I concentrate on operations.  That's the area that I work in.  And the response here is that operations people are going to have to become like developers.  The developer, we're espousing that development methodology is the key, the way forward.  And for me, that's simply not true.  The thing for us is that both sides need to learn from one another.  So, for us, for me, it's understanding that both sides need to actually collaborate.  Both sides need to learn from one another.  And yes, someone actually posted you're a bunch of elitist European wankers,  which I was a bit horrified by because I'd love an EU passport and I didn't have one.  So, the other danger is that where DevOps ends up.  And for me, as I said earlier, there's been a couple of products that have said they're DevOps certified.  The new hotness is DevOps and therefore my product does DevOps.  I've also seen Forrester and Gartner both have DevOps events.  I know, speaking to Patrick and Chris, neither of them were invited to that.  And it's always fascinating when analysts, some of you may understand the analyst market is very soft-serving.  The analysts get a question from customers saying,  I've heard about this DevOps thing, what's it all about?  And they all go around and ask all their questions of their other customers  and then feed the result back and it becomes a self-fulfilling loop.  The key thing for me is that around DevOps and understanding how to avoid that for DevOps is to make it more than marketing speak.  So, actually put this thing stuff in practice in organizations to actually solve problems using DevOps concepts,  to solve problems by actually making operations and develop work better together.  The other thing is it's very easy to pay lip service to this.  It's very easy to say, we're going to have a role called a DevOps person and we're going to have a DevOps team.  It's not a role. It's not a thing you can build a team around.  It's about a different way of looking at IT and IT operations.  It's also a concept that's very easy to cause disenchantment amongst people.  Mostly because people are uncertain about those changes.  They're uncertain about how the world will change for them in that sort of DevOps environment.  It's very easy for people to lose faith in the organization that they work for  because they don't feel that they're going to get anything out of this.  And that whole disenfranchisement of people is something that I think every organization struggles with.  And it's something that I think DevOps has a real opportunity to change that frustrated mentality  that a lot of CIS admins have and the frustrated mentality that a lot of developers have dealing with those CIS admins.  And that change is around that whole learning, understanding from each other and understanding what's going on.  I've found the last few years that I spend a lot of time cutting code where I didn't used to.  I used to spend a lot of time writing batch scripts.  I now find myself cutting Ruby code, learning things, learning new things.  And I get a lot more pleasure out of my role than I did previously.  And the last one, anti-disestablishment-terrorism.  I just wanted to put in because I like anti-disestablishment-terrorism.  So I think I've spoken very quickly and I think rather poorly, unfortunately.  But I did want to ask if anyone had any questions before we go on.  When you look at the role of development and operations in your environments,  what are some of the tangible things that change about their roles?  Like I know for agile principles, for example, I mean things like between agile and testing tend to blur the lines.  What about in DevOps? Does it change the... Does it change some of the activities?  Does it blur some of the lines between development and operations?  So I think classically, the whole... I think configuration management is the prime example there.  So we used to spend a lot of time doing things like creating users, installing packages,  managing services, monitoring things.  Those are not cool things. Those are boring tasks.  The thing about that DevOps allows is to say, let's take all of that stuff, let's make it automated,  let's make it something that we don't have to worry about.  Because the really cool thing I'd like to do is learning how to scale this particular application.  Or I'd like to learn how the no-SQL stuff that the developers have introduced works.  I think the tangible outcome for a lot of organizations is that you get away with saying that all the boring shit,  the stuff that has occupied your time, fixing things, installing things, changing basic configuration things  that is fairly sort of boring and monotonous, you can actually work on the projects that are actually interesting.  The other thing is that being for a lot of organizations, having that conversation with development teams  means that you're actually in the guts of the development process.  That for me is a far more interesting activity than sitting around being an operator and watching the meters and running...  If you run a whole bunch of people through and, for example, I have a couple of organizations  that I've worked for that have embedded operations people into the developers' scrum.  They can't necessarily cut the same code, but they're part of the actual development of the application.  By the time the application gets to production, those guys are like, this is really cool.  And they become evangelists for the applications developed.  Not only do they understand the application better, they understand the metrics and the basic concept  the developers put into the thing, but they actually care about it.  I think that's a really important... I've seen too many applications that get thrown over the fence,  and you go, I don't really understand what this does, I don't understand why it breaks all the time,  I don't understand why it performs like shit.  But if you're actually involved from day one, you can go, well, we made a compromise here.  We actually decided to do it like this instead of this because of this particular reason.  And you go, we can make it better like this, this, this, but here's the caveats we had.  I think a lot of that sort of knowledge is not something that gets passed on otherwise.  When will your big and explicit book about Puppet will be finished finally?  So I'm hugely jet lagged, and that question has been asked 17 times since I landed on Belgian soil.  March or April, maybe? No.  Unfortunately, yeah, I think I spent about 20-hour days at the moment, and it's almost done.  Almost. Two chapters left.  And I had the help of a very good colleague who was pushing along.  Hello.  Obviously, one of the keys behind DevOps is a better collaboration between the development and the IT operations, guys.  Coming from the IT operations field yourself, what would you expect from a development team  for you to relate better to it?  What could the developers do to ease your way towards this collaboration?  So I talked about embedding operations people into Scrum or into Yakanban or whatever developers do.  Do it the other way around as well.  So drag development people into incidents and outages.  If you have an operation stand up, drag the developers in.  Make them understand the sort of problems you have.  A lot of developers operate on the principle that we've built something that's really cool,  that's really hot, and then we piss off down the pub and the operations people have to manage it.  If they actually understand that there are implications and consequences to their choices,  that if they choose a particular application architecture,  it's going to have an impact on the operations team at the end of the day.  The number of developers I have a conversation with who go,  oh, you didn't tell us that you don't do things like this or it's hard to package this  or this set of Java libraries are like this.  If you actually have teams involved on both sides from day one of a project,  there's a far greater appreciation for how that works.  Developers also, developers are like, you have a particular operations problem.  We spend to be fairly metric centric people, operations people.  We're interested in metrics, we're interested in measuring things.  Developers are also interested in that sort of thing,  and they also have a set of things they'd like to know.  When something goes wrong, something's performing badly,  they have that desire for that information.  If you have a conversation with a developer that says,  let me tell you about the metrics I have,  you can tell me about how your application is instrumented,  and then I can tell you what I'd like to have added to that instrumentation,  you can tell me what data you'd like from my end.  Those sort of conversations are really powerful,  and I found that a lot of developers are sort of stunned by the depth  to which operations people use metrics,  and they've sort of appreciated our perception of that.  So I guess there's some conversations around that that are sort of really healthy,  and it promotes that sort of cooperation.  So this sounds really interesting,  and it sounds like what I've done actually in some of the companies I've worked with.  It works really well, but does this really scale at all?  Does this scale to a bigger team,  and bigger teams of operations and development people?  Because that's one of the main problems when the teams become big enough,  it's really hard for them to mangle.  And the other thing is mostly if you've done this,  but I have this very weird observation that development people think,  for example, that CPU memory and hard drives are infinite,  and most of their resources are infinite,  and this is one of the main issues with deploying their software,  so if you've managed to teach anyone that, that's not exactly right.  I think the challenge, and you can see a lot of pushback around DevOps  and the fact that it works best in small teams, for example.  People say it's much easier to do this if you haven't got,  you're not an enterprise and you haven't got 500 people.  I think that's, it's true, it's easy to have relationships in small teams.  I think the larger teams, you do things like,  you move whole groups of people through different teams and different organizations.  You shuffle people around, you keep people actually learning new things,  you don't silo things.  I think that the key mistake people make is you build silos,  you build a group of people who are all the Java developers,  and you build another group of people that are all of the WebOps people,  and then you build another two, that is not only an unhealthy way of building teams,  but it's a way that doesn't promote that sort of cooperation and coordination.  He's cancelled his question.  Hi. I have a quick question over here.  When would be the exact perfect time to start doing DevOps when you're doing a startup?  When you're doing a beginning of a startup and you found it yourself,  you don't have that much capital.  I can't see why, for example, a whole bunch of things are useful,  but the whole configuration management and all that might just be a drag in the beginning,  because you really need to move quickly forward in order to get capital  and first clients and angel and whatever.  So how would you approach that one?  I think Greenfields is far easier.  If you have a Greenfields environment, it's far easier to introduce new concepts.  People don't have that it's not invented here,  or we've always done it this way,  mentality in a Greenfields sort of environment.  I think though there are real advantages to looking at something like DevOps in legacy environments.  So legacy environments, we've all got them.  We've all got lots of organizations have their collection of REL4 machines  that have been living there forever and they can't change or upgrade them.  They're a bunch of Windows NT machines that they've ring fenced off from the rest of the environment.  Those things cost lots of money to maintain,  because they're all sort of, that becomes sort of boutique or no one wants to change anything.  There's real opportunities in that sort of environment to actually,  if you automate stuff in those environments to save a lot of money and to become a lot more efficient,  but it's a lot harder to do because making those changes, systems like that,  tend to be sort of very fossilized.  There tend to be sort of things that have the organizations become risk adverse about.  We can't change that no one who works here understand how that works anymore.  But as a result, the opportunities there are quite large, but the risks are quite high.  But Greenfields is certainly easier.  If you're a startup from day one, the classic example is Agile.  If you start from day one using Agile, it's far easier to go from a legacy development process to an Agile  than going from a legacy to an Agile process.  If you start from scratch there.  I'm in operation myself.  And in the area, in this not uncommon situation where all the applications,  all the applications that we are deploying, we are not developed in-house.  So is DevOps still relevant?  And how do you trigger this cultural change about automating things?  Because we can only automate some baseline, basic things like SSH configuration,  VM provisioning, and very basic stuff.  All the rest is too diverse and too complex to do automatic deployment.  So I deal a lot with customers that have outsourced stuff to India and China.  And it's really interesting.  Outsourcing in a lot of cases gets a very bad name.  It's the lowest bidder who's outsourced, you outsource your stuff to,  and they have no interest in quality or changing things.  I think that's the wrong attitude to take.  The opportunity is there for you to infect those organizations.  If they understand, and there's a couple of people here,  I know there's one person here from India who works in a big Indian IT company  who actually can tell you that those guys are interested in learning new things.  They want to work on cool technology,  and if you have an opportunity to infect them with a whole different idea about how to do things,  I think those things will be picked up.  And there's big cultural barriers in any outsourcing arrangement,  but I think those are also opportunities.  And it's also about demonstrating value.  So if you can demonstrate that it's cheaper for the organization to do it a particular way,  then they're going to jump on that because they want to deliver the service and make a bigger profit.  If you can deliver, you get something out of that, you get better availability,  you get a better quality of service, you get a better SLA,  and they get to do it cheaper or faster or whatever it is.  So it's a green-win situation for both organizations.  Hi.  I luckily started two weeks ago,  and I'm responsible for implementing application monitoring  in a quite very big corporate Belgian environment,  which already says a lot for people to know about that.  And I'm facing the fact that the operation people have a solar system  to be able to participate in that,  but on the other hand, they're saying,  don't flood us with all your application-level events  and all the business process metrics that you would like to see on the dashboard for the business.  Do you have any comment on that, or do you just want to show some empathy?  So I'm not sure I understood the question, Tarlot, totally,  but I think, again, it's about demonstrating what...  So if you demonstrate that...  Was the question, if I'm paraphrased,  they're not interested in looking at the business metrics,  they're not interested in looking at the application metrics?  Well, the developers and the business are,  but the iOS people, they want to limit the number of events  that will go through their monitoring systems.  So what I'm trying to promote is to help them change their mind  in such a way that they accept that monitoring should go a bit higher level.  I think it is about demonstrating the value of it,  so demonstrating that monitoring the CPU, the memory, and the disk  is not what's important about the application.  What's important about the application is what business results it delivers.  You still want to know about the other stuff,  but if all of those metrics are perfect,  if all of those dashboards are perfect,  but the response time over here and the application doesn't do what it's supposed to do,  then it's not working.  So you need to demonstrate that if they have an underpicture of that,  they're necessarily the same metrics that all the dashboards and the business people look at,  but a view of that and how it links to those other metrics  that they care about, infrastructure metrics,  that's a way that you can say,  this is the picture your customers get, this is the picture that they're seeing,  and they pay your salaries, they pay all of us,  and we have to work together and solve in that particular problem,  or give that particular view.  I think it's probably...  I think there's time for one more question.  Any more questions here?  I don't see myself.  Thanks very much.
devops_More_than_Marketing_EjWlXcX-p4c_1_0_131.png devops_More_than_Marketing_EjWlXcX-p4c_1_132_179.png devops_More_than_Marketing_EjWlXcX-p4c_1_180_551.png devops_More_than_Marketing_EjWlXcX-p4c_1_552_647.png devops_More_than_Marketing_EjWlXcX-p4c_1_648_803.png devops_More_than_Marketing_EjWlXcX-p4c_1_804_887.png devops_More_than_Marketing_EjWlXcX-p4c_1_888_935.png devops_More_than_Marketing_EjWlXcX-p4c_1_936_1031.png devops_More_than_Marketing_EjWlXcX-p4c_1_1032_1259.png devops_More_than_Marketing_EjWlXcX-p4c_1_1260_1415.png devops_More_than_Marketing_EjWlXcX-p4c_1_1416_1799.png devops_More_than_Marketing_EjWlXcX-p4c_1_1800_1919.png devops_More_than_Marketing_EjWlXcX-p4c_1_1920_2471.png devops_More_than_Marketing_EjWlXcX-p4c_1_2472_2891.png devops_More_than_Marketing_EjWlXcX-p4c_1_2892_3851.png devops_More_than_Marketing_EjWlXcX-p4c_1_3852_4259.png devops_More_than_Marketing_EjWlXcX-p4c_1_4260_4775.png devops_More_than_Marketing_EjWlXcX-p4c_1_4776_5255.png devops_More_than_Marketing_EjWlXcX-p4c_1_5256_5531.png devops_More_than_Marketing_EjWlXcX-p4c_1_5532_7619.png devops_More_than_Marketing_EjWlXcX-p4c_1_7620_8207.png devops_More_than_Marketing_EjWlXcX-p4c_1_8208_8267.png devops_More_than_Marketing_EjWlXcX-p4c_1_8268_8327.png devops_More_than_Marketing_EjWlXcX-p4c_1_8328_8963.png devops_More_than_Marketing_EjWlXcX-p4c_1_8964_9095.png devops_More_than_Marketing_EjWlXcX-p4c_1_9096_9455.png devops_More_than_Marketing_EjWlXcX-p4c_1_9456_9863.png devops_More_than_Marketing_EjWlXcX-p4c_1_9864_9983.png devops_More_than_Marketing_EjWlXcX-p4c_1_9984_10247.png devops_More_than_Marketing_EjWlXcX-p4c_1_10248_10367.png devops_More_than_Marketing_EjWlXcX-p4c_1_10368_10691.png devops_More_than_Marketing_EjWlXcX-p4c_1_10692_10955.png devops_More_than_Marketing_EjWlXcX-p4c_1_10956_11771.png devops_More_than_Marketing_EjWlXcX-p4c_1_11772_12839.png devops_More_than_Marketing_EjWlXcX-p4c_1_12840_14147.png devops_More_than_Marketing_EjWlXcX-p4c_1_14148_20471.png devops_More_than_Marketing_EjWlXcX-p4c_1_20472_21743.png devops_More_than_Marketing_EjWlXcX-p4c_1_21744_22415.png devops_More_than_Marketing_EjWlXcX-p4c_1_22416_25727.png devops_More_than_Marketing_EjWlXcX-p4c_1_25728_26087.png devops_More_than_Marketing_EjWlXcX-p4c_1_26088_26159.png devops_More_than_Marketing_EjWlXcX-p4c_1_26160_26243.png devops_More_than_Marketing_EjWlXcX-p4c_1_26244_27143.png devops_More_than_Marketing_EjWlXcX-p4c_1_27144_27395.png devops_More_than_Marketing_EjWlXcX-p4c_1_27396_27647.png devops_More_than_Marketing_EjWlXcX-p4c_1_27648_27779.png devops_More_than_Marketing_EjWlXcX-p4c_1_27780_28295.png devops_More_than_Marketing_EjWlXcX-p4c_1_28296_29339.png devops_More_than_Marketing_EjWlXcX-p4c_1_29340_29555.png devops_More_than_Marketing_EjWlXcX-p4c_1_29556_29615.png devops_More_than_Marketing_EjWlXcX-p4c_1_29616_29699.png devops_More_than_Marketing_EjWlXcX-p4c_1_29700_29831.png devops_More_than_Marketing_EjWlXcX-p4c_1_29832_29951.png devops_More_than_Marketing_EjWlXcX-p4c_1_29952_30335.png devops_More_than_Marketing_EjWlXcX-p4c_1_30336_30695.png devops_More_than_Marketing_EjWlXcX-p4c_1_30696_30743.png devops_More_than_Marketing_EjWlXcX-p4c_1_30744_35723.png devops_More_than_Marketing_EjWlXcX-p4c_1_35724_41579.png devops_More_than_Marketing_EjWlXcX-p4c_1_41580_45179.png devops_More_than_Marketing_EjWlXcX-p4c_1_45180_45551.png devops_More_than_Marketing_EjWlXcX-p4c_1_45552_46799.png devops_More_than_Marketing_EjWlXcX-p4c_1_46800_49595.png devops_More_than_Marketing_EjWlXcX-p4c_1_49596_53291.png devops_More_than_Marketing_EjWlXcX-p4c_1_53292_54275.png devops_More_than_Marketing_EjWlXcX-p4c_1_54276_54323.png devops_More_than_Marketing_EjWlXcX-p4c_1_54324_54383.png devops_More_than_Marketing_EjWlXcX-p4c_1_54384_75239.png devops_More_than_Marketing_EjWlXcX-p4c_1_75240_75935.png devops_More_than_Marketing_EjWlXcX-p4c_1_75936_76057.png