How Ticketmaster are measuring and managing technical debt
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Tarry, Simon
“Software decays. Martin Fowler tells us that debt is inevitable. Even with the cleanest code in the world and the tightest dev practices and methodologies, technological change will still outpace you. The app you're working on today is already out of date. Apps depreciate. Infrastructure depreciates. Architecture depreciates. Just as Lehman brothers did, we all risk paying the ultimate price for unmanaged debt. Resurrected dead code took out Knight Capital, an investment house by racking up almost $1/2bn within hours. Ultimately debt management is a business decision - so how do we as IT professionals source and present the right data to influence the decision makers? Join Simon Tarry, engineering manager at TicketMaster for a journey of how to measure the unmeasurable. From the heart of the systems that provide massively scaling services for Olympic ticketing, to the cutting edge mobile technology that lets you choose the best seats, we'll discover how we got the right data to the right people to get maintenance on the map.”
 All right, everybody good? Okay, it's the last session on the Friday, so how are the energy levels? Yeah, good, good, great. Okay, so you went through the agenda and you saw this last thing on the Friday and you thought, ooh, wonder what that's about? The only reason I'm really here is because my boss said sort of this time last year that we need to try and crack this problem of technical debt because he'd interviewed about 150 engineers in Ticketmaster who work on all kinds of different bits of software and apparently one of the biggest things that people moaned about as developers was technical debt and now our leader of engineering in Ticketmaster is not a technical background, so that's quite unusual and he was like, I don't know what this thing is so you guys have to come and tell me. And being engineers, of course, we all had different ways of talking about or describing technical debt, so that was the first question to answer and going through this journey, we picked up a lot of interesting stuff, so that's kind of what I want to share with you guys is like, because let's be honest, it's probably the least sexy subject, right? It's not like some flashy new technology, it's something that most people think is desperately uncool, right? And yet you've come along to find out about it, so that's no reflection on you. That's maybe because you genuinely have a problem with it, so that's a good place to start. Can I just get a feeling for in the room who here is like a developer? Oh, wow, okay, brilliant. Any architects? Yeah, a few architects. Any hands-off people? One hands-off, great. Okay, so just know kind of what the audience is. So I put a blog together on this of what we'd done and very surprisingly it went viral and we got about 13,000 views of it. So it got picked up on Hacker News and if you read through it, there's a lot of stuff about the US guy's hating ticketmaster which apparently is quite common. But as an engineering professional, I will now say that from my point of view, this is now the single biggest problem that will affect you in your career. And it's like a debt that's not on a balance sheet anywhere, but it can destroy a business and we'll see an example of how technical debt destroyed a business a bit later on. And what I found, like I've been an engineering manager for a few years and I just bumped into a guy I used to manage which is quite fun, Hikers. And managing teams of developers is really fun because engineers are really interesting people and love doing things. But what I found from interviewing a lot of people is that the way this stuff is managed in companies is often done really, really badly. There's only a few places that I've come across now having gone into the subject that actually do this stuff well. And anyone remember what this was? Lehman Brothers, right. What happened to Lehman Brothers? They went bankrupt. Yeah, they were destroyed by bad debt, right? Not technical debt, but still debt. So we're just going to run through this real fast, right? So the core of the problem for Lehman was we had all these things, about 100% mortgages. Anybody sign up for 100% mortgage? Back in the day? Or 120% mortgage or whatever they were? In any case, people were selling, especially in the States, these kind of loans to people that didn't have like a regular income. But then these mortgages were sold on by Lehman to these credit rating agencies would look at these mortgage-backed securities and would say, you know, triple A stamp, even though some of those, because they've been packaged up, were, you know, with people who couldn't actually afford to pay their mortgages. But because all the house prices were going up and up and up, everyone was like, it's all good. So what happens is that the pension funds who sell you this vision of your future, that's you in old age, right? Yeah? That's what you're being sold. The pension funds can buy these mortgage-backed securities, which turned out to be, you know, a complete load of, I don't want to swear. But you get the picture, right? It wasn't built on anything solid. It was built on bad debt. Can come across Martin Fowler? Yes, of course. So Martin Fowler talks about different sources of technical debt. And a lot of the time, you may have heard these phrases while you're, you know, as a software engineer and professional developing or, you know, and hopefully with Agile, we're getting a bit better in terms of design or, you know, doing our best practices, right, in terms of making sure we've got our code properly tested. Just in the session with Dan North, anybody else in the session with Dan North just now? Right. Great guy. And, you know, and he was saying that we really need to take responsibility to make software engineering a profession, right? It's like only two generations old as a profession. It's a very immature profession. And so, you know, a lot of the stuff that gets built is actually crap, right? A lot of the stuff that gets built is crap. From a quality point of view, you know, and often what we struggle with is this last one, now we know we should have done it, right? Like there's this discovery phase of doing software where, you know, you don't know what's going to happen because you haven't got a good map of the landscape of where you're going, which is why it's so exciting, right? You're trying out some new technology to solve some problem no one else has ever solved before. And it's only at the end of that project you can say, hey, this is, you know, now we know how we should have done it. And so, you know, we also have these constraints that come from doing software because we have people shouting at us telling us that they want their features delivered more quickly. And if we think about it from the point of view of, you know, working with a product team, the product team probably don't really care too much about the debt that's being created, right? They care about their features going out on time so that they get their, you know, return on investment. And you know, we have to ship to a target. The business represented here by these attractive people, you know, we always say, we just termed the business. So everyone else use that to say the business, right? It's like these sort of mythical people that tell us what to do. Now they have no understanding, right, of the debt in our code base. I'm assuming that. That's why you're here. They don't care about the debt, they care about the bottom line, which, you know, we should care about too because that pays our wages. This is an actual picture that was on our wall in Ticketmaster that a developer put up there because he was so fed up with the amount of debt. He was like, I was spending 80% of my time writing tests, 20% writing code and something really wrong. And he only lasted three months, even though he was like the best developer I've come across because he was so fed up with it. And because that debt is hidden, again, it's not visible to investors, right? We can't make this visible. It's not on a balance sheet. We've got every other debt on a balance sheet financially, but not technical debt. And we can't expect, you know, our pension companies that are investing in companies, you know, they don't know. They don't know that this debt exists. So I said we were going to look at how technical debt killed a company. Anyone remember this? Who the company was? Night capital, right? 125,000 pounds a second. And this is, I really enjoyed researching this, right? Because what you look at, like over months you go on the register and you find out some great article about pulling together, you know, pulling it apart about how it really happened. And I always find that really fascinating. So we're going to just spend a few minutes looking at this because there's your forces, right, of, you know, just get this stuff done. And they have a month and a half to put some code in. That's all we need to really know here, that they had a month and a half to get this project out. And they're going to be going straight away on where the quality is going to suffer. And this new feature was intended to replace some unused code, which, you know, was still there. And they had intended to delete the old code. You can see where this is going, can't you? And the business made some quotes. So on the day when this actually happened. So this software went live with these new features. And suddenly it was starting to rapidly make millions and millions of trades, like these micro trades in very, very small iterations. But the problem was, you know, it wasn't a test environment. This was in a live environment. And what it was doing, it wasn't, you know, placing those trades where they should have been. So it was like buying too high or selling too low and, you know, losing them an insane amount of money. And the PR that was coming out of the business, first it was this. I love it. A very large software bug, if you could see the size of the bug. You know, that's how non-technical people view systems. Then it became an infrastructure problem. Yeah, blame the ops guys. Then more of a networking problem. So you could see the blame shifting. And yes, we need to do a better job on our testing environment. Well, yeah, duh. And obviously the implications, so reading through the deep dive, this is there, during the deployment of the new code, one of the technicians did not copy the new code to one of eight servers. So it was missing on one of the servers. None of this was reviewed, the deployment, so there's no written procedures. So there was no playbook of going through all the steps when you're releasing, which obviously you've all got in place, right? This eight server had this defective code, right? They hadn't been running for years and years and years. And in the middle of this disaster going on, they made it worse by uninstalling the new code so that that defective code was then replicated across all the other servers. So that's how to kill a company in what, an hour and a half? So, you know, and that's from the point of view of what Dan North was just talking about is professionalism, is about us doing our jobs and about making sure we've got good DevOps practices in place, about making sure that we can do this stuff well and not just hack it together because we're being pressurized by the business, right? And we'll come onto some of that, why that happens. Yeah, so now happy retirement. I looked at that. Okay, so hands up if you know the amount, the number of lines of code roughly in your platforms in your business. Yeah, any guess? Yeah, that'll do. 80,000 lines. Okay, so 80,000 lines of running code. Okay, okay, but across your company as a whole, would you have an idea? 8 million. Okay, anyone know how many Google have got? That's kind of out there. Two billion, two billion lines of code Google have got. Okay, how about number of servers you're running? Want to volunteer a number? Yeah, go on. Several thousand, several hundred. You can't say, okay, but you know, that's good. But what I'm saying is, okay, a lot of us don't know this stuff. As developers, we don't know this stuff. We don't measure it. Now, the thing is that the guy said to us, he was like, I want you to measure technical debt. I was like, okay, interesting challenge, right? Interesting challenge, how do we do this? So the first place I went to, obviously, you just go on to Google and say, measure technical debt, right? That's the first thing you do. And we came up with this tool, SonaCube, which is out there. Anyone using SonaCube? Okay, yeah, it's free. It pulls together a number of metrics on your code. It plugs nicely into GitHub. And it's really quite easy to install and get the hang of it. Go on. How? Yeah. You know, at the moment, we try to set in some of this stuff up and start to find the reaction. Oh, my God, it's so terrible. We can't start to do anything. Okay. I'll keep talking. All right. And if we'll see you over again. So what I loved about this was it gives you this dashboard view, which includes the number of technical debt days, which you can't actually see on this picture. So that's going to make you go on to the SonaCube site and look for it. It gives you a technical debt percentage, but it breaks it down like it uses a formula based on your code. And I was like, this is great. We can just get a nice dashboard view, give it to the exec, and we can look at the number of days. Okay. But, sorry. Oh, yeah, it's there. Yeah, you're right. Thanks. Great. Yeah, 40,000 days. That's quite a lot. I don't know how you want to measure that in numbers or in the data. Or in dollars. But yeah. So that's why I love this tool. But, you know, so we gave it to him and he said, well, is this going to tell me about that issue that happened last week when your virus software was out of date and we had a breach because that component wasn't up to date? Well, no, that's not going to do that. And I was like, but that's not technical debt either. Right. Technical debt is maintainability of application code. That's what it is. He said, well, that's not good enough. So I kind of realized that there's like engineering here and then there's the business here and there's this gap in between. And if he wasn't going to cross the bridge, then I had to. So I said, okay, fine. We'll call tell technical debt something that includes upgrading our components, even though we'll keep our maintainability because that's what as engineers, we know that it is. So we kind of had to compromise on it. And what we learned was, yeah, maintainability of application code, infrastructure debt. Okay. That's a term we started to use for this kind of stuff of like components in our infrastructure throughout the day. As in stuff that, you know, as developers, we didn't really care about it from a day-to-day point of view, but we should care about it because we need to know this stuff is up to date. So that's an ops thing. Ah, architecture debt. Now this is a big one, right? Because it's kind of you, anyone heard of the term architecture debt being used? One hand. Okay. Yeah. And there's no definition though, right? There's no way of defining it, but people are kind of conscious of the term. So what we did was, we ended up after a few months of going and going backwards and forwards and having internal engineering conference and debating this stuff out and think, what are we going to measure? We came up with something that works for us. And there are other models of tech debt out there. This just happened to work for us. It's not definitive. Some of this stuff we could pull out of tools. In fact, all of these things we could pull out of tools. Code coverage, complexity, performance and security. And on the infrastructure side, we looked at performance and scalability. So literally performance on the application side because most of our stuff is web-based. We're looking at time to interact just as a measure. And on the infrastructure side, we just looked at server response time and throughput. And our ability to scale up. That's as far as we could go with automated tooling. But we wanted to measure more stuff as well. We wanted an overall maintainability measure of our code. And we wanted to be able to look at stuff in our infrastructure like security, the state of our DR, so continuity. And a few other things that we've listed there. And then on the architecture side, we kind of bashed this out with the EA team. The most important one for them was deviation from reference architecture. So who on their platform has a reference architecture? Great. So what is a reference architecture? Okay, right. So it's how you would build it if you started today. Right, right. So that's kind of your ideal, right? That's what everybody wants. Everybody in this room wants what's now, what's ideal. You want the best. So that's what we said, okay. So how far away are we from that? Let's define what ideal is and then we can measure how far we are away from it. And those other things helped us as well. So that's just what works for us, obviously, for your context that would be different. Now, great book. Anybody read this book? No? Really, really great read. Well, the first few chapters, after the first few, it gets quite mathematical and I hate maths. So I was like, yeah, I'll take the first bit. But it was really interesting because he analyzed over decades how people in business make decisions compared to how we as engineers or more academic guys sort of do that kind of stuff. And from a business point of view, you're just trying to reduce the uncertainty around a decision. So it's not often in engineering we try and make things really precise or really accurate because that's kind of how machines work. But people just kind of want to slightly reduce the risk around something. So this was the rough principle we went off and kind of came up with this process of manual mapping. So we take a component diagram of our system. So here's just a simple component diagram for an e-commerce platform. And then what we would do with the development team is just sit in a room and we'd say, okay, each component just give us a, we picked this guy, one to five. So where one is low debt, five is high debt, yeah, or maintainability. Give us a number. So in a group, everyone would sit round and they'd on their own write down a number and then everyone, you know, like a poker planning session come up with, okay, four, five, four, five. Why is it? And then you'd kind of agree on a number. And the cool thing about this is that people who didn't know so much about one component would be able to share their knowledge about that component and how well it's structured and so on. And then you'd come up with, you know, a number for that component and say, this is how much debt. Then you could say, okay, we've agreed that number. Now what would we need to do to reduce the amount of debt in that component? Yeah, it's the four. We want to bring it to a two. What have we got to do? What work have we got to do? You know, go crazy. Just tell us everything. You know, don't hold back. It doesn't matter how big it is. And then we could prioritize that list and say, okay, if we're going to start today, what would you work on to, you know, reduce that debt? What would be the first thing we're going to do? So we come up with a prioritized list. Great. So that's the process we've applied for all those things that we couldn't measure. So in that model, all those things that we couldn't measure with a tool to get a number, we use this process. So with the ops guys every quarter now, we sit around in a room. We stick up their infrastructure diagram and we go through all the components and we say, okay, from a security point of view, where is this at? From a performance point of view, where is this at? And we go through that, come up with a list of work that we need doing, and then, you know, we've measured it and we've figured out what we need to do to reduce those issues. So we measured in Tickermaster International. We have five engineering teams across Europe and Canada. We've got four and a half million lines of code. So this is roughly where we sit. That's quite a lot less than Google's two billion. So still a significant amount. And what that's helped us to do is to then compare ourselves to industry standards and look at some more of those stats later on. But it's good to get a handle on how much you've gotten, how big the problem is. Okay, so what I realized as I was gathering all this data was, sheesh, we need some kind of way of a strategy for dealing with this. Because we could figure out that there's a load of debt in one platform or in a certain component of this platform. But if that component is really stable and doesn't need to change, then we don't care, right? We don't need to change that. So we don't care about the debt in it. So we had to figure out some way of saying, okay, this is what we really care about. This is what we need to change. And so we came up, or there are, these are different strategies for managing debt. So this was a survey done by on developers as to how much time people spend on technical debt. Maybe that reflects your experience. So very few spend a lot of time working on debt. And obviously trying to prevent the debt in the first place is really important. So what we came up with, which I can't show you the similar reasons as the gentleman over there, is we've got some greenfield projects. And by applying that model and measuring those greenfield projects, we could start to see the beginning, or sort of midway through last year, that some of the debt was creeping up in the greenfield stuff, right? And if you think about a startup, that's quite often the case. A startup just really needs to get, prove that business model, get those features out so they can prove that business works, and then they can go back in and deal with the debt that they've built up. But if you can bake those practices in from the start, like test-driven development, make sure your developers know they're solid principles, right? It's amazing how many developers I interviewed, not last year, the year before, and I say, okay, tell me about solid principles, and they'd be stuck, right? And that's people we've already screened from a good CV who couldn't tell me what solid code was. Kind of scary. So that's one way of doing it, is make sure it doesn't get in there in the first place. Another nice approach is this, that as you're going through your sprints, through your retrospective, is just collect what's built up in this sprint over these two weeks that we know is debt, put it onto a card, and then every few months, just have a sprint of burning that debt down. That's a really healthy way of dealing with it if your organization is in a good place. And then there's the strangler pattern. And then in the head of the strangler pattern, yep, couple of hands, a few hands. So anybody, here we go, Chris, sorry to pick on you. So I worked with Chris at OpenGI, an insurance company which had built software from the sort of mid, late 70s, early 80s, I think? 60s. Yeah. So, and they still had their core platform running, didn't they? The thing called Tripos, which, and they still had the guy in the corner who was probably what, 70 odd? You know, just a guy in the corner working on this core platform. And they tried to replace that system three times, and three times they'd failed with the big bang, let's rewrite it. So, and that's pretty much been my experience in every single company. Ticketmaster years ago started doing a big bang, let's rewrite it all. They've literally just chucked about two-thirds of that code away. So you know, big bangs tend not to work. So strangler is what's great because you can pick out those bits that you want, and you can slowly rewrite them as new services, you're not changing the services to your clients or your customers. So microservices. So who's built a microservice? Yay. So the learning from microservices, I believe, is don't make too many. Yeah? Learn from experience. Yeah. Because of the comms, the interplay between. Yeah. Yeah. Loads of network traffic. So don't go crazy with them. Okay. Is this a fair fight? It's getting there, isn't it? It's not one against. So the final part is really about trying to overwhelm the enemy, like bringing people from outside of engineering. So this is really, I think, what made the difference for us was the fact that our VP of tech was not a technical person, but he did know how to sell ideas into the exec team to the people who make the big decisions, and that's what we really need. And there was kind of this, right? We gathered all this data, but we only really need to present the right data to the execs to say, this is what data you need to be able to make decisions to say, let's do some maintenance on this platform, this is what maintenance we need to do, this is why we need to do it, if you don't do it, this is what's going to happen, this is what bad is going to happen. And that's what we put on a report that we created. And now this may look fairly dull to you. There's a line, it's red at one end, it's green at another, there's a couple of dots in the middle. But apparently for people who aren't technical, this is very sexy, I didn't realise this. They love looking at this kind of thing because they can straight away see where something is at. So this is what we did. We got a report produced which basically you can see straight away how much debt from all these categories that we put together is in a platform with some text alongside to say, this is what we need to do to fix it and if you don't do it, this is what is bad is going to happen, which is the sort of risk analysis. Then we put that into one big report, which some people seem to love reading through massive reports. But what the kind of cool thing was this rolled up score, so basically we could give each platform a score out of 100 to say, this platform is a 60, this platform is a 30, this platform is an 80. And so straight away, it's really visible to the business, the business, how much debt we're carrying as a business, which is really helpful. And in there as well, we could put a technical backlog to say, now we understand all the stuff we need to do to reduce our technical debt, there you go, there's a technical backlog. So it's now a fair fight. We've got more people involved in killing this monster. So what happened for us? Because we did this over the course of last year, we put out three of these reports, we did a bunch of measuring. So just a few interesting data points. This guy called Randall Jensen, anyone come across Randall Jensen? No, we had really done this book about software estimation. And he basically said that an engineer can handle about 25,000 lines of code. This isn't lines of code per day because that's a stupid metric, right? It's in terms of an overall code base, how much can one engineer really handle before they get completely overwhelmed? Now I worked out that Google, with two billion lines of code, about 25,000 engineers, they got about 80,000 lines of code per developer. Now maybe a bunch of that isn't used, but that's quite high. This guy was saying 25,000 is about right. We're slightly over that. It does mean we can say to the business people, hey, if you're worried about stuff coming out slowly, it's because we're trying to handle this much code compared to an industry average. So that gives us a lever to say, let's fix some of this stuff, right? Let's spend more time doing maintenance and less time on features just until we get this under control. And what we measured over last year, we could say at the end of the year, we've reduced our debt by 10%. Great. We've done so purely on how we've measured it. But that's mostly on the application side. That was most of the stuff we could fix quite easily. We didn't really tackle any of the architecture side of things, which is what the developers are really complaining about. I've got three data sources instead of one. Just give me one data source and an API, and I can test on that. Make things easy, make things simple. And it's that kind of thing that we're now trying to... Yeah, go on. Yes, but not as part of this program. So we've got a separate KPI program, which covers meantime to fail, meantime to react, and so on. So, yeah, that's in a separate area. But for anybody else's purpose, you could include whatever the hell you want in your own model. It's just a case of measuring stuff and trying to make better decisions. So is that good? Is that bad? We don't know. We can tell a story around it. We personally think we feel it's bad because we need to be getting through it quicker. We need to get that dealt with better and get it out of the way so that we can have more flexible platforms, we can deliver stuff more quickly. And that's me. So any other questions? Wait. Yeah, the waiting. Okay, did we wait the architecture stuff higher than the others? No. We've waited it completely flat to start with, but thinking about it now, arguably you could say definitely we'd want to wait the architecture stuff higher. But again, for us, we just feel it's a tool to be able to get more time to work on our platforms instead of doing features. So yeah, there's all sorts of things we could do like that. I'm not sure how to ask this, but in regards to technical debt, did any of the solutions that are removing personnel? Removing people, personnel? Okay, so the question is, would you look at removing actual personnel who are contributing debt to the system? Not that I'm aware of, but as part of trying to stop debt getting worse, we do have a separate basically training program about making sure that we're doing best practices, right? So yeah, can we fire people because they're writing crap code? No, I wish we could. But we're working on training people up to a certain level so that it doesn't get introduced. Yes, go on. Yeah, yeah. So the question is, is there a balance around the cost-benefit analysis of addressing debt against not addressing it? Yeah, and for me, that is absolutely one of the most relevant points because as engineers you could say, yeah, we totally need to re-engineer this thing, but if the outcome isn't, if it's only saving time for a team of five developers, 10% of their time, and it's going to cost a year to do it, absolutely. It's got to have value to doing it, and that is a key part of making this work. Yeah. Sorry, there's another one there. Yeah, so the question was, did we get pushback from developers who wanted to work on features rather than debt? I haven't come across, obviously, these teams are scattered around Europe, so I haven't been to all the teams, so I haven't come across it. What more, the case is that because we've got more legacy systems, is that developers would rather be sorting the debt out than contributing to it by adding more features to an already debt-ridden system, if that makes sense. But I totally appreciate that that could happen. So I'd probably be out rotating the work around so that no one's stuck on it for too long, I guess. Right, so the question is, do the developers support the code in production? So yeah, we do have a DevOps program we've been running for a couple of years. We are starting to move to that, so some platforms are better at it than others, and we do have automation engineers, DevOps engineers, who are sort of trying to fill that space, but it's definitely a mind-shift trying to get developers to actually care about their code in production. But that's, yeah, thankfully DevOps has come along to give us that awareness, right? Great, well, have a nice weekend, everyone, and safe journeys and all that. I hope you had a good week. Thanks, and long ride, great stuff. All right. I'll see you guys later. Bye. Goodbye.
 All right, everybody good?  Okay, it's the last session on the Friday, so how are the energy levels?  Yeah, good, good, great.  Okay, so you went through the agenda and you saw this last thing on the Friday and you  thought, ooh, wonder what that's about?  The only reason I'm really here is because my boss said sort of this time last year  that we need to try and crack this problem of technical debt because he'd interviewed  about 150 engineers in Ticketmaster who work on all kinds of different bits of software  and apparently one of the biggest things that people moaned about as developers was technical  debt and now our leader of engineering in Ticketmaster is not a technical background,  so that's quite unusual and he was like, I don't know what this thing is so you guys  have to come and tell me.  And being engineers, of course, we all had different ways of talking about or describing  technical debt, so that was the first question to answer and going through this journey,  we picked up a lot of interesting stuff, so that's kind of what I want to share with  you guys is like, because let's be honest, it's probably the least sexy subject, right?  It's not like some flashy new technology, it's something that most people think is desperately  uncool, right?  And yet you've come along to find out about it, so that's no reflection on you.  That's maybe because you genuinely have a problem with it, so that's a good place to  start.  Can I just get a feeling for in the room who here is like a developer?  Oh, wow, okay, brilliant.  Any architects?  Yeah, a few architects.  Any hands-off people?  One hands-off, great.  Okay, so just know kind of what the audience is.  So I put a blog together on this of what we'd done and very surprisingly it went viral and  we got about 13,000 views of it.  So it got picked up on Hacker News and if you read through it, there's a lot of stuff  about the US guy's hating ticketmaster which apparently is quite common.  But as an engineering professional, I will now say that from my point of view, this is  now the single biggest problem that will affect you in your career.  And it's like a debt that's not on a balance sheet anywhere, but it can destroy a business  and we'll see an example of how technical debt destroyed a business a bit later on.  And what I found, like I've been an engineering manager for a few years and I just bumped into  a guy I used to manage which is quite fun, Hikers.  And managing teams of developers is really fun because engineers are really interesting  people and love doing things.  But what I found from interviewing a lot of people is that the way this stuff is managed  in companies is often done really, really badly.  There's only a few places that I've come across now having gone into the subject that actually  do this stuff well.  And anyone remember what this was?  Lehman Brothers, right.  What happened to Lehman Brothers?  They went bankrupt.  Yeah, they were destroyed by bad debt, right?  Not technical debt, but still debt.  So we're just going to run through this real fast, right?  So the core of the problem for Lehman was we had all these things, about 100% mortgages.  Anybody sign up for 100% mortgage?  Back in the day?  Or 120% mortgage or whatever they were?  In any case, people were selling, especially in the States, these kind of loans to people  that didn't have like a regular income.  But then these mortgages were sold on by Lehman to these credit rating agencies would look  at these mortgage-backed securities and would say, you know, triple A stamp, even though  some of those, because they've been packaged up, were, you know, with people who couldn't  actually afford to pay their mortgages.  But because all the house prices were going up and up and up, everyone was like, it's  all good.  So what happens is that the pension funds who sell you this vision of your future, that's  you in old age, right?  Yeah?  That's what you're being sold.  The pension funds can buy these mortgage-backed securities, which turned out to be, you know,  a complete load of, I don't want to swear.  But you get the picture, right?  It wasn't built on anything solid.  It was built on bad debt.  Can come across Martin Fowler?  Yes, of course.  So Martin Fowler talks about different sources of technical debt.  And a lot of the time, you may have heard these phrases while you're, you know, as a  software engineer and professional developing or, you know, and hopefully with Agile, we're  getting a bit better in terms of design or, you know, doing our best practices, right,  in terms of making sure we've got our code properly tested.  Just in the session with Dan North, anybody else in the session with Dan North just now?  Right.  Great guy.  And, you know, and he was saying that we really need to take responsibility to make software  engineering a profession, right?  It's like only two generations old as a profession.  It's a very immature profession.  And so, you know, a lot of the stuff that gets built is actually crap, right?  A lot of the stuff that gets built is crap.  From a quality point of view, you know, and often what we struggle with is this last one,  now we know we should have done it, right?  Like there's this discovery phase of doing software where, you know, you don't know what's  going to happen because you haven't got a good map of the landscape of where you're going,  which is why it's so exciting, right?  You're trying out some new technology to solve some problem no one else has ever solved before.  And it's only at the end of that project you can say, hey, this is, you know, now we know  how we should have done it.  And so, you know, we also have these constraints that come from doing software because we have  people shouting at us telling us that they want their features delivered more quickly.  And if we think about it from the point of view of, you know, working with a product  team, the product team probably don't really care too much about the debt that's being  created, right?  They care about their features going out on time so that they get their, you know, return  on investment.  And you know, we have to ship to a target.  The business represented here by these attractive people, you know, we always say, we just termed  the business.  So everyone else use that to say the business, right?  It's like these sort of mythical people that tell us what to do.  Now they have no understanding, right, of the debt in our code base.  I'm assuming that.  That's why you're here.  They don't care about the debt, they care about the bottom line, which, you know, we  should care about too because that pays our wages.  This is an actual picture that was on our wall in Ticketmaster that a developer put  up there because he was so fed up with the amount of debt.  He was like, I was spending 80% of my time writing tests, 20% writing code and something  really wrong.  And he only lasted three months, even though he was like the best developer I've come  across because he was so fed up with it.  And because that debt is hidden, again, it's not visible to investors, right?  We can't make this visible.  It's not on a balance sheet.  We've got every other debt on a balance sheet financially, but not technical debt.  And we can't expect, you know, our pension companies that are investing in companies,  you know, they don't know.  They don't know that this debt exists.  So I said we were going to look at how technical debt killed a company.  Anyone remember this?  Who the company was?  Night capital, right?  125,000 pounds a second.  And this is, I really enjoyed researching this, right?  Because what you look at, like over months you go on the register and you find out some  great article about pulling together, you know, pulling it apart about how it really happened.  And I always find that really fascinating.  So we're going to just spend a few minutes looking at this because there's your forces,  right, of, you know, just get this stuff done.  And they have a month and a half to put some code in.  That's all we need to really know here, that they had a month and a half to get this project  out.  And they're going to be going straight away on where the quality is going to suffer.  And this new feature was intended to replace some unused code, which, you know, was still  there.  And they had intended to delete the old code.  You can see where this is going, can't you?  And the business made some quotes.  So on the day when this actually happened.  So this software went live with these new features.  And suddenly it was starting to rapidly make millions and millions of trades, like these  micro trades in very, very small iterations.  But the problem was, you know, it wasn't a test environment.  This was in a live environment.  And what it was doing, it wasn't, you know, placing those trades where they should have  been.  So it was like buying too high or selling too low and, you know, losing them an insane  amount of money.  And the PR that was coming out of the business, first it was this.  I love it.  A very large software bug, if you could see the size of the bug.  You know, that's how non-technical people view systems.  Then it became an infrastructure problem.  Yeah, blame the ops guys.  Then more of a networking problem.  So you could see the blame shifting.  And yes, we need to do a better job on our testing environment.  Well, yeah, duh.  And obviously the implications, so reading through the deep dive, this is there, during  the deployment of the new code, one of the technicians did not copy the new code to one  of eight servers.  So it was missing on one of the servers.  None of this was reviewed, the deployment, so there's no written procedures.  So there was no playbook of going through all the steps when you're releasing, which  obviously you've all got in place, right?  This eight server had this defective code, right?  They hadn't been running for years and years and years.  And in the middle of this disaster going on, they made it worse by uninstalling the new  code so that that defective code was then replicated across all the other servers.  So that's how to kill a company in what, an hour and a half?  So, you know, and that's from the point of view of what Dan North was just talking about  is professionalism, is about us doing our jobs and about making sure we've got good  DevOps practices in place, about making sure that we can do this stuff well and not just  hack it together because we're being pressurized by the business, right?  And we'll come onto some of that, why that happens.  Yeah, so now happy retirement.  I looked at that.  Okay, so hands up if you know the amount, the number of lines of code roughly in your  platforms in your business.  Yeah, any guess?  Yeah, that'll do.  80,000 lines.  Okay, so 80,000 lines of running code.  Okay, okay, but across your company as a whole, would you have an idea?  8 million.  Okay, anyone know how many Google have got?  That's kind of out there.  Two billion, two billion lines of code Google have got.  Okay, how about number of servers you're running?  Want to volunteer a number?  Yeah, go on.  Several thousand, several hundred.  You can't say, okay, but you know, that's good.  But what I'm saying is, okay, a lot of us don't know this stuff.  As developers, we don't know this stuff.  We don't measure it.  Now, the thing is that the guy said to us, he was like, I want you to measure technical  debt.  I was like, okay, interesting challenge, right?  Interesting challenge, how do we do this?  So the first place I went to, obviously, you just go on to Google and say, measure technical  debt, right?  That's the first thing you do.  And we came up with this tool, SonaCube, which is out there.  Anyone using SonaCube?  Okay, yeah, it's free.  It pulls together a number of metrics on your code.  It plugs nicely into GitHub.  And it's really quite easy to install and get the hang of it.  Go on.  How?  Yeah.  You know, at the moment, we try to set in some of this stuff up and start to find the  reaction.  Oh, my God, it's so terrible.  We can't start to do anything.  Okay.  I'll keep talking.  All right.  And if we'll see you over again.  So what I loved about this was it gives you this dashboard view, which includes the number  of technical debt days, which you can't actually see on this picture.  So that's going to make you go on to the SonaCube site and look for it.  It gives you a technical debt percentage, but it breaks it down like it uses a formula  based on your code.  And I was like, this is great.  We can just get a nice dashboard view, give it to the exec, and we can look at the number  of days.  Okay.  But, sorry.  Oh, yeah, it's there.  Yeah, you're right.  Thanks.  Great.  Yeah, 40,000 days.  That's quite a lot.  I don't know how you want to measure that in numbers or in the data.  Or in dollars.  But yeah.  So that's why I love this tool.  But, you know, so we gave it to him and he said, well, is this going to tell me about  that issue that happened last week when your virus software was out of date and we had  a breach because that component wasn't up to date?  Well, no, that's not going to do that.  And I was like, but that's not technical debt either.  Right.  Technical debt is maintainability of application code.  That's what it is.  He said, well, that's not good enough.  So I kind of realized that there's like engineering here and then there's the business here and  there's this gap in between.  And if he wasn't going to cross the bridge, then I had to.  So I said, okay, fine.  We'll call tell technical debt something that includes upgrading our components, even though  we'll keep our maintainability because that's what as engineers, we know that it is.  So we kind of had to compromise on it.  And what we learned was, yeah, maintainability of application code, infrastructure debt.  Okay.  That's a term we started to use for this kind of stuff of like components in our infrastructure  throughout the day.  As in stuff that, you know, as developers, we didn't really care about it from a day-to-day  point of view, but we should care about it because we need to know this stuff is up to  date.  So that's an ops thing.  Ah, architecture debt.  Now this is a big one, right?  Because it's kind of you, anyone heard of the term architecture debt being used?  One hand.  Okay.  Yeah.  And there's no definition though, right?  There's no way of defining it, but people are kind of conscious of the term.  So what we did was, we ended up after a few months of going and going backwards and forwards  and having internal engineering conference and debating this stuff out and think, what  are we going to measure?  We came up with something that works for us.  And there are other models of tech debt out there.  This just happened to work for us.  It's not definitive.  Some of this stuff we could pull out of tools.  In fact, all of these things we could pull out of tools.  Code coverage, complexity, performance and security.  And on the infrastructure side, we looked at performance and scalability.  So literally performance on the application side because most of our stuff is web-based.  We're looking at time to interact just as a measure.  And on the infrastructure side, we just looked at server response time and throughput.  And our ability to scale up.  That's as far as we could go with automated tooling.  But we wanted to measure more stuff as well.  We wanted an overall maintainability measure of our code.  And we wanted to be able to look at stuff in our infrastructure like security, the state  of our DR, so continuity.  And a few other things that we've listed there.  And then on the architecture side, we kind of bashed this out with the EA team.  The most important one for them was deviation from reference architecture.  So who on their platform has a reference architecture?  Great.  So what is a reference architecture?  Okay, right.  So it's how you would build it if you started today.  Right, right.  So that's kind of your ideal, right?  That's what everybody wants.  Everybody in this room wants what's now, what's ideal.  You want the best.  So that's what we said, okay.  So how far away are we from that?  Let's define what ideal is and then we can measure how far we are away from it.  And those other things helped us as well.  So that's just what works for us, obviously, for your context that would be different.  Now, great book.  Anybody read this book?  No?  Really, really great read.  Well, the first few chapters, after the first few, it gets quite mathematical and I hate  maths.  So I was like, yeah, I'll take the first bit.  But it was really interesting because he analyzed over decades how people in business make decisions  compared to how we as engineers or more academic guys sort of do that kind of stuff.  And from a business point of view, you're just trying to reduce the uncertainty around  a decision.  So it's not often in engineering we try and make things really precise or really accurate  because that's kind of how machines work.  But people just kind of want to slightly reduce the risk around something.  So this was the rough principle we went off and kind of came up with this process of manual  mapping.  So we take a component diagram of our system.  So here's just a simple component diagram for an e-commerce platform.  And then what we would do with the development team is just sit in a room and we'd say,  okay, each component just give us a, we picked this guy, one to five.  So where one is low debt, five is high debt, yeah, or maintainability.  Give us a number.  So in a group, everyone would sit round and they'd on their own write down a number and  then everyone, you know, like a poker planning session come up with, okay, four, five, four,  five.  Why is it?  And then you'd kind of agree on a number.  And the cool thing about this is that people who didn't know so much about one component  would be able to share their knowledge about that component and how well it's structured  and so on.  And then you'd come up with, you know, a number for that component and say, this is how much  debt.  Then you could say, okay, we've agreed that number.  Now what would we need to do to reduce the amount of debt in that component?  Yeah, it's the four.  We want to bring it to a two.  What have we got to do?  What work have we got to do?  You know, go crazy.  Just tell us everything.  You know, don't hold back.  It doesn't matter how big it is.  And then we could prioritize that list and say, okay, if we're going to start today,  what would you work on to, you know, reduce that debt?  What would be the first thing we're going to do?  So we come up with a prioritized list.  Great.  So that's the process we've applied for all those things that we couldn't measure.  So in that model, all those things that we couldn't measure with a tool to get a number,  we use this process.  So with the ops guys every quarter now, we sit around in a room.  We stick up their infrastructure diagram and we go through all the components and we say,  okay, from a security point of view, where is this at?  From a performance point of view, where is this at?  And we go through that, come up with a list of work that we need doing, and then, you  know, we've measured it and we've figured out what we need to do to reduce those issues.  So we measured in Tickermaster International.  We have five engineering teams across Europe and Canada.  We've got four and a half million lines of code.  So this is roughly where we sit.  That's quite a lot less than Google's two billion.  So still a significant amount.  And what that's helped us to do is to then compare ourselves to industry standards and  look at some more of those stats later on.  But it's good to get a handle on how much you've gotten, how big the problem is.  Okay, so what I realized as I was gathering all this data was, sheesh, we need some kind  of way of a strategy for dealing with this.  Because we could figure out that there's a load of debt in one platform or in a certain  component of this platform.  But if that component is really stable and doesn't need to change, then we don't care,  right?  We don't need to change that.  So we don't care about the debt in it.  So we had to figure out some way of saying, okay, this is what we really care about.  This is what we need to change.  And so we came up, or there are, these are different strategies for managing debt.  So this was a survey done by on developers as to how much time people spend on technical  debt.  Maybe that reflects your experience.  So very few spend a lot of time working on debt.  And obviously trying to prevent the debt in the first place is really important.  So what we came up with, which I can't show you the similar reasons as the gentleman over  there, is we've got some greenfield projects.  And by applying that model and measuring those greenfield projects, we could start to see  the beginning, or sort of midway through last year, that some of the debt was creeping  up in the greenfield stuff, right?  And if you think about a startup, that's quite often the case.  A startup just really needs to get, prove that business model, get those features out  so they can prove that business works, and then they can go back in and deal with the  debt that they've built up.  But if you can bake those practices in from the start, like test-driven development, make  sure your developers know they're solid principles, right?  It's amazing how many developers I interviewed, not last year, the year before, and I say,  okay, tell me about solid principles, and they'd be stuck, right?  And that's people we've already screened from a good CV who couldn't tell me what solid  code was.  Kind of scary.  So that's one way of doing it, is make sure it doesn't get in there in the first place.  Another nice approach is this, that as you're going through your sprints, through your retrospective,  is just collect what's built up in this sprint over these two weeks that we know is debt,  put it onto a card, and then every few months, just have a sprint of burning that debt down.  That's a really healthy way of dealing with it if your organization is in a good place.  And then there's the strangler pattern.  And then in the head of the strangler pattern, yep, couple of hands, a few hands.  So anybody, here we go, Chris, sorry to pick on you.  So I worked with Chris at OpenGI, an insurance company which had built software from the  sort of mid, late 70s, early 80s, I think?  60s.  Yeah.  So, and they still had their core platform running, didn't they?  The thing called Tripos, which, and they still had the guy in the corner who was probably  what, 70 odd?  You know, just a guy in the corner working on this core platform.  And they tried to replace that system three times, and three times they'd failed with  the big bang, let's rewrite it.  So, and that's pretty much been my experience in every single company.  Ticketmaster years ago started doing a big bang, let's rewrite it all.  They've literally just chucked about two-thirds of that code away.  So you know, big bangs tend not to work.  So strangler is what's great because you can pick out those bits that you want, and you  can slowly rewrite them as new services, you're not changing the services to your clients  or your customers.  So microservices.  So who's built a microservice?  Yay.  So the learning from microservices, I believe, is don't make too many.  Yeah?  Learn from experience.  Yeah.  Because of the comms, the interplay between.  Yeah.  Yeah.  Loads of network traffic.  So don't go crazy with them.  Okay.  Is this a fair fight?  It's getting there, isn't it?  It's not one against.  So the final part is really about trying to overwhelm the enemy, like bringing people  from outside of engineering.  So this is really, I think, what made the difference for us was the fact that our VP  of tech was not a technical person, but he did know how to sell ideas into the exec team  to the people who make the big decisions, and that's what we really need.  And there was kind of this, right?  We gathered all this data, but we only really need to present the right data to the execs  to say, this is what data you need to be able to make decisions to say, let's do some maintenance  on this platform, this is what maintenance we need to do, this is why we need to do  it, if you don't do it, this is what's going to happen, this is what bad is going to happen.  And that's what we put on a report that we created.  And now this may look fairly dull to you.  There's a line, it's red at one end, it's green at another, there's a couple of dots  in the middle.  But apparently for people who aren't technical, this is very sexy, I didn't realise this.  They love looking at this kind of thing because they can straight away see where something  is at.  So this is what we did.  We got a report produced which basically you can see straight away how much debt from all  these categories that we put together is in a platform with some text alongside to say,  this is what we need to do to fix it and if you don't do it, this is what is bad is going  to happen, which is the sort of risk analysis.  Then we put that into one big report, which some people seem to love reading through massive  reports.  But what the kind of cool thing was this rolled up score, so basically we could give each  platform a score out of 100 to say, this platform is a 60, this platform is a 30, this platform  is an 80.  And so straight away, it's really visible to the business, the business, how much debt  we're carrying as a business, which is really helpful.  And in there as well, we could put a technical backlog to say, now we understand all the  stuff we need to do to reduce our technical debt, there you go, there's a technical backlog.  So it's now a fair fight.  We've got more people involved in killing this monster.  So what happened for us?  Because we did this over the course of last year, we put out three of these reports, we  did a bunch of measuring.  So just a few interesting data points.  This guy called Randall Jensen, anyone come across Randall Jensen?  No, we had really done this book about software estimation.  And he basically said that an engineer can handle about 25,000 lines of code.  This isn't lines of code per day because that's a stupid metric, right?  It's in terms of an overall code base, how much can one engineer really handle before  they get completely overwhelmed?  Now I worked out that Google, with two billion lines of code, about 25,000 engineers, they  got about 80,000 lines of code per developer.  Now maybe a bunch of that isn't used, but that's quite high.  This guy was saying 25,000 is about right.  We're slightly over that.  It does mean we can say to the business people, hey, if you're worried about stuff coming  out slowly, it's because we're trying to handle this much code compared to an industry average.  So that gives us a lever to say, let's fix some of this stuff, right?  Let's spend more time doing maintenance and less time on features just until we get this  under control.  And what we measured over last year, we could say at the end of the year, we've reduced  our debt by 10%.  Great.  We've done so purely on how we've measured it.  But that's mostly on the application side.  That was most of the stuff we could fix quite easily.  We didn't really tackle any of the architecture side of things, which is what the developers  are really complaining about.  I've got three data sources instead of one.  Just give me one data source and an API, and I can test on that.  Make things easy, make things simple.  And it's that kind of thing that we're now trying to...  Yeah, go on.  Yes, but not as part of this program.  So we've got a separate KPI program, which covers meantime to fail, meantime to react,  and so on.  So, yeah, that's in a separate area.  But for anybody else's purpose, you could include whatever the hell you want in your  own model.  It's just a case of measuring stuff and trying to make better decisions.  So is that good?  Is that bad?  We don't know.  We can tell a story around it.  We personally think we feel it's bad because we need to be getting through it quicker.  We need to get that dealt with better and get it out of the way so that we can have  more flexible platforms, we can deliver stuff more quickly.  And that's me.  So any other questions?  Wait.  Yeah, the waiting.  Okay, did we wait the architecture stuff higher than the others?  No.  We've waited it completely flat to start with, but thinking about it now, arguably you could  say definitely we'd want to wait the architecture stuff higher.  But again, for us, we just feel it's a tool to be able to get more time to work on our  platforms instead of doing features.  So yeah, there's all sorts of things we could do like that.  I'm not sure how to ask this, but in regards to technical debt, did any of the solutions  that are removing personnel?  Removing people, personnel?  Okay, so the question is, would you look at removing actual personnel who are contributing  debt to the system?  Not that I'm aware of, but as part of trying to stop debt getting worse, we do have a separate  basically training program about making sure that we're doing best practices, right?  So yeah, can we fire people because they're writing crap code?  No, I wish we could.  But we're working on training people up to a certain level so that it doesn't get introduced.  Yes, go on.  Yeah, yeah.  So the question is, is there a balance around the cost-benefit analysis of addressing debt  against not addressing it?  Yeah, and for me, that is absolutely one of the most relevant points because as engineers  you could say, yeah, we totally need to re-engineer this thing, but if the outcome isn't, if  it's only saving time for a team of five developers, 10% of their time, and it's going to cost  a year to do it, absolutely.  It's got to have value to doing it, and that is a key part of making this work.  Yeah.  Sorry, there's another one there.  Yeah, so the question was, did we get pushback from developers who wanted to work on features  rather than debt?  I haven't come across, obviously, these teams are scattered around Europe, so I haven't  been to all the teams, so I haven't come across it.  What more, the case is that because we've got more legacy systems, is that developers  would rather be sorting the debt out than contributing to it by adding more features  to an already debt-ridden system, if that makes sense.  But I totally appreciate that that could happen.  So I'd probably be out rotating the work around so that no one's stuck on it for too  long, I guess.  Right, so the question is, do the developers support the code in production?  So yeah, we do have a DevOps program we've been running for a couple of years.  We are starting to move to that, so some platforms are better at it than others, and we do have  automation engineers, DevOps engineers, who are sort of trying to fill that space, but  it's definitely a mind-shift trying to get developers to actually care about their code  in production.  But that's, yeah, thankfully DevOps has come along to give us that awareness, right?  Great, well, have a nice weekend, everyone, and safe journeys and all that.  I hope you had a good week.  Thanks, and long ride, great stuff.  All right.  I'll see you guys later.  Bye.  Goodbye.
How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_0_59.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_60_179.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_180_3503.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_3504_4931.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_4932_5411.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_5412_7415.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_7416_7487.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_7488_9503.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_9504_10355.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_10356_12287.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_12288_12395.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_12396_13127.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_13128_14135.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_14136_16895.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_16896_17723.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_17724_17795.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_17796_17879.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_17880_19739.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_19740_20243.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_20244_22007.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_22008_23675.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_23676_23747.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_23748_23807.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_23808_24827.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_24828_25535.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_25536_26171.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_26172_28415.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_28416_29651.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_29652_29699.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_29700_29975.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_29976_32687.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_32688_33587.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_33588_34535.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_34536_34583.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_34584_36599.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_36600_36659.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_36660_37331.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_37332_37379.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_37380_40151.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_40152_40535.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_40536_41147.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_41148_41687.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_41688_42935.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_42936_42983.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_42984_43031.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_43032_43079.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_43080_44051.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_44052_44219.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_44220_44315.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_44316_44483.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_44484_44531.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_44532_46295.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_46296_48983.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_48984_56039.png How_Ticketmaster_are_measuring_and_managing_technical_debt_-_Simon_Tarry_HD_WEB_56040_56074.png