'Acing' Infrastructure Testing with Chef
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
O'Connell, Brian
Sports move fast; infrastructure testing has to keep pace. The team responsible for the cloud that powers Wimbledon, The US Open and some of the world's largest sporting events uses Chef to manage their infrastructure. Developing a testing framework presented many challenges, including: Scaling infrastructure testing to handle a distributed enterprise. Testing in remote locations without the fastest internet speeds. Contending with the Great Firewall in China. The team has spent 100s of hours optimizing test environments to support fast testing by distributed teams. Learn how the team approached test optimization, what worked and what failed.
 So to get started here, this is my own personal redemption story. It's about the mistakes I made when transitioning our environment to a test-driven infrastructure and what we have done so far to resolve them. But before I dive into that, what exactly does my team at IBM do? We're known for sports. We do the Australian Open, we do Roland Garros, or the French Open, which is going on right now, Wimbledon, the US Open. We also do a large golf tournament in Georgia. But it's not just sports. We also do IBM.com and all of the digital properties that comprise the IBM.com portfolio. And we have this focus on what we call continuous availability. In fact, we've been running IBM.com with zero outages and zero downtime since June of 2001. And in that time span, we actually moved data centers 18 times with zero seconds of downtime. And we do this across a hybrid cloud. We have three private data centers and four public cloud regions in the IBM cloud that we serve these digital properties off of. And we do it with a large team. We have 80 plus people in eight different time zones in 35 work locations. And so we need these tools and methods to let us work independently, yet collaborate when necessary. And our workload is highly mission critical. Our partners and teams spend 50 weeks a year preparing for two or less weeks. There's no, oh, we'll fix this next week. That just doesn't happen. They're going to hold the US Open, whether we're ready or not. And we have to be ready. And not only is these properties in this time span on the world stage, it's also when they derive most of their revenue. So we have to deliver flawlessly. Now here was the mistake I made. When I was doing this transformation, I went out there and I surveyed the landscape. And I looked at all the best practices. I said, OK, we're going to move this stuff into Git. We're going to do code reviews. We're going to have a delivery pipeline. We're going to do infrastructure as code. We're going to do local testing. And I made all these changes at once and did not go over very well. It was a lot of change to push through all at once. And the biggest problem we had was that simple tasks that used to take three minutes were now taking hours. It used to be I'd log into the server and I'd edit the file and I move on to the next thing. And now doing a Git clone, editing the file, committing it. It's going to a code review, building it locally. And it's taking hours. So a three-minute change was now taking an hour to do. And it was this combination of these best practices, which are great practices and other challenges within our organization and our structure that was causing these changes to take a long time. The other problems that we faced was that we have to put out a lot of enterprise software. It's part of our platform. And some of the software is very big. We have software that's over a gigabyte in size. And if you're working locally and you're trying to do a vagrant up or a test kitchen and test that and you're pulling down a one gigabyte package every time, that can take a while. And our team is distributed. We have people all over the world. We have a large part of our team is in China. We have people in India. We have people all over the United States and Europe. And they're not all sitting right next to the source of those binary artifacts. So download time is very important. And they're in remote locations. Either they live in a remote location possibly or we send them to one. If we send them to the US Open, there's a fixed amount of bandwidth available for you to download things constantly and test them. And the biggest feedback I got was that this combination of factors was resulting in changes that you take minutes taking hours. And I thought, you know, I brought this problem onto the team. So I decided this was my problem to work really hard to resolve. So we had to find a solution. And our solution was we moved all of our testing away from local laptops to the cloud. In particular, we're on the IBM cloud. And we use a service that used to be called Blue Box. It's now IBM Private Cloud as a service. That's an open stack based cloud. And to do this, we used the kitchen open stack plugin to let us spin up test machines on the cloud and test our changes out there instead of locally. Now we're still developing locally. We're still doing the Git polls and the code reviews. But the actual execution of all of our testing takes place now in the IBM cloud. And we're able to do this and make this transition easy because how we manage everyone's laptops. When you come to our team and you start to work on Chef, you essentially run a curl bash. And from that point on, we take over the management of the development tool in your laptop. So we install Git at the right level. We install Vagrant, VirtualBox, ChefDK, all the things that you need to be able to get your environment going at the right level. And this is great because it lets us make changes easily and transition things like local testing to cloud testing without a lot of impact or breakage. And so we introduced this thing we call KCloud. This was our effort to let you have your local testing and also bring in this cloud testing. And what it really was was a gem that we wrote called KCloud Convert. And it did a couple of things for us. First thing it does is it reads in your kitchen yaml. And I should back up a little bit. That KCloud Convert, we have access to everyone's bash RC. And so we add it to their environment. So instead of typing kitchen to do a test, it's just KCloud instead. And then under the covers, we read in the kitchen yaml. We validate that they're only testing Red Hat Enterprise Linux. And that's an implementation detail we have that we're moving all our stuff to that platform. So if you're on a different platform, you still test locally. But in our platform for our stuff, it's all Red Hat. So we just validate that. We check the includes, excludes that you might have because you could be only testing against a version of Red Hat 6 or Red Hat 7. And we want to keep that across the cloud and the local testing. We delete the platform stanza, which is in the kitchen yaml because it doesn't apply when you move it to the open stack testing. We then restore those includes, excludes. So you still have them as you test in the cloud. And then we convert the hostname directive. So if you're doing vagrant, it's like a VM hostname. And if you're on open stack, it's server name. So we just transparently move that along for you. And then this was only found as a learning lesson. We prepend all of your suite names with cloud. And the reason we did this was if you were testing both locally and the cloud, at the same time, you wanted to make sure it worked locally, then check it in the cloud. You get these name conflicts because the way Test Kitchen worked is the suite name identified the resource. And so you would go to test it locally and then do KCloud. And we think the service was not running. And so we solve that by prepending all of our suite names with cloud. And then because we did that, we had to set the verifier name directly if you're familiar with Test Kitchen. It infers the verifier name based on the suite name. And now that we've changed the suite name, we have to explicitly tell it what the verifier name is. And then we write that.kitchen.kcloud.yaml to disk. So this all happens in a matter of milliseconds. And it immediately prepares you to take what you were testing locally and instead now test it in our IBM cloud test setup. And I kind of thought that was going to be the end of it, like, OK, I've done this work. I got this gem. I converted it. You know, while you're testing locally, things are going to be really, really quick. And they weren't. And it was very disappointing because I thought I had found, you know, my golden ticket to winning everybody back. We were spending like seven and a half minutes of all tests not actually converging anything. We had seven and a half minutes of time spent in the image boot, image create, the preparing for the test before we even got to the converge. And seven and a half minutes is a long time to wait for nothing to happen. So we had to attack this. And we attacked it in four steps. We attacked the create, the prepare, the converge, and the verify. And those are the four test kitchen steps that are out there. So our baseline for create, a minute and a half, which if you are typing something you wait a minute and a half, it feels like an eternity if nothing is happening. And I spent a lot of time trying to optimize this. And I realized after the fact I over-optimized it in terms of the amount of time that I saved. But to me, this was the most important step because it's the first one. They all depend upon it. And if things don't immediately start coming back to your laptop as you're testing, the user perception is that it's very slow. So if you can optimize this phase, even if it doesn't impact the overall test time all that much, the user perception is much better. And user experience is much better. And that's what I was trying to win back was that user experience. So I broke my create optimization into two different steps, the launch and the boot. The launch was everything up until the first system D message. One system D wrote a log entry, then booting has started. And the initial launch was 22 seconds, which wasn't bad, but it wasn't great. So dug into that a little bit. The first thing I did, I was like, okay, we got this image, it's like 800 megabytes. I will shrink it down. So there are some old kernels in there. I added to my packer setup to remove those kernels. I then wrote a bunch of zeros to where files were deleted. And that helps compression. And then ran versus prep. And for those not familiar with versus prep, it prepares a VM by clearing things out and making it ready to be relaunched again. And I was like, perfect, I've cut out like 20% of my file size. My launch time is going to go down by 20%. I'm going to go home early today. It's going to be great. And my image launch time was the exact same. So what I learned is that the image size doesn't really matter that much. These images are cached locally on the hypervisor. On OpenStack systems, it uses copy on write. So it's not going to be copying the image that much. It's just creating a clone of it when you change anything. So I learned, okay, don't spend any time on an image optimization. Where else can we get some wins here? I did notice this, though. The round trip time to look up the UUIDs for the various OpenStack entities was taking six seconds. So if you look, they're like an image reference and a network reference and a flavor reference. Each one of those, you have the test kitchen driver calling into OpenStack saying, okay, resolve for me what this name means and it comes back with a UUID. And then once it gets that, it does it for the flavor and then for the network. And these systems are remote. And so you have this speed of light problem. Each one of these requests was taking about two seconds happening in serial. So we have six seconds before we actually told OpenStack, okay, give me a node. What we did was submit a PR, it was merged, it's for your request 159 and it lets you hard code the UUIDs. And this took six seconds off of our image create time. And this was important because that blocks every other optimization. If you don't actually tell your OpenStack system I need a server, nothing else is going to happen. So it was important to get that part down as quick as possible. And the next thing was grub. So grub timeouts are there. Everyone from grub wears the thing that comes up in your boot and you can press escape to get out of it and choose your kernel. And that's great on a laptop or desktop. On a cloud system, not a lot of benefit. It's just sat there for five seconds waiting for input that was never going to happen, which is another five seconds before we could do something useful for our users. So drop that to zero. Now skip right past that and we bypass the timeout. So those two minor changes, they dropped our launch time in half down to 11 seconds, which is when the actual work could then get started. And then image boot. So out of the gate, and this is a Linux system, booting was taking like 70 seconds. That is a really long time for Linux box to boot. So I knew something here was not right. It's not a classic pattern. And I know some people don't like system D. They like the old init system. But system D analyzed blame is really, really good for digging into and figuring out why your system is booting the way it is and what is taking time. And it's a great tool for diagnosing those boot problems. What I found looking through our Git logs actually was that we had a configuration error. And the system and cloud init were recording these multiple timeouts. And we had essentially set up our images for when they were local images to not have DNS be managed by the network manager, which was great when it was local because we were coming through Vagrant and hitting our own internal DNS servers and it was all great. But in the public cloud, it turns out we needed DNS to do our job. And so we reverted that change for these public cloud images and got better. 25 seconds boot. That's definitely better than 69 seconds. But I knew we could optimize this further. And so rolled up our sleeves some more, got in there together, looked at system D analyzed, and kept diving in to make this better and better. The next thing was DHCP. And I was looking at the logs and it was taking like nine seconds to get an IP. And I was no DHCP protocol expert. So I went to Google and I was like, Wikipedia, a DHCP protocol. It was like a nice little diagram of like, these are the steps DHCP takes to give you an IP. And they were not the steps I was seeing in my log. So looked a little bit more. And I noticed that the system was requesting an IP. It wasn't asking DHCP for an IP. It was like, can I have this IP? And the DHCP server was like, I'm not even talking to you. And then it would say, can I have this IP again? Like please? And it's like, I'm not talking to you still. And finally it gave up and said, OK, give me an IP. And right about the behavior, it was essentially the fact that it was trying to ask for its old IP from the image it was created. Which I found bizarre because I ran Virtus.prep, which very loudly stated that it was a very simple method to clear out all the DHCP IPs. It turned out that there was an incompatibility between our version of Virtus.prep and our version of Red Hat Linux. And so, RM-RF fixes all the problems. And so, that was the way we went down. We added that script to Packer and dropped the boot down to 15 seconds. So we're making some good progress on the optimization front here. But I wanted better than 15 seconds. So the system de-analyze again looked more and network manager Wade online screamed out to me, is still taking a fairly long time. So we looked at that and what is that service? What is that doing? Its job is to block your system from booting. That sounds bad just to begin with until the network is stable. When it does this for good reasons, there's lots of services out there that get very upset when they come up and there's no network or even more upset when there's a network and it disappears on them and a new one comes up behind. Turns out though SSHD doesn't care, which is great because that's the only network service we care about when we're doing test kitchen. So we mask that service so that it essentially doesn't run on boot and we bypass that whole wait for the network to come up and when SSHD comes up and the network is there, it's well enough behaved to understand that and that works fine for testing. I don't recommend this for production systems. So great for test systems but don't do that in production. So we dropped it down to 8 seconds. So we went from 69 seconds to 8 seconds and then for the entire create process on minute 22 down to 20 seconds. And at this pace, you're starting to get a good user experience. You're typing pay cloud or kitchen and converges are starting. You see things on the screen. Images are booting under 20 seconds. So it's a much better experience for the user. But we weren't done. There's other steps we had to optimize. The next one was prepare and prepare was taking a very, very long time. So a kitchen prepare, those of you not familiar, that's where test kitchen grabs all of your cookbook files and all of your environment data and your data bags and your roles and it sends it to the remote system for testing. And this was taking over five and a half minutes, which was peculiar because when we ran it locally, it took like less than a second. So we had to look at the code and thankfully it's open source. You can figure out what's going on. And it's using essentially SCP under the covers to handle this. But in the version that we were using, it was doing it a file at a time. And our role cookbooks have over 1,100 files in them. So each one of those round trips to a remote system was adding up, adding up to over five minutes of sitting there with no feedback in your console as to what's going on. So we looked around. What could we do to resolve this? We found a couple of plug-ins out there for test kitchen that really helped us along. First one was speedy SSH. What this one does is it tarrs up those files, it tarrs up the cookbooks and it tarrs up the no data, it tarrs up the role data. SCPs it over and then untars it. And it dropped it down to 5.6 seconds, which was pretty good. We are at five minutes and now it's 5.6 seconds. It would be a little bit faster if it was a single tar because each of those connections is still around trip time back and forth. And so we looked at kitchen sync. And this uses the R-sync transport. And it seemed to just do a single connection to sync everything. And it was down to 3.4 seconds. So bringing in those two elements, the optimized create and the optimized prepare, we had now gotten down to the 20 second boot and the 3.4 second prepare and then converging could begin, which was really what we are trying to do in the first place is converge and test our stuff. Now on converging, we didn't optimize across 100 plus cookbooks to make them all as fast as possible. What we did was we implemented optimizations that would help all of our cookbooks in testing. And a lot of these are optimizations for remote testing, but even if you're a local testing, they could still apply. The first one that we did was we utilize a tool called FPM Cookery. Is anyone here familiar with FPM Cookery by any chance? Okay. Go become familiar because it's awesome. It's a tool that lets you very easily make packages. And so what we would do is we would get all of our enterprise software that usually came with its own separate installer. It would be a Java installer or it would be a shell script you would run. And historically, when we started down our configuration management path, we would write all these custom resources for these code and have to deal with the fact that the custom resource has to be idempotent and we have to deal with upgrades and all this churn just to install software. But it turns out that software installation is a pretty well solved problem and we didn't have to reinvent that. So we turned to this tool FPM Cookery, which lets you easily make packages from other artifacts. And because we manage everyone's development environment, we can give them a quick command to generate all the scaffolding required to build one of these packages. And this is what this looks like. So it's a simple script that does a clean and install and then runs the actual package command inside of a vagrant file. And we do a couple of things in this vagrant file. We sync over some required directories. So recipes, that's where your code goes that explains how you make a package. The staging is where your binary artifacts are stored and we store those in source control under git LFS for large storage. And then package is the output of the recipe. That's what we're building. In our case, it's an RPM. If you're a Debian shop, you can build Debians or whatever you want to build. It works transparently. And then we also run a Chef recipe to set up the server to give a git and Ruby and everything else that needs to build these packages. And then next use the script. And the recipes are really simple. So here's an example recipe. This is how we repackage a tool we use called Urban Code Deploy, the client for it. We point to the zip file to tell what the source is and FPM Cookery will automatically go and uncompress that for us. We tell it it's a omnibus recipe, which we give it a directory and all that means is when this thing is done, take all the content in that directory and put it into a package. We set some metadata such as the version and revision and stuff like that. But this is the actual recipe here to do all the work to take this piece of software and turn it into an RPM. We essentially copy over a properties file and then we run the installer.sh and we're done. That's it. The end result is an RPM. This runs in vagrant. So for us, we're running on Red Hat machine, so we get a RPM. If you ran it on Debian, you get a.dev at the end. It automatically uses the proper packaging tool. That saved us a lot of time because when you are downloading large software packages and then spinning up Java to install them or whatever method they're using, it just adds up under your test time. A lot of these packages run across all of our systems such as our monitoring tools, so it's important to optimize that side of it. Now the first time in a Red Hat system when you install a package with Chef, what happens under the covers is Yum needs to make a cache. So it goes and it downloads all the repo data from all of its known repos and creates a binary database and from there it can quickly check and see this version of software is available. This is where I go to download it. But it takes about 15 seconds to do that. So you pay this 15-second tax on the first package you install on every single test. And I didn't want to pay that 15-second tax. So we found a way around it. We found a loophole, sort of a packaging tax loophole. We turned something called Cloud in it. So if you're familiar with OpenStack and other cloud providers, there's a tool called Cloud in it that gets run whenever the node boots for the first time and you can pass it some data that it will execute. So we write out a simple file, a simple script called Yum make cache. And then we tell Cloud in it to run that script using at and to do it at now, which just means go do it immediately. And the reason we do that rather than running it directly is if we just ran Yum make cache in Cloud in it, all we would do was slow down our Cloud in it time by 15 seconds. But that's going to return immediately. And in the background, while the node's finishing booting, while Chef is doing the resolution of the cookbooks, while the kitchen prepare is happening, the Yum cache can be made. And these are a multi-processor system. So we have CPU to spare. It's essentially free optimization that we get. And this saved 15 seconds off of every single test. And you can just sit there in that first RPM package wondering what was going on. So that was a good general speedup that we had. And then data locality. So moving our testing to the Cloud was great because all of our binary artifacts were also right there next to it in object storage. So when you're on your laptop, like here, if I go to, you know, install an RPM, I have to download it from somewhere. And some of our RPMs are over gigabyte in size. And so that can take a lot of time to pull that down and be able to even install the package. But when it's right there, you've got these quick wins for all of our packages, like converges were just like happening like that for all of our package installations. The last phase that we spent some time optimizing was the setup of the verification. So the verify phase for test kitchen installs a couple of gems and then it can execute your test plan. So we essentially did the same trick with at in which we wrote out a script to install, reinstall the stuff needed for the verification and then ran it at now. And we saved five seconds. So the small optimization was still 50% of the time was saved by doing that. So overall reduction, we started at like seven and a half minutes of non-converged time to prepare to converge down to less than 30 seconds. And it gave us a really good overall user experience. But it was really only half of the benefit of the cloud-based testing. The other part was that it is awesome for concurrent testing. We have recipes and cookbooks that can have like seven or eight test suites across two different operating systems. So that's 16 tests you have to run, 16 VMs you need to spin up, and some of them were large. So we had tests that would take four hours to push out to test across all those platforms locally. But the benefit of doing it remotely is that we can do that in parallel or concurrently for free. There's no slowdown when you're doing 16 versus one. If you tried to do 16 on your laptop, it would crush it. You couldn't do it. But we could take things that were not taking four hours, we're taking minutes just by doing them in parallel and doing them out on remote resources. So we had to write a couple of tools to support this testing. The first one was that we built a kill switch into our test servers using cloud init again. You test locally and you forget to destroy your instance. It's not a big deal because you remember at some point in time. But if you do it in the cloud, you're now blocking a resource for someone on your team. So the kill switch is set to run 30 minutes after the node is created. You just use an add job again and make sure that Python client is there and it kills the node from within itself. And that was great except for people were like, hey, I was working on this node and it went away on me and that wasn't really cool. For some reason, I don't know why that popped up then, but it did. So we added a reset into this. So any sort of login into the test system, whether it's test kitchen doing a converge or verify or you logging in yourself to look at something, kill that old add job, start it a new one and reset the timer. And that works very well for us. This is some tooling that I wrote to help find hotspots for optimization. It works both remotely and locally depending on what you're doing. I'd hope to open source this for today. I could have it already, but I'm still waiting for the legal approval. So as soon as I get that, I will fully open source it so you guys can grab it if you want to optimize your local or your remote test runs. And it gives you three analyzing tools built into it. The first one lets you analyze the create and it splits between the boot and the image launch. The second one gives you recipe timing. So you can just run it and see which recipes take the longest across one or more of your test runs. And then the last one is the duration of individual steps. So it just takes all your steps and times them and tells you which ones take the longest. And if you have it across some roll cookbooks, you can sort of dive in and say, I can spend some time here and get a good optimization for all of my tests. And this was very helpful because we had one package we had that was not built with FBM cookery. It was actually a handcrafted, artistically made package that we had. And the default for that package was LZMA2 compression, which was great because it made it really small. But it was also a gigabyte size package. And so it took a minute and a half to uncompress it and install it. And we changed that to GZIP and it went up by 15% in size, but installs in 17 seconds. So that data locality I could talk about sort of got rid of any sort of benefit of smaller compression. All right. So I have a lot more details on my blog about the individual steps and optimizations about how we optimize various phases. And I'm also on Twitter if you have questions. Did you guys like that? It was good. It was helpful. Applause. Thank you.
 So to get started here, this is my own personal redemption story.  It's about the mistakes I made when transitioning our environment to a test-driven infrastructure  and what we have done so far to resolve them.  But before I dive into that, what exactly does my team at IBM do?  We're known for sports.  We do the Australian Open, we do Roland Garros, or the French Open, which is going on right  now, Wimbledon, the US Open.  We also do a large golf tournament in Georgia.  But it's not just sports.  We also do IBM.com and all of the digital properties that comprise the IBM.com portfolio.  And we have this focus on what we call continuous availability.  In fact, we've been running IBM.com with zero outages and zero downtime since June of 2001.  And in that time span, we actually moved data centers 18 times with zero seconds of downtime.  And we do this across a hybrid cloud.  We have three private data centers and four public cloud regions in the IBM cloud that  we serve these digital properties off of.  And we do it with a large team.  We have 80 plus people in eight different time zones in 35 work locations.  And so we need these tools and methods to let us work independently, yet collaborate  when necessary.  And our workload is highly mission critical.  Our partners and teams spend 50 weeks a year preparing for two or less weeks.  There's no, oh, we'll fix this next week.  That just doesn't happen.  They're going to hold the US Open, whether we're ready or not.  And we have to be ready.  And not only is these properties in this time span on the world stage, it's also when they  derive most of their revenue.  So we have to deliver flawlessly.  Now here was the mistake I made.  When I was doing this transformation, I went out there and I surveyed the landscape.  And I looked at all the best practices.  I said, OK, we're going to move this stuff into Git.  We're going to do code reviews.  We're going to have a delivery pipeline.  We're going to do infrastructure as code.  We're going to do local testing.  And I made all these changes at once and did not go over very well.  It was a lot of change to push through all at once.  And the biggest problem we had was that simple tasks that used to take three minutes were  now taking hours.  It used to be I'd log into the server and I'd edit the file and I move on to the next  thing.  And now doing a Git clone, editing the file, committing it.  It's going to a code review, building it locally.  And it's taking hours.  So a three-minute change was now taking an hour to do.  And it was this combination of these best practices, which are great practices and other  challenges within our organization and our structure that was causing these changes to  take a long time.  The other problems that we faced was that we have to put out a lot of enterprise software.  It's part of our platform.  And some of the software is very big.  We have software that's over a gigabyte in size.  And if you're working locally and you're trying to do a vagrant up or a test kitchen  and test that and you're pulling down a one gigabyte package every time, that can take  a while.  And our team is distributed.  We have people all over the world.  We have a large part of our team is in China.  We have people in India.  We have people all over the United States and Europe.  And they're not all sitting right next to the source of those binary artifacts.  So download time is very important.  And they're in remote locations.  Either they live in a remote location possibly or we send them to one.  If we send them to the US Open, there's a fixed amount of bandwidth available for you  to download things constantly and test them.  And the biggest feedback I got was that this combination of factors was resulting in changes  that you take minutes taking hours.  And I thought, you know, I brought this problem onto the team.  So I decided this was my problem to work really hard to resolve.  So we had to find a solution.  And our solution was we moved all of our testing away from local laptops to the cloud.  In particular, we're on the IBM cloud.  And we use a service that used to be called Blue Box.  It's now IBM Private Cloud as a service.  That's an open stack based cloud.  And to do this, we used the kitchen open stack plugin to let us spin up test machines on  the cloud and test our changes out there instead of locally.  Now we're still developing locally.  We're still doing the Git polls and the code reviews.  But the actual execution of all of our testing takes place now in the IBM cloud.  And we're able to do this and make this transition easy because how we manage everyone's laptops.  When you come to our team and you start to work on Chef, you essentially run a curl bash.  And from that point on, we take over the management of the development tool in your laptop.  So we install Git at the right level.  We install Vagrant, VirtualBox, ChefDK, all the things that you need to be able to get  your environment going at the right level.  And this is great because it lets us make changes easily and transition things like  local testing to cloud testing without a lot of impact or breakage.  And so we introduced this thing we call KCloud.  This was our effort to let you have your local testing and also bring in this cloud testing.  And what it really was was a gem that we wrote called KCloud Convert.  And it did a couple of things for us.  First thing it does is it reads in your kitchen yaml.  And I should back up a little bit.  That KCloud Convert, we have access to everyone's bash RC.  And so we add it to their environment.  So instead of typing kitchen to do a test, it's just KCloud instead.  And then under the covers, we read in the kitchen yaml.  We validate that they're only testing Red Hat Enterprise Linux.  And that's an implementation detail we have that we're moving all our stuff to that platform.  So if you're on a different platform, you still test locally.  But in our platform for our stuff, it's all Red Hat.  So we just validate that.  We check the includes, excludes that you might have because you could be only testing against  a version of Red Hat 6 or Red Hat 7.  And we want to keep that across the cloud and the local testing.  We delete the platform stanza, which is in the kitchen yaml because it doesn't apply  when you move it to the open stack testing.  We then restore those includes, excludes.  So you still have them as you test in the cloud.  And then we convert the hostname directive.  So if you're doing vagrant, it's like a VM hostname.  And if you're on open stack, it's server name.  So we just transparently move that along for you.  And then this was only found as a learning lesson.  We prepend all of your suite names with cloud.  And the reason we did this was if you were testing both locally and the cloud, at the  same time, you wanted to make sure it worked locally, then check it in the cloud.  You get these name conflicts because the way Test Kitchen worked is the suite name identified  the resource.  And so you would go to test it locally and then do KCloud.  And we think the service was not running.  And so we solve that by prepending all of our suite names with cloud.  And then because we did that, we had to set the verifier name directly if you're familiar  with Test Kitchen.  It infers the verifier name based on the suite name.  And now that we've changed the suite name, we have to explicitly tell it what the verifier  name is.  And then we write that.kitchen.kcloud.yaml to disk.  So this all happens in a matter of milliseconds.  And it immediately prepares you to take what you were testing locally and instead now test  it in our IBM cloud test setup.  And I kind of thought that was going to be the end of it, like, OK, I've done this work.  I got this gem.  I converted it.  You know, while you're testing locally, things are going to be really, really quick.  And they weren't.  And it was very disappointing because I thought I had found, you know, my golden ticket to  winning everybody back.  We were spending like seven and a half minutes of all tests not actually converging anything.  We had seven and a half minutes of time spent in the image boot, image create, the preparing  for the test before we even got to the converge.  And seven and a half minutes is a long time to wait for nothing to happen.  So we had to attack this.  And we attacked it in four steps.  We attacked the create, the prepare, the converge, and the verify.  And those are the four test kitchen steps that are out there.  So our baseline for create, a minute and a half, which if you are typing something you  wait a minute and a half, it feels like an eternity if nothing is happening.  And I spent a lot of time trying to optimize this.  And I realized after the fact I over-optimized it in terms of the amount of time that I saved.  But to me, this was the most important step because it's the first one.  They all depend upon it.  And if things don't immediately start coming back to your laptop as you're testing, the  user perception is that it's very slow.  So if you can optimize this phase, even if it doesn't impact the overall test time all  that much, the user perception is much better.  And user experience is much better.  And that's what I was trying to win back was that user experience.  So I broke my create optimization into two different steps, the launch and the boot.  The launch was everything up until the first system D message.  One system D wrote a log entry, then booting has started.  And the initial launch was 22 seconds, which wasn't bad, but it wasn't great.  So dug into that a little bit.  The first thing I did, I was like, okay, we got this image, it's like 800 megabytes.  I will shrink it down.  So there are some old kernels in there.  I added to my packer setup to remove those kernels.  I then wrote a bunch of zeros to where files were deleted.  And that helps compression.  And then ran versus prep.  And for those not familiar with versus prep, it prepares a VM by clearing things out and  making it ready to be relaunched again.  And I was like, perfect, I've cut out like 20% of my file size.  My launch time is going to go down by 20%.  I'm going to go home early today.  It's going to be great.  And my image launch time was the exact same.  So what I learned is that the image size doesn't really matter that much.  These images are cached locally on the hypervisor.  On OpenStack systems, it uses copy on write.  So it's not going to be copying the image that much.  It's just creating a clone of it when you change anything.  So I learned, okay, don't spend any time on an image optimization.  Where else can we get some wins here?  I did notice this, though.  The round trip time to look up the UUIDs for the various OpenStack entities was taking  six seconds.  So if you look, they're like an image reference and a network reference and a flavor reference.  Each one of those, you have the test kitchen driver calling into OpenStack saying, okay,  resolve for me what this name means and it comes back with a UUID.  And then once it gets that, it does it for the flavor and then for the network.  And these systems are remote.  And so you have this speed of light problem.  Each one of these requests was taking about two seconds happening in serial.  So we have six seconds before we actually told OpenStack, okay, give me a node.  What we did was submit a PR, it was merged, it's for your request 159 and it lets you  hard code the UUIDs.  And this took six seconds off of our image create time.  And this was important because that blocks every other optimization.  If you don't actually tell your OpenStack system I need a server, nothing else is going to  happen.  So it was important to get that part down as quick as possible.  And the next thing was grub.  So grub timeouts are there.  Everyone from grub wears the thing that comes up in your boot and you can press escape to  get out of it and choose your kernel.  And that's great on a laptop or desktop.  On a cloud system, not a lot of benefit.  It's just sat there for five seconds waiting for input that was never going to happen,  which is another five seconds before we could do something useful for our users.  So drop that to zero.  Now skip right past that and we bypass the timeout.  So those two minor changes, they dropped our launch time in half down to 11 seconds, which  is when the actual work could then get started.  And then image boot.  So out of the gate, and this is a Linux system, booting was taking like 70 seconds.  That is a really long time for Linux box to boot.  So I knew something here was not right.  It's not a classic pattern.  And I know some people don't like system D. They like the old init system.  But system D analyzed blame is really, really good for digging into and figuring out why  your system is booting the way it is and what is taking time.  And it's a great tool for diagnosing those boot problems.  What I found looking through our Git logs actually was that we had a configuration error.  And the system and cloud init were recording these multiple timeouts.  And we had essentially set up our images for when they were local images to not have DNS  be managed by the network manager, which was great when it was local because we were coming  through Vagrant and hitting our own internal DNS servers and it was all great.  But in the public cloud, it turns out we needed DNS to do our job.  And so we reverted that change for these public cloud images and got better.  25 seconds boot.  That's definitely better than 69 seconds.  But I knew we could optimize this further.  And so rolled up our sleeves some more, got in there together, looked at system D analyzed,  and kept diving in to make this better and better.  The next thing was DHCP.  And I was looking at the logs and it was taking like nine seconds to get an IP.  And I was no DHCP protocol expert.  So I went to Google and I was like, Wikipedia, a DHCP protocol.  It was like a nice little diagram of like, these are the steps DHCP takes to give you  an IP.  And they were not the steps I was seeing in my log.  So looked a little bit more.  And I noticed that the system was requesting an IP.  It wasn't asking DHCP for an IP.  It was like, can I have this IP?  And the DHCP server was like, I'm not even talking to you.  And then it would say, can I have this IP again?  Like please?  And it's like, I'm not talking to you still.  And finally it gave up and said, OK, give me an IP.  And right about the behavior, it was essentially the fact that it was trying to ask for its  old IP from the image it was created.  Which I found bizarre because I ran Virtus.prep, which very loudly stated that it was a very  simple method to clear out all the DHCP IPs.  It turned out that there was an incompatibility between our version of Virtus.prep and our  version of Red Hat Linux.  And so, RM-RF fixes all the problems.  And so, that was the way we went down.  We added that script to Packer and dropped the boot down to 15 seconds.  So we're making some good progress on the optimization front here.  But I wanted better than 15 seconds.  So the system de-analyze again looked more and network manager Wade online screamed  out to me, is still taking a fairly long time.  So we looked at that and what is that service?  What is that doing?  Its job is to block your system from booting.  That sounds bad just to begin with until the network is stable.  When it does this for good reasons, there's lots of services out there that get very  upset when they come up and there's no network or even more upset when there's a network  and it disappears on them and a new one comes up behind.  Turns out though SSHD doesn't care, which is great because that's the only network  service we care about when we're doing test kitchen.  So we mask that service so that it essentially doesn't run on boot and we bypass that whole  wait for the network to come up and when SSHD comes up and the network is there, it's well  enough behaved to understand that and that works fine for testing.  I don't recommend this for production systems.  So great for test systems but don't do that in production.  So we dropped it down to 8 seconds.  So we went from 69 seconds to 8 seconds and then for the entire create process on minute  22 down to 20 seconds.  And at this pace, you're starting to get a good user experience.  You're typing pay cloud or kitchen and converges are starting.  You see things on the screen.  Images are booting under 20 seconds.  So it's a much better experience for the user.  But we weren't done.  There's other steps we had to optimize.  The next one was prepare and prepare was taking a very, very long time.  So a kitchen prepare, those of you not familiar, that's where test kitchen grabs all of your  cookbook files and all of your environment data and your data bags and your roles and  it sends it to the remote system for testing.  And this was taking over five and a half minutes, which was peculiar because when we  ran it locally, it took like less than a second.  So we had to look at the code and thankfully it's open source.  You can figure out what's going on.  And it's using essentially SCP under the covers to handle this.  But in the version that we were using, it was doing it a file at a time.  And our role cookbooks have over 1,100 files in them.  So each one of those round trips to a remote system was adding up, adding up to over five  minutes of sitting there with no feedback in your console as to what's going on.  So we looked around.  What could we do to resolve this?  We found a couple of plug-ins out there for test kitchen that really helped us along.  First one was speedy SSH.  What this one does is it tarrs up those files, it tarrs up the cookbooks and it tarrs up the  no data, it tarrs up the role data.  SCPs it over and then untars it.  And it dropped it down to 5.6 seconds, which was pretty good.  We are at five minutes and now it's 5.6 seconds.  It would be a little bit faster if it was a single tar because each of those connections  is still around trip time back and forth.  And so we looked at kitchen sync.  And this uses the R-sync transport.  And it seemed to just do a single connection to sync everything.  And it was down to 3.4 seconds.  So bringing in those two elements, the optimized create and the optimized prepare, we had now  gotten down to the 20 second boot and the 3.4 second prepare and then converging could  begin, which was really what we are trying to do in the first place is converge and test  our stuff.  Now on converging, we didn't optimize across 100 plus cookbooks to make them all as fast  as possible.  What we did was we implemented optimizations that would help all of our cookbooks in testing.  And a lot of these are optimizations for remote testing, but even if you're a local  testing, they could still apply.  The first one that we did was we utilize a tool called FPM Cookery.  Is anyone here familiar with FPM Cookery by any chance?  Okay.  Go become familiar because it's awesome.  It's a tool that lets you very easily make packages.  And so what we would do is we would get all of our enterprise software that usually came  with its own separate installer.  It would be a Java installer or it would be a shell script you would run.  And historically, when we started down our configuration management path, we would write  all these custom resources for these code and have to deal with the fact that the custom  resource has to be idempotent and we have to deal with upgrades and all this churn just  to install software.  But it turns out that software installation is a pretty well solved problem and we didn't  have to reinvent that.  So we turned to this tool FPM Cookery, which lets you easily make packages from other artifacts.  And because we manage everyone's development environment, we can give them a quick command  to generate all the scaffolding required to build one of these packages.  And this is what this looks like.  So it's a simple script that does a clean and install and then runs the actual package  command inside of a vagrant file.  And we do a couple of things in this vagrant file.  We sync over some required directories.  So recipes, that's where your code goes that explains how you make a package.  The staging is where your binary artifacts are stored and we store those in source control  under git LFS for large storage.  And then package is the output of the recipe.  That's what we're building.  In our case, it's an RPM.  If you're a Debian shop, you can build Debians or whatever you want to build.  It works transparently.  And then we also run a Chef recipe to set up the server to give a git and Ruby and everything  else that needs to build these packages.  And then next use the script.  And the recipes are really simple.  So here's an example recipe.  This is how we repackage a tool we use called Urban Code Deploy, the client for it.  We point to the zip file to tell what the source is and FPM Cookery will automatically  go and uncompress that for us.  We tell it it's a omnibus recipe, which we give it a directory and all that means is  when this thing is done, take all the content in that directory and put it into a package.  We set some metadata such as the version and revision and stuff like that.  But this is the actual recipe here to do all the work to take this piece of software and  turn it into an RPM.  We essentially copy over a properties file and then we run the installer.sh and we're  done.  That's it.  The end result is an RPM.  This runs in vagrant.  So for us, we're running on Red Hat machine, so we get a RPM.  If you ran it on Debian, you get a.dev at the end.  It automatically uses the proper packaging tool.  That saved us a lot of time because when you are downloading large software packages and  then spinning up Java to install them or whatever method they're using, it just adds up under  your test time.  A lot of these packages run across all of our systems such as our monitoring tools, so  it's important to optimize that side of it.  Now the first time in a Red Hat system when you install a package with Chef, what happens  under the covers is Yum needs to make a cache.  So it goes and it downloads all the repo data from all of its known repos and creates a  binary database and from there it can quickly check and see this version of software is  available.  This is where I go to download it.  But it takes about 15 seconds to do that.  So you pay this 15-second tax on the first package you install on every single test.  And I didn't want to pay that 15-second tax.  So we found a way around it.  We found a loophole, sort of a packaging tax loophole.  We turned something called Cloud in it.  So if you're familiar with OpenStack and other cloud providers, there's a tool called Cloud  in it that gets run whenever the node boots for the first time and you can pass it some  data that it will execute.  So we write out a simple file, a simple script called Yum make cache.  And then we tell Cloud in it to run that script using at and to do it at now, which just means  go do it immediately.  And the reason we do that rather than running it directly is if we just ran Yum make cache  in Cloud in it, all we would do was slow down our Cloud in it time by 15 seconds.  But that's going to return immediately.  And in the background, while the node's finishing booting, while Chef is doing the resolution  of the cookbooks, while the kitchen prepare is happening, the Yum cache can be made.  And these are a multi-processor system.  So we have CPU to spare.  It's essentially free optimization that we get.  And this saved 15 seconds off of every single test.  And you can just sit there in that first RPM package wondering what was going on.  So that was a good general speedup that we had.  And then data locality.  So moving our testing to the Cloud was great because all of our binary artifacts were also  right there next to it in object storage.  So when you're on your laptop, like here, if I go to, you know, install an RPM, I have  to download it from somewhere.  And some of our RPMs are over gigabyte in size.  And so that can take a lot of time to pull that down and be able to even install the  package.  But when it's right there, you've got these quick wins for all of our packages, like converges  were just like happening like that for all of our package installations.  The last phase that we spent some time optimizing was the setup of the verification.  So the verify phase for test kitchen installs a couple of gems and then it can execute your  test plan.  So we essentially did the same trick with at in which we wrote out a script to install,  reinstall the stuff needed for the verification and then ran it at now.  And we saved five seconds.  So the small optimization was still 50% of the time was saved by doing that.  So overall reduction, we started at like seven and a half minutes of non-converged time to  prepare to converge down to less than 30 seconds.  And it gave us a really good overall user experience.  But it was really only half of the benefit of the cloud-based testing.  The other part was that it is awesome for concurrent testing.  We have recipes and cookbooks that can have like seven or eight test suites across two  different operating systems.  So that's 16 tests you have to run, 16 VMs you need to spin up, and some of them were  large.  So we had tests that would take four hours to push out to test across all those platforms  locally.  But the benefit of doing it remotely is that we can do that in parallel or concurrently  for free.  There's no slowdown when you're doing 16 versus one.  If you tried to do 16 on your laptop, it would crush it.  You couldn't do it.  But we could take things that were not taking four hours, we're taking minutes just by doing  them in parallel and doing them out on remote resources.  So we had to write a couple of tools to support this testing.  The first one was that we built a kill switch into our test servers using cloud init again.  You test locally and you forget to destroy your instance.  It's not a big deal because you remember at some point in time.  But if you do it in the cloud, you're now blocking a resource for someone on your team.  So the kill switch is set to run 30 minutes after the node is created.  You just use an add job again and make sure that Python client is there and it kills  the node from within itself.  And that was great except for people were like, hey, I was working on this node and  it went away on me and that wasn't really cool.  For some reason, I don't know why that popped up then, but it did.  So we added a reset into this.  So any sort of login into the test system, whether it's test kitchen doing a converge  or verify or you logging in yourself to look at something, kill that old add job, start  it a new one and reset the timer.  And that works very well for us.  This is some tooling that I wrote to help find hotspots for optimization.  It works both remotely and locally depending on what you're doing.  I'd hope to open source this for today.  I could have it already, but I'm still waiting for the legal approval.  So as soon as I get that, I will fully open source it so you guys can grab it if you want  to optimize your local or your remote test runs.  And it gives you three analyzing tools built into it.  The first one lets you analyze the create and it splits between the boot and the image  launch.  The second one gives you recipe timing.  So you can just run it and see which recipes take the longest across one or more of your  test runs.  And then the last one is the duration of individual steps.  So it just takes all your steps and times them and tells you which ones take the longest.  And if you have it across some roll cookbooks, you can sort of dive in and say, I can spend  some time here and get a good optimization for all of my tests.  And this was very helpful because we had one package we had that was not built with FBM  cookery.  It was actually a handcrafted, artistically made package that we had.  And the default for that package was LZMA2 compression, which was great because it made  it really small.  But it was also a gigabyte size package.  And so it took a minute and a half to uncompress it and install it.  And we changed that to GZIP and it went up by 15% in size, but installs in 17 seconds.  So that data locality I could talk about sort of got rid of any sort of benefit of smaller  compression.  All right.  So I have a lot more details on my blog about the individual steps and optimizations about  how we optimize various phases.  And I'm also on Twitter if you have questions.  Did you guys like that?  It was good.  It was helpful.  Applause.  Thank you.
_Acing__Infrastructure_Testing_with_Chef_0_131.png _Acing__Infrastructure_Testing_with_Chef_132_707.png _Acing__Infrastructure_Testing_with_Chef_708_1139.png _Acing__Infrastructure_Testing_with_Chef_1140_1835.png _Acing__Infrastructure_Testing_with_Chef_1836_2171.png _Acing__Infrastructure_Testing_with_Chef_2172_2579.png _Acing__Infrastructure_Testing_with_Chef_2580_3443.png _Acing__Infrastructure_Testing_with_Chef_3444_5315.png _Acing__Infrastructure_Testing_with_Chef_5316_5987.png _Acing__Infrastructure_Testing_with_Chef_5988_6431.png _Acing__Infrastructure_Testing_with_Chef_6432_6827.png _Acing__Infrastructure_Testing_with_Chef_6828_7367.png _Acing__Infrastructure_Testing_with_Chef_7368_8495.png _Acing__Infrastructure_Testing_with_Chef_8496_9527.png _Acing__Infrastructure_Testing_with_Chef_9528_13775.png _Acing__Infrastructure_Testing_with_Chef_13776_14915.png _Acing__Infrastructure_Testing_with_Chef_14916_15203.png _Acing__Infrastructure_Testing_with_Chef_15204_16571.png _Acing__Infrastructure_Testing_with_Chef_16572_17279.png _Acing__Infrastructure_Testing_with_Chef_17280_18287.png _Acing__Infrastructure_Testing_with_Chef_18288_20219.png _Acing__Infrastructure_Testing_with_Chef_20220_21599.png _Acing__Infrastructure_Testing_with_Chef_21600_21755.png _Acing__Infrastructure_Testing_with_Chef_21756_22031.png _Acing__Infrastructure_Testing_with_Chef_22032_22487.png _Acing__Infrastructure_Testing_with_Chef_22488_23963.png _Acing__Infrastructure_Testing_with_Chef_23964_24419.png _Acing__Infrastructure_Testing_with_Chef_24420_26015.png _Acing__Infrastructure_Testing_with_Chef_26016_28715.png _Acing__Infrastructure_Testing_with_Chef_28716_29459.png _Acing__Infrastructure_Testing_with_Chef_29460_32063.png _Acing__Infrastructure_Testing_with_Chef_32064_32339.png _Acing__Infrastructure_Testing_with_Chef_32340_32927.png _Acing__Infrastructure_Testing_with_Chef_32928_33599.png _Acing__Infrastructure_Testing_with_Chef_33600_36803.png _Acing__Infrastructure_Testing_with_Chef_36804_38387.png _Acing__Infrastructure_Testing_with_Chef_38388_40151.png _Acing__Infrastructure_Testing_with_Chef_40152_41915.png _Acing__Infrastructure_Testing_with_Chef_41916_42923.png _Acing__Infrastructure_Testing_with_Chef_42924_43343.png _Acing__Infrastructure_Testing_with_Chef_43344_43691.png _Acing__Infrastructure_Testing_with_Chef_43692_43847.png _Acing__Infrastructure_Testing_with_Chef_43848_44411.png _Acing__Infrastructure_Testing_with_Chef_44412_45779.png _Acing__Infrastructure_Testing_with_Chef_45780_46847.png _Acing__Infrastructure_Testing_with_Chef_46848_47471.png _Acing__Infrastructure_Testing_with_Chef_47472_48107.png _Acing__Infrastructure_Testing_with_Chef_48108_48731.png _Acing__Infrastructure_Testing_with_Chef_48732_50195.png _Acing__Infrastructure_Testing_with_Chef_50196_50747.png _Acing__Infrastructure_Testing_with_Chef_50748_50843.png _Acing__Infrastructure_Testing_with_Chef_50844_50863.png