Sweeter image builds with KIWI
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Gompa, Neal
How Datto uses KIWI to simplify building appliance images One of the more heavily underrated openSUSE projects is the [KIWI image builder](github). In the last few years, [Datto] has started using KIWI to replace the patchwork of custom image build tools to provide a consistent toolchain for producing various appliance images. This talk introduces the KIWI appliance image builder, outlines some of Datto's use-cases for KIWI, and how Datto uses KIWI to support those use-cases. This also includes a brief demo of building an image with KIWI.
 Hello everyone, my name is Neil Gompine. I'm here to talk to you about sweeter image builds with Kiwi. Now, Dato's using Kiwi. So first, talk a little bit about me. I call myself a professional technologist. I've been in technology since I could remember, and I've been involved in Linux for nearly 15 years, and as a contributor and developer in Fedora, Magia, OpenSUSE, and OpenMandriva Linux distributions. Note, also contributor to the RPM, DNF, and various related software management, systems management, and image building tools. Of course, that includes Kiwi, the tool we're talking about today, and I'm a senior DevOps engineer at Dato. So a little bit about Dato. We were founded in 2007. We've got 22 global locations, over 1,600 employees worldwide, 17,000 managed service provider partners, and we operate entirely in the channel, meaning that all of our products are not directly sold to you. We sell to companies that take our services and optimize them for their clients' needs, and they sell them to you. We have local offices in nine countries that help service providers and serve over 1 million small to medium businesses around the world, or small to medium enterprises. You may have heard that term instead. We offer a wide variety of products and solutions to support our MSP partners to enable their business to support their clients, and that ranges from disaster recovery, backups business continuity stuff with our unified continuity products, with being able to do managed networking solutions with data networking, professional services automation, and remote monitoring and management with our RMM and PSA products, as well as inventory and quoting, and revenue management with our commerce solutions. So taking all that and going back to building images at Dato. So for building images at Dato, we got a little bit of a problem. We can build images for days. Practically everyone did image building differently with a different set of issues caused by these methods. So some tools we're using is Packer, Debian's Live Build for ISOs, custom shell scripts for making disk images and custom base images and all kinds of weird little things. And we use Vert Builder for making various custom virtual machine disk images that don't really fit into clouds and are used for very specialty use cases. The problem with this is that we wind up having problems that are just odd. And this example is great of Packer just not creating images that are sane because the actual mechanism in which it produces the image is not sane. It goes by pausing the virtual machine and taking a snapshot and exporting it. Well, this kind of was a problem when you're using Puppet to configure the environment and that includes like setting the machines up for getting unattended updates and things like that. And the apt daily service in particular just was always started and running just as we were pausing the machine and that left apt and d package in a broken state when you tried to boot up the machine in a new instance. So we had to do all kinds of things to work around that. Another example is that when the Spectra meltdown fixes landed a couple years back, Live Build didn't really cope with that very well initially. And we had to do funny things to kind of make that work. And that also uncovered another random thing where suddenly XZ compression didn't work anymore, so we had to switch to Gzip. We don't know. And this kind of led to a core problem that we've discovered is that some of these tools have seemingly hysterical behaviors, but something that's also been kind of increasingly common among these tools is that they're poorly maintained and that the actual capabilities of them are incomplete. And these tools don't have, you know, what I would like to call a method to their madness reasoning how they work was too hard. And that make using them difficult because when something went weird in an image build process or in the image produced by the build, it was difficult to walk back and figure out like what happened and what went wrong. And that really eats a bunch of time that could be better spent doing more valuable things like iterating on it to, you know, build more capabilities, layered systems, solutions and things like that. So I started looking at Kiwi as a solution to solve this problem, because Kiwi was straightforward and idiomatic. It has XML, YAML or JSON based descriptions with some simple script books that you can use for other flexibility. And speaking of flexibility, it can build almost any type of image. And if it doesn't know what type of image it is, you can use the Python API for Kiwi to construct custom types of images. So you can build upon the framework and the tooling inside of Kiwi to construct anything you want. And it's free and open source software under the GNU General Public License version three, and it's actively developed and maintained and the developers are friendly and helpful and that was huge to me because it didn't actually work right out of the box. So DATO produces a lot of Ubuntu based images, and that means we have to work with the, you know, making images with the Bootstrap and apt and, you know, there were a couple of issues that I discovered along the way and so I went and fixed them. Like when I went and sent pull requests, they were very quick to respond with feedback and they worked with me on making sure that it was right and then we got it in and made releases. They're very good about releasing fixes as they're being merged in. So that was great. And when we started evolving into like handling some sentos based images, that also was somewhat incomplete. And so I added support for features that we needed that just weren't wired up yet. Perhaps they didn't know or whatever, but like I figured it out and added the functionality. And this particular change was interesting and cool because I didn't really know how to work with, you know, the extensive test suite that they have for Kiwi. And they were really quick and responsive and friendly and helpful. All these things for helping me make the unit tests to make sure that the behavior worked because and they were very patient with me trying to figure out the test suite and gave me hints and such. And that made the huge difference. I was able to be successful and very quickly iterate on this and get this into a state that it could be merged and it was. And so I want to show you a little bit about Kiwi with, you know, a sample that I from one of the descriptions that I've been working with. So we have here a container appliance based on sentos stream nine. And this is working off of test composes that are being released right now sentos stream nine stuff became available last month. And so I started working with it. So this container appliance is configured using the data package manager you can see here. It's English only because that's the only locale I care about time zone is UTC because screw daylight savings time. We have locale filtering and turned on so that means that if there's any other locale data or any other locales that are being installed because of packages they'll get filtered out by RPM. But I also have checked signatures disabled because right now this content is not signed. And so you can't really do anything about that. So I just turned it off. This other flag which is neat is the exclude doc. So I wanted to make the image super small so I don't need to include documentation stuff since RPM has flags for that we just set it. And so all the documentation was just stripped out because it just wasn't installed. And we have a small set of packages here just file system the branding packages and some the package manager and some utilities. And that's about it. And this is just a simple TVZ type image which basically means it's just a tar ball with files in it. And we also should say the this is the shell hook that is used you could actually replace this with calling Ansible or or running some kind of other tool, another language or whatnot because this can be anything as long as it is called config.sh it will execute it. And so you can have this do anything you'd like. This one's just super simple about setting it up to be multi user with the root user populated and setting the host name is local host here. And then cleaning out any extra unwanted locales that may have not been correctly marked as local data by RPM. And so let's just go ahead and build it. Right. So you can see it's running this through. And we want to see the log here. And you can see this is setting up the image there and it's all going through Kiwi here. You see that it was setting up the repos then it bootstrapped and installed the packages. After the package installation was done, it ran the scripts, the script helpers, and is creating the tar ball right now. So you can see it's doing that. It's an XZ compressed tar ball with multiple threads. So it's like using all the cores on the computer to actually produce the tar ball. And after that, it will. And while it's creating that I've already pre created this before so we're going to go ahead and boot it up in and spawn. So we'll just start that up. And now you see that it's got the it starts up like it would a regular computer. So this container was particularly configured to behave like. You know how you would start up a VM so it starts up system D and then starts log in D. So I can do route my super secure passwords. And go in here and there's nothing in here but I can see this and I see that and then Etsy you can see there's all these files. So let's catOS release. You see it sent us stream nine with all the stuff in here, you know, and then CDDNF. There's all the directories you'd expect here. But maybe I wanted it, you know, I wanted an editor to edit a files like, you know, let's let's edit Dnf. And oh, that's not there. Well, then what do I do. So we will go back and do that. So like just seeing here. But before we go back and do that, let's take a look at what it actually produced. So I showed you what it looks like. Let's let's take a look at what it actually what it actually produced temp output container. So the changes file is really just a concatenation of change logs for all the different packages that were included. So GCC you can see in here and you see open SSL, Python, all kinds of stuff in here. And then if you look at packages, this one's actually just a list of all the packages and this has the name group, the name of the version release architecture and the license data. And this is useful if you want to see whether things have changed between build to build, and also to see what licenses are included in the whole thing. And then the verified one is for checking the file system structure to see how much of it differs from what the RPM database knows. So you can see that yeah there was some files that were deleted because I purged extra locale data. And you can see that there's you know some modifications of some config files and stuff. And that's fine. So, but we want to have a text editor in there. So let's add that. So TXC build config XML. And we're just going to go ahead and add another package. So we'll add nano because nano is the best text editor ever. And we're going to go ahead and create this to build again. We're going to call it temp output two for this directory. We're going to run it again. And so this sets it up again and it goes through this process. And if we want to take a look at what this looks like, we'll take a look at the log here. It's already at the point of creating the image again. And once that's actually done, we will have something to show off. But meanwhile, while that is happening, we should go ahead and clean up or not actually, we can. Well, we do need to be root here because otherwise we can't actually write the terrible but probably want to be in and disc. Vc temp output two. So we're ready for that. And it's actually a it's done. So the that part is created. And we're going to go ahead and tar XVF container C bar live machines. EL nine appliance to. All right, let's make the directory bar live machines. EL nine appliance to. And so that extracts all the files for that because this is just a regular tar ball. So it only has those particular things. So now we will boot up the second appliance that we just created. So this is just like the first one except now it has nano installed. And now, if I go to at C, we go to DNF, and I go to DNF dot com. I have the nano text editor. Now let's let's check and see like what this difference is actually look like. All right, so we made the temp output and temp output two directories. And so let's take a look at diff of temp output. If you have container EL nine packages to temp output to container EL nine packages. And you can see we added just one package nano. Now let's see what that looks like for change logs. This is actually going to be pretty bad because change logs are kind of huge. But it'll kind of illustrate the point I was trying to make. So you see there's a whole new change log that was added that is literally just for nano. There was also some other sorting things and that's kind of the reason why I didn't want to show that real thing about that too hard. Let's take a look at verified. Let's see if there was any other differences. I expect that we might not have any see no differences. So from a file system structure, they look that there was no extra modification. So you can be relatively assured that the only actual change was adding nano to it. And so that's that's super cool and super helpful if you're doing continuous integration and continuous development of these sorts of things. So yeah. So images with Kiwi are sweet as a well documented well maintained project goes way beyond everything else like extremely simple to get started and the community is friendly and helpful for developing advanced setups and the wide range of platform support is unmatched by anything else I've seen so far. And it has a great mechanism for supporting reproducibly built images and the way and it lets you easily track how the artifacts are changing. This is really, really well done stuff and with friendly developers and a friendly community. You really can't go wrong. So here's some references, you know, for the Kiwi website Kiwi get up projects. They've got some sample descriptions that you can use to kind of take a look at how to use it for various, you know, distros and platforms and image types. And then I've got my own demos that I put up on on data is GitHub. There's the link there slides will be available after the talk anyway. And yeah, thank you for coming to my talk. And thanks.
 Hello everyone, my name is Neil Gompine. I'm here to talk to you about sweeter image builds with Kiwi.  Now, Dato's using Kiwi. So first, talk a little bit about me. I call myself a professional technologist.  I've been in technology since I could remember, and I've been involved in Linux for nearly 15 years,  and as a contributor and developer in Fedora, Magia, OpenSUSE, and OpenMandriva Linux distributions.  Note, also contributor to the RPM, DNF, and various related software management, systems management, and image building tools.  Of course, that includes Kiwi, the tool we're talking about today, and I'm a senior DevOps engineer at Dato.  So a little bit about Dato. We were founded in 2007. We've got 22 global locations, over 1,600 employees worldwide,  17,000 managed service provider partners, and we operate entirely in the channel, meaning that all of our products are not directly sold to you.  We sell to companies that take our services and optimize them for their clients' needs, and they sell them to you.  We have local offices in nine countries that help service providers and serve over 1 million small to medium businesses around the world,  or small to medium enterprises. You may have heard that term instead.  We offer a wide variety of products and solutions to support our MSP partners to enable their business to support their clients,  and that ranges from disaster recovery, backups business continuity stuff with our unified continuity products,  with being able to do managed networking solutions with data networking, professional services automation,  and remote monitoring and management with our RMM and PSA products, as well as inventory and quoting, and revenue management with our commerce solutions.  So taking all that and going back to building images at Dato.  So for building images at Dato, we got a little bit of a problem.  We can build images for days.  Practically everyone did image building differently with a different set of issues caused by these methods.  So some tools we're using is Packer, Debian's Live Build for ISOs, custom shell scripts for making disk images and custom base images and all kinds of weird little things.  And we use Vert Builder for making various custom virtual machine disk images that don't really fit into clouds and are used for very specialty use cases.  The problem with this is that we wind up having problems that are just odd.  And this example is great of Packer just not creating images that are sane because the actual mechanism in which it produces the image is not sane.  It goes by pausing the virtual machine and taking a snapshot and exporting it.  Well, this kind of was a problem when you're using Puppet to configure the environment and that includes like setting the machines up for getting unattended updates and things like that.  And the apt daily service in particular just was always started and running just as we were pausing the machine and that left apt and d package in a broken state when you tried to boot up the machine in a new instance.  So we had to do all kinds of things to work around that.  Another example is that when the Spectra meltdown fixes landed a couple years back, Live Build didn't really cope with that very well initially.  And we had to do funny things to kind of make that work.  And that also uncovered another random thing where suddenly XZ compression didn't work anymore, so we had to switch to Gzip.  We don't know.  And this kind of led to a core problem that we've discovered is that some of these tools have seemingly hysterical behaviors, but something that's also been kind of increasingly common among these tools is that they're poorly maintained and that the actual capabilities of them are incomplete.  And these tools don't have, you know, what I would like to call a method to their madness reasoning how they work was too hard. And that make using them difficult because when something went weird in an image build process or in the image produced by the build, it was difficult to walk back and figure out like what happened and what went wrong.  And that really eats a bunch of time that could be better spent doing more valuable things like iterating on it to, you know, build more capabilities, layered systems, solutions and things like that.  So I started looking at Kiwi as a solution to solve this problem, because Kiwi was straightforward and idiomatic. It has XML, YAML or JSON based descriptions with some simple script books that you can use for other flexibility.  And speaking of flexibility, it can build almost any type of image. And if it doesn't know what type of image it is, you can use the Python API for Kiwi to construct custom types of images.  So you can build upon the framework and the tooling inside of Kiwi to construct anything you want.  And it's free and open source software under the GNU General Public License version three, and it's actively developed and maintained and the developers are friendly and helpful and that was huge to me because it didn't actually work right out of the box.  So DATO produces a lot of Ubuntu based images, and that means we have to work with the, you know, making images with the Bootstrap and apt and, you know, there were a couple of issues that I discovered along the way and so I went and fixed them.  Like when I went and sent pull requests, they were very quick to respond with feedback and they worked with me on making sure that it was right and then we got it in and made releases.  They're very good about releasing fixes as they're being merged in. So that was great. And when we started evolving into like handling some sentos based images, that also was somewhat incomplete.  And so I added support for features that we needed that just weren't wired up yet. Perhaps they didn't know or whatever, but like I figured it out and added the functionality.  And this particular change was interesting and cool because I didn't really know how to work with, you know, the extensive test suite that they have for Kiwi.  And they were really quick and responsive and friendly and helpful. All these things for helping me make the unit tests to make sure that the behavior worked because and they were very patient with me trying to figure out the test suite and gave me hints and such.  And that made the huge difference. I was able to be successful and very quickly iterate on this and get this into a state that it could be merged and it was.  And so I want to show you a little bit about Kiwi with, you know, a sample that I from one of the descriptions that I've been working with.  So we have here a container appliance based on sentos stream nine.  And this is working off of test composes that are being released right now sentos stream nine stuff became available last month. And so I started working with it.  So this container appliance is configured using the data package manager you can see here. It's English only because that's the only locale I care about time zone is UTC because screw daylight savings time.  We have locale filtering and turned on so that means that if there's any other locale data or any other locales that are being installed because of packages they'll get filtered out by RPM.  But I also have checked signatures disabled because right now this content is not signed. And so you can't really do anything about that. So I just turned it off.  This other flag which is neat is the exclude doc. So I wanted to make the image super small so I don't need to include documentation stuff since RPM has flags for that we just set it.  And so all the documentation was just stripped out because it just wasn't installed. And we have a small set of packages here just file system the branding packages and some the package manager and some utilities.  And that's about it. And this is just a simple TVZ type image which basically means it's just a tar ball with files in it. And we also should say the this is the shell hook that is used you could actually replace this with calling Ansible or or running some kind of  other tool, another language or whatnot because this can be anything as long as it is called config.sh it will execute it. And so you can have this do anything you'd like.  This one's just super simple about setting it up to be multi user with the root user populated and setting the host name is local host here.  And then cleaning out any extra unwanted locales that may have not been correctly marked as local data by RPM. And so let's just go ahead and build it.  Right. So you can see it's running this through. And we want to see the log here.  And you can see this is setting up the image there and it's all going through Kiwi here. You see that it was setting up the repos then it bootstrapped and installed the packages.  After the package installation was done, it ran the scripts, the script helpers, and is creating the tar ball right now. So you can see it's doing that.  It's an XZ compressed tar ball with multiple threads. So it's like using all the cores on the computer to actually produce the tar ball.  And after that, it will.  And while it's creating that I've already pre created this before so we're going to go ahead and boot it up in and spawn. So we'll just start that up.  And now you see that it's got the it starts up like it would a regular computer. So this container was particularly configured to behave like.  You know how you would start up a VM so it starts up system D and then starts log in D. So I can do route my super secure passwords.  And go in here and there's nothing in here but I can see this and I see that and then Etsy you can see there's all these files.  So let's catOS release. You see it sent us stream nine with all the stuff in here, you know, and then CDDNF.  There's all the directories you'd expect here. But maybe I wanted it, you know, I wanted an editor to edit a files like, you know, let's let's edit Dnf.  And oh, that's not there. Well, then what do I do. So we will go back and do that. So like just seeing here. But before we go back and do that, let's take a look at what it actually produced.  So I showed you what it looks like. Let's let's take a look at what it actually what it actually produced temp output container.  So the changes file is really just a concatenation of change logs for all the different packages that were included.  So GCC you can see in here and you see open SSL, Python, all kinds of stuff in here. And then if you look at packages, this one's actually just a list of all the packages and this has the name group, the name of the version release architecture and the license data.  And this is useful if you want to see whether things have changed between build to build, and also to see what licenses are included in the whole thing.  And then the verified one is for checking the file system structure to see how much of it differs from what the RPM database knows.  So you can see that yeah there was some files that were deleted because I purged extra locale data. And you can see that there's you know some modifications of some config files and stuff. And that's fine.  So, but we want to have a text editor in there. So let's add that. So TXC build config XML. And we're just going to go ahead and add another package.  So we'll add nano because nano is the best text editor ever. And we're going to go ahead and create this to build again.  We're going to call it temp output two for this directory. We're going to run it again.  And so this sets it up again and it goes through this process. And if we want to take a look at what this looks like, we'll take a look at the log here.  It's already at the point of creating the image again.  And once that's actually done, we will have something to show off. But meanwhile, while that is happening, we should go ahead and  clean up or not actually, we can.  Well, we do need to be root here because otherwise we can't actually write the terrible but probably want to be in and disc.  Vc temp output two. So we're ready for that. And it's actually a it's done. So the that part is created. And we're going to go ahead and tar XVF container C bar live machines.  EL nine appliance to. All right, let's make the directory bar live machines.  EL nine appliance to.  And so that extracts all the files for that because this is just a regular tar ball. So it only has those particular things. So now we will boot up the second appliance that we just created.  So this is just like the first one except now it has nano installed.  And now, if I go to at C, we go to DNF, and I go to DNF dot com. I have the nano text editor.  Now let's let's check and see like what this difference is actually look like.  All right, so we made the temp output and temp output two directories. And so let's take a look at diff of temp output.  If you have container EL nine packages to temp output to container EL nine packages.  And you can see we added just one package nano. Now let's see what that looks like for change logs. This is actually going to be pretty bad because change logs are kind of huge. But it'll kind of illustrate the point I was trying to make.  So you see there's a whole new change log that was added that is literally just for nano.  There was also some other sorting things and that's kind of the reason why I didn't want to show that real thing about that too hard. Let's take a look at verified. Let's see if there was any other differences.  I expect that we might not have any see no differences. So from a file system structure, they look that there was no extra modification. So you can be relatively assured that the only actual change was adding nano to it.  And so that's that's super cool and super helpful if you're doing continuous integration and continuous development of these sorts of things. So yeah.  So images with Kiwi are sweet as a well documented well maintained project goes way beyond everything else like extremely simple to get started and the community is friendly and helpful for developing advanced setups and the wide range of platform support is unmatched by anything else I've seen so far.  And it has a great mechanism for supporting reproducibly built images and the way and it lets you easily track how the artifacts are changing. This is really, really well done stuff and with friendly developers and a friendly community.  You really can't go wrong.  So here's some references, you know, for the Kiwi website Kiwi get up projects. They've got some sample descriptions that you can use to kind of take a look at how to use it for various, you know, distros and platforms and image types.  And then I've got my own demos that I put up on on data is GitHub. There's the link there slides will be available after the talk anyway.  And yeah, thank you for coming to my talk. And thanks.
openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_0_3635.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_3636_3719.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_3720_7547.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_7548_7727.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_7728_10919.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_10920_14555.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_14556_14987.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_14988_15995.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_15996_16151.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_16152_16199.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_16200_16283.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_16284_16391.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_16392_18479.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_18480_18551.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_18552_18647.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_18648_19199.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_19200_19523.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_19524_19643.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_19644_22823.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_22824_23699.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_23700_23747.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_23748_26039.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_26040_26447.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_26448_26495.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_26496_27887.png openSUSE_Virtual_Conference_2021_-_The_path_to_swe_KcK2D9JC0IE_27888_28293.png