System containers at scale
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Graber, Stéphane
LXD is a system container manager, its goal is to safely run full Linux systems at very high density and low overhead. Containers may be created from pre-made images, covering most Linux distributions, or by importing an existing virtual machine or physical system. Advanced resource control and device passthrough is available to expose as much or as little system resources to those containers. Snapshot and backup tooling is available to safeguard those containers and data. Storage pools and networks can be used to offer a variety of storage and network options to the containers. Management happens through a REST API with a default CLI client. LXD has built-in support for clustering which makes it trivial to scale a deployment to dozens of servers, all acting as one virtual LXD server. In this presentation, we'll go over LXD's main features through a demonstration including usage of LXD's clustering abilities, running a variety of Linux distributions and converting existing systems to containers.
 Our next speaker is Stefan Graber. He's the LexD project leader and one of the maintainers and one of the core developers. And he's going to talk about system containers at scale. Hello. All right. So let's talk about first system containers and LexD in clustering. So what are system containers? System containers are effectively the oldest type of containers. They've been around for quite a while, originating with BSDJLs. Then as Linux Vserver on Linux, it was a patch set from over a decade ago. Then Zororiz Zones, sorry, obviously, OpenVZ on Linux, which was also an insanely large patch set on top of Linux kernel. And then as we were streaming things in Linux kernel, LXC and LexD came as the runtime to drive that. System containers behave very much like standard load systems. You run a full Linux distribution. It's not like a single process type thing that you get with Docker and other application containers. You don't need specialized software images or anything. You treat them just like virtual machines, effectively. They're very low overhead, easy to manage. You can run thousands of them on systems. There is no vert overhead that just take up physical resources. There's no need for hardware, accelerated, anything. And as far as the host is concerned, it's just a bunch of processes. So you can go on the host and you see all your processes. It's nice and easy. Now what's LexD? So LexD is a modern system container manager. That's written in Go. It is its libxc to drive containers. It's got a rest API, a bunch of rest API clients. And as you can see here, you can have multiple hosts running Linux, then the LXC layer to drive the kernel, and then next to the on top, expose the rest API, and then we've got a number of clients that can talk to that. More of what's LexD? So LexD is designed to be simple. It's a very clean common line interface, pretty simple rest API. We've got bindings in a lot of languages to make it easy for people to drive system containers for LexD. It is very fast. It is based on its images. So it's no more like creating a root file system with the bootstrap or whatever. It's got optimized storage and migration over the network. It's got direct hardware access because there are containers and we've got nice semantics to pass GPUs, USB devices, et cetera. It's secure. So we use all of the kernel namespaces by default. We also use LSMs like Habana. We use SecComp. We use capabilities. We use pretty much everything at this puzzle to make it safe. It's scalable. And that's what we'll see most in this talk. It can go from just a single container on the laptop to tens of thousands of containers running in a cluster. As far as what we can run on top of LexD, we've got a lot of images that are generated daily for all of those distros plus a few more that literally couldn't fit in the slide anymore. So we built for about 18 different distros, about 77 different releases all combined, which ends up being over 300 images we build every day that people can use to run on LexD. You can also build your own, but we've got a lot of them. LexD is printed, is effectively on Chromebooks. So if you've seen that Linux feature on Chromebooks, then gets you like a Debian shell, that's using LexD. So we've got a decent user base throughout. And that feature includes integration for like snapshots, backups, file transfer, GUI access, GPU access, Suncard access, webcam access. They really went with it on the Chromebooks. The little piece of what LexD is used right now is on TravisCR. So you can, if you run any job on Travis that is not Intel 64-bit, so if you use ARM64, if you use PowerPC, or if you use IBM Z, all of those platforms are using LexD containers with like an extremely quick start of time of usually less than two seconds running on shared systems with all of the security in place and some of the SQL Interception stuff that Christian demoed has been done partly as part of that. Now, for the LexD components, go through this quickly. That's kind of the main things we've got in IAPI. Clustering is what we'll demo today. I'll say to image base, we've got images and then image aliases to have nice names on images. We've got instances, so those are containers, but these days it can also be virtual machines. That's a new thing we added a few weeks back. We've got snapshots and backups for instances. We've got network management to create new network bridges that you can use for your instances. We've got projects that get you your own individual view on a shared LexD server, effectively. So you can have conflicting, like there's no more conflict with container names or anything of that, so long as they're in different projects. And we've got storage with a variety of storage drivers with support, and you can create custom volumes and do snapshots and all that. Some internal bits are mostly to get notified when something happens on LexD or for access control. And we support doing file transfers and spawning applications directly in containers and virtual machines, accessing console, and publishing containers to images. So now for the main topic of this talk. LexD has had clustering support for about two years now. It's really built into LexD. There are no external dependencies. It works on LexD's chip or no hire. Insertions can just be turned into a cluster member, and you can easily join an insertion into the cluster. There's really no external component you need for any other art. It works using the same API as you have for a single node. It's got a few more bits you can use through the API to see, actually, one something to be specifically on this machine. But if your client is not aware of clustering and just throws things at LexD exactly like if it was a standard node, things will just work. The cluster will just balance things for you, and you'll never know if you're even talking to a cluster. And it can scale quite nicely. So we can run containers and dozens of nodes. We've actually run clusters of 50 to 100 nodes, and they still mostly work. And each of those can run hundreds to thousands of containers, so very high density, depending on what you're running. We've also added, and that's very recent as a few weeks ago, support for mixed architecture. So you can have cluster nodes that are different architectures. And when you schedule, when you ask for a particular image to be used to create a container or virtual machine, you just pick whatever node is capable of running that given architecture. All right. Now for an interesting part of this. Let's see how that works. OK, so for this, I've got three systems. Actually, I need to connect to a third one. OK, now that it's connecting. So what we'll do is LexD is installed. That's LexD 320 that we released two days ago. So just configure the first node. So you want to set up a cluster. Let's go with yes. Need to enter its IP because the link log all is not going to be fun enough. This one. We're not joining any existing cluster, because we want to build a new one. Yeah, we need to set the password. Let's configure some storage. So let's go barfs. Create barfs. That's fine. Is there anything in special to do on this one? Yeah, OK, that one is a bit different. So we just need to tell it what's the shared subnet for all of those. So that's my subnet at home. OK. All right, so right now you've got a single LexD part of a cluster, but it's the only one in there. So you can see. Now let's go on to the next one. And repeat this thing. It's going to ask less questions because it's just joining. So one's clustering. Its IP address is the art, I believe. Joining is in clusters. Yes, and the other node was on 16, 46. OK, so it's asking for the password entered earlier. Yes, everything's going to go away when we're joining. Size, we don't care. Size, we don't care. OK, so now we're joined. We should see that we've got two nodes and things to work. Now to make things slightly more interesting. So those systems were in the Lex86, nothing super special that Xeon CPUs. Now we've got one that is not in the Lex86. So this one is running on 64. Same thing. Next to init, cluster, name is fine. This is wrong. I forgot to connect that. So it's actually a nested container because I didn't have a spare 64 system. So I'm just doing next to nesting for that one. But I connected it to the wrong network, so I'm just fixing that. OK, so that's it again. The IP should be right now. OK, so cluster and yes. Name is correct. IP is correct this time. Joining these interesting clusters, yes. And we set the 1646 cluster password. Yes, we're cool with that. Size, we don't care. Because it's a nested container, it can't create a loop device. I need to actually tell it where the storage is. And it should be the end of that. OK, let's go back to one of the X86 nodes. So now if I list the cluster, we see we've got three. And one of them is AR64 instead of Intel. Now let's show some stuff at it. So just create a container called C1. This one is, I didn't specify what architecture actually want, so it's kind of surprised me. LexD will pick whichever it considers to be the least busy server and just schedule a container there. So it's probably going to be on one of the X86 ones. Yeah, it's on AdFu, which is one of the X86 servers. OK, now let's do another one. Let's do centers. I think my guess would be it's going to go on a tarot, which is the other X86 system. And then the third one will most likely be scheduled on ARM. Let's see. Yep. It doesn't have an IP yet, but that's going to fix itself. There we go. And let's do alpine. That name is already taken. And it's just out of order. OK, so C2, and that's going to be on ARM. OK, so now if I go on there. So keep forgetting that alpine is in Parabash. There we go. So yeah, I'm just executing a comment in there. And we can see it's running on the R64. So LexD is doing all the API forwarding for us. So I'm on one machine talking to the cluster, and just go on to the right node and kind of query it there. The other thing that's somewhat interesting is we've got a tool to convert a system into a container. So that's what we've got here. That VM01 is a CentOS 7 VM. That's just doing nothing but it's there. We've got a tool called LexD P2C that can take the address of the cluster. We'll ask for the same password we set, and we'll then transfer the entire thing over the network into LexD, creating a new container for you. That uses the entire content of the system. That's going to take a little while. So I'm just going to let it run. While that's going on, I want to show the new cool thing we've added. So all of the LexD networking, storage, and configuration bits, because our containers act so much like virtual machines, the same concept really applied to actual virtual machines. So we figured, well, why not just allow running virtual machines as well using the exact same storage and configuration? So that's what we've got. I can do launch. And you'll notice I've got just an extra thing at the end. You just dash, dash VM. That's pretty much the only difference. In this case, I don't want it to go on ARM, because since that ARM host is a container inside a VM on ARM, there's no way I can run a VM inside there. But the X86 machines are running on physical hardware, so those will be just fine. We do support running VMs on ARM, but you need to run on the actual hardware, which is not the case here. So the images are a bit larger, because there's no machines. But still downloading, unpacking that, creating the storage volume on the RFS, I think it was this time. And now if you do console. So console works fine on containers, too, just to show you. If I do C1, there you go. So on the container, you get attached to the console, and on a VM. Did I call it? Yeah. Yeah, not fun. Well, VM01 is the one I'm transferring in as a container. I would have expected console V1 to actually function. Where did it go? Is it just because it's confused? I swear that worked on it. OK, all right. All right, so it was a bit picky. So we see the same thing, a VM was booting. I touched a bit late. Let's just go back to this guy here. We can just launch a second one of them. Let's see if this one behaves properly. Creating V2. Come on. You can do it. Oh, yeah, that one takes a tiny bit, because since it's load balancing within the cluster, and the node it picked doesn't have the image yet, it's doing an internal cluster transfer of the image. So it's not putting it from the internet again. But it still needs to move it around. It's optimized. It uses burrfs send receive in this case. It doesn't use rsync or anything. It's pretty optimized. So we can see we're in the boot loader, and then booting the VM. And lastly, just to show you that I'm hoping that VM is done transferring the CentOS thing. It is. So if we go here, we can start VM01, which is a container that was treated from the CentOS system. And there we go. All right. How am I doing on time? I'm getting two minutes behind. So LexD is available on, obviously Linux. That's where we run. But we also have a Windows and Mac OS client so that you can talk to a remote LexD if you've got an Raspberry Pi or Intel NUC or something you want to run LexD on. If you want to contribute to LexD, it's written in Go. It's pretty translatable. We've got client libraries for a bunch of languages. It's a Apache 2 license. There's no copyright assignment or anything in there. We've got a good community you can work with. And we've got usually a bunch of smaller issues, good starting points to contribute. That's it. If we've got some questions, we can take them now. And we've got stickers towards the exit when you leave if you want any others. Questions? Sorry for speaking so fast, but it turns out 20 minutes is pretty short. Thank you very much. Two quick questions. What do you think about running Kubernetes inside the LexD containers or LexD containers here? Yeah. So yes, we can do it either way around. Some people have been, you can run Kubernetes, especially things like API servers and stuff inside LexD containers, no problem. You can even run KubeNet inside LexD containers, because we support nesting and we support running Docker inside LexD containers. So that's possible. And people have done it before. And you can actually do it the other way around as well, where LXE is a community project that implements a CRI for Kubernetes that then drives LexD containers. So you can do it either way. But yeah, it's possible. And then question, why cut a container exist if LexD is secure? Why what? Why cut a container exist if LexD is secure system containers? It's always, so that depends on people. That depends what they trust. We've seen that hardware is not particularly safe either. Some people think that relying on the Linux kernel for the entirety of the security story is quite fine. Some people think that VMs are the only option. Some people think that you need both. In fact, on Chromebooks, Google is on purpose using both. So they're using a virtual machine layer and then running only on previous containers inside there, so that if the kernel is busted, you're staying in a VM. If the VM is busted, you're staying on a previous user namespace. Because we've seen exploits against both in the past. Recently, we've actually seen more CVEs and security issues around both the hardware bits of attrition and some of the hypervisor stuff than we have against the Linux kernel as far as escape of containers. But I mean, there's always a risk. That's kind of up to you what's fine with you, what's not. Combining both is also the slowest option. But it's there. Some people have done it. OK. Two short questions. First, do you support foreign containers on the host? I mean, running ARM containers on 64 or something like this? Sorry, I'm not asking a question. Foreign containers. Oh, yeah. OK. So you architecture emulation on the system, effectively. So like ARM on x86? Yeah. So we did that in the past. I did implement support for that in LXC, almost, I don't know, five, six, seven years ago, using QMU user static. It is possible. It is not pleasant. And it's not something we want to ever have to support again. The main issue being that the QMU user static layer cannot handle properly threads or net link and some other things like that. So we had to do a very, very weird container where most of the binaries were indeed ARM. But the init system and the network tools were x86, which works, but is really, really weird. And as soon as you start doing updates and stuff against those containers, just quickly get into a really weird state. So not something we're particularly keen on revisiting at this point. It is possible. You can make it work. Like you could create a custom image that bundles QMU user static at the right location and with the right bin format configuration. Lexi will let you do it and it will just work. But not something we want to support. Actually, I do just a bind of host system works. Want to see a native implementation. But anyway, second question. You talk about clustering. What about roaming clustering nodes? If you remove your notebook, for example, with a node, somewhere else. Yeah. So that's what is kind of tricky. So Lexi has moved supports to move containers around. That works fine, but usually you want to stop them. If they're running, then you just create you, that Adrian talked about earlier, which can work in some cases, but can also tends to fail with a lot of modern software. As far as the storage bits, one thing that's interesting is that Lexi does support Ceph as a storage driver. And so if your container is backed by Ceph, at least if a node goes down, you can always start the containers back up anywhere else you want, because the data is on the network. So that's kind of what we have there. Anything we're out of time? Yep. We're out of time. Thanks very much. Thanks. Thank you.
 Our next speaker is Stefan Graber.  He's the LexD project leader and one of the maintainers  and one of the core developers.  And he's going to talk about system containers at scale.  Hello.  All right.  So let's talk about first system containers and LexD  in clustering.  So what are system containers?  System containers are effectively the oldest type  of containers.  They've been around for quite a while,  originating with BSDJLs.  Then as Linux Vserver on Linux, it was a patch set  from over a decade ago.  Then Zororiz Zones, sorry, obviously,  OpenVZ on Linux, which was also an insanely large patch set  on top of Linux kernel.  And then as we were streaming things in Linux kernel,  LXC and LexD came as the runtime to drive that.  System containers behave very much like standard load systems.  You run a full Linux distribution.  It's not like a single process type thing  that you get with Docker and other application containers.  You don't need specialized software images or anything.  You treat them just like virtual machines, effectively.  They're very low overhead, easy to manage.  You can run thousands of them on systems.  There is no vert overhead that just  take up physical resources.  There's no need for hardware, accelerated, anything.  And as far as the host is concerned,  it's just a bunch of processes.  So you can go on the host and you see all your processes.  It's nice and easy.  Now what's LexD?  So LexD is a modern system container manager.  That's written in Go.  It is its libxc to drive containers.  It's got a rest API, a bunch of rest API clients.  And as you can see here, you can have multiple hosts running  Linux, then the LXC layer to drive the kernel,  and then next to the on top, expose the rest API,  and then we've got a number of clients that can talk to that.  More of what's LexD?  So LexD is designed to be simple.  It's a very clean common line interface,  pretty simple rest API.  We've got bindings in a lot of languages  to make it easy for people to drive system containers for LexD.  It is very fast.  It is based on its images.  So it's no more like creating a root file system  with the bootstrap or whatever.  It's got optimized storage and migration over the network.  It's got direct hardware access because there are containers  and we've got nice semantics to pass GPUs, USB devices,  et cetera.  It's secure.  So we use all of the kernel namespaces by default.  We also use LSMs like Habana.  We use SecComp.  We use capabilities.  We use pretty much everything at this puzzle to make it safe.  It's scalable.  And that's what we'll see most in this talk.  It can go from just a single container on the laptop  to tens of thousands of containers running in a cluster.  As far as what we can run on top of LexD,  we've got a lot of images that are generated daily  for all of those distros plus a few more that literally  couldn't fit in the slide anymore.  So we built for about 18 different distros,  about 77 different releases all combined,  which ends up being over 300 images we build every day  that people can use to run on LexD.  You can also build your own, but we've got a lot of them.  LexD is printed, is effectively on Chromebooks.  So if you've seen that Linux feature on Chromebooks,  then gets you like a Debian shell, that's using LexD.  So we've got a decent user base throughout.  And that feature includes integration  for like snapshots, backups, file transfer, GUI access,  GPU access, Suncard access, webcam access.  They really went with it on the Chromebooks.  The little piece of what LexD is used right now is on TravisCR.  So you can, if you run any job on Travis  that is not Intel 64-bit, so if you use ARM64,  if you use PowerPC, or if you use IBM Z,  all of those platforms are using LexD containers  with like an extremely quick start of time of usually  less than two seconds running on shared systems  with all of the security in place  and some of the SQL Interception stuff  that Christian demoed has been done partly as part of that.  Now, for the LexD components, go through this quickly.  That's kind of the main things we've got in IAPI.  Clustering is what we'll demo today.  I'll say to image base, we've got images and then image  aliases to have nice names on images.  We've got instances, so those are containers,  but these days it can also be virtual machines.  That's a new thing we added a few weeks back.  We've got snapshots and backups for instances.  We've got network management to create new network bridges  that you can use for your instances.  We've got projects that get you your own individual view  on a shared LexD server, effectively.  So you can have conflicting, like there's  no more conflict with container names or anything of that,  so long as they're in different projects.  And we've got storage with a variety of storage drivers  with support, and you can create custom volumes  and do snapshots and all that.  Some internal bits are mostly to get notified when something  happens on LexD or for access control.  And we support doing file transfers and spawning  applications directly in containers and virtual machines,  accessing console, and publishing containers to images.  So now for the main topic of this talk.  LexD has had clustering support for about two years now.  It's really built into LexD.  There are no external dependencies.  It works on LexD's chip or no hire.  Insertions can just be turned into a cluster member,  and you can easily join an insertion into the cluster.  There's really no external component  you need for any other art.  It works using the same API as you have for a single node.  It's got a few more bits you can use through the API  to see, actually, one something to be specifically  on this machine.  But if your client is not aware of clustering  and just throws things at LexD exactly like if it  was a standard node, things will just work.  The cluster will just balance things for you,  and you'll never know if you're even talking to a cluster.  And it can scale quite nicely.  So we can run containers and dozens of nodes.  We've actually run clusters of 50 to 100 nodes,  and they still mostly work.  And each of those can run hundreds to thousands  of containers, so very high density,  depending on what you're running.  We've also added, and that's very recent as a few weeks ago,  support for mixed architecture.  So you can have cluster nodes that  are different architectures.  And when you schedule, when you ask for a particular image  to be used to create a container or virtual machine,  you just pick whatever node is capable of running that  given architecture.  All right.  Now for an interesting part of this.  Let's see how that works.  OK, so for this, I've got three systems.  Actually, I need to connect to a third one.  OK, now that it's connecting.  So what we'll do is LexD is installed.  That's LexD 320 that we released two days ago.  So just configure the first node.  So you want to set up a cluster.  Let's go with yes.  Need to enter its IP because the link log all  is not going to be fun enough.  This one.  We're not joining any existing cluster,  because we want to build a new one.  Yeah, we need to set the password.  Let's configure some storage.  So let's go barfs.  Create barfs.  That's fine.  Is there anything in special to do on this one?  Yeah, OK, that one is a bit different.  So we just need to tell it what's the shared subnet  for all of those.  So that's my subnet at home.  OK.  All right, so right now you've got a single LexD part  of a cluster, but it's the only one in there.  So you can see.  Now let's go on to the next one.  And repeat this thing.  It's going to ask less questions because it's just joining.  So one's clustering.  Its IP address is the art, I believe.  Joining is in clusters.  Yes, and the other node was on 16, 46.  OK, so it's asking for the password entered earlier.  Yes, everything's going to go away when we're joining.  Size, we don't care.  Size, we don't care.  OK, so now we're joined.  We should see that we've got two nodes and things to work.  Now to make things slightly more interesting.  So those systems were in the Lex86,  nothing super special that Xeon CPUs.  Now we've got one that is not in the Lex86.  So this one is running on 64.  Same thing.  Next to init, cluster, name is fine.  This is wrong.  I forgot to connect that.  So it's actually a nested container  because I didn't have a spare 64 system.  So I'm just doing next to nesting for that one.  But I connected it to the wrong network, so I'm just fixing that.  OK, so that's it again.  The IP should be right now.  OK, so cluster and yes.  Name is correct.  IP is correct this time.  Joining these interesting clusters, yes.  And we set the 1646 cluster password.  Yes, we're cool with that.  Size, we don't care.  Because it's a nested container, it can't create a loop device.  I need to actually tell it where the storage is.  And it should be the end of that.  OK, let's go back to one of the X86 nodes.  So now if I list the cluster, we see we've got three.  And one of them is AR64 instead of Intel.  Now let's show some stuff at it.  So just create a container called C1.  This one is, I didn't specify what architecture actually  want, so it's kind of surprised me.  LexD will pick whichever it considers to be the least busy  server and just schedule a container there.  So it's probably going to be on one of the X86 ones.  Yeah, it's on AdFu, which is one of the X86 servers.  OK, now let's do another one.  Let's do centers.  I think my guess would be it's going to go on a tarot, which  is the other X86 system.  And then the third one will most likely be scheduled on ARM.  Let's see.  Yep.  It doesn't have an IP yet, but that's going to fix itself.  There we go.  And let's do alpine.  That name is already taken.  And it's just out of order.  OK, so C2, and that's going to be on ARM.  OK, so now if I go on there.  So keep forgetting that alpine is in Parabash.  There we go.  So yeah, I'm just executing a comment in there.  And we can see it's running on the R64.  So LexD is doing all the API forwarding for us.  So I'm on one machine talking to the cluster,  and just go on to the right node and kind of query it there.  The other thing that's somewhat interesting  is we've got a tool to convert a system into a container.  So that's what we've got here.  That VM01 is a CentOS 7 VM.  That's just doing nothing but it's there.  We've got a tool called LexD P2C that  can take the address of the cluster.  We'll ask for the same password we set,  and we'll then transfer the entire thing over the network  into LexD, creating a new container for you.  That uses the entire content of the system.  That's going to take a little while.  So I'm just going to let it run.  While that's going on, I want to show the new cool thing  we've added.  So all of the LexD networking, storage, and configuration  bits, because our containers act so much like virtual machines,  the same concept really applied to actual virtual machines.  So we figured, well, why not just allow running virtual machines  as well using the exact same storage and configuration?  So that's what we've got.  I can do launch.  And you'll notice I've got just an extra thing at the end.  You just dash, dash VM.  That's pretty much the only difference.  In this case, I don't want it to go on ARM,  because since that ARM host is a container inside a VM on ARM,  there's no way I can run a VM inside there.  But the X86 machines are running on physical hardware,  so those will be just fine.  We do support running VMs on ARM,  but you need to run on the actual hardware, which  is not the case here.  So the images are a bit larger, because there's no machines.  But still downloading, unpacking that,  creating the storage volume on the RFS,  I think it was this time.  And now if you do console.  So console works fine on containers, too, just to show you.  If I do C1, there you go.  So on the container, you get attached to the console,  and on a VM.  Did I call it?  Yeah.  Yeah, not fun.  Well, VM01 is the one I'm transferring in as a container.  I would have expected console V1 to actually function.  Where did it go?  Is it just because it's confused?  I swear that worked on it.  OK, all right.  All right, so it was a bit picky.  So we see the same thing, a VM was booting.  I touched a bit late.  Let's just go back to this guy here.  We can just launch a second one of them.  Let's see if this one behaves properly.  Creating V2.  Come on.  You can do it.  Oh, yeah, that one takes a tiny bit,  because since it's load balancing within the cluster,  and the node it picked doesn't have the image yet,  it's doing an internal cluster transfer of the image.  So it's not putting it from the internet again.  But it still needs to move it around.  It's optimized.  It uses burrfs send receive in this case.  It doesn't use rsync or anything.  It's pretty optimized.  So we can see we're in the boot loader,  and then booting the VM.  And lastly, just to show you that I'm  hoping that VM is done transferring the CentOS thing.  It is.  So if we go here, we can start VM01,  which is a container that was treated from the CentOS system.  And there we go.  All right.  How am I doing on time?  I'm getting two minutes behind.  So LexD is available on, obviously Linux.  That's where we run.  But we also have a Windows and Mac OS client  so that you can talk to a remote LexD  if you've got an Raspberry Pi or Intel NUC or something  you want to run LexD on.  If you want to contribute to LexD,  it's written in Go.  It's pretty translatable.  We've got client libraries for a bunch of languages.  It's a Apache 2 license.  There's no copyright assignment or anything in there.  We've got a good community you can work with.  And we've got usually a bunch of smaller issues,  good starting points to contribute.  That's it.  If we've got some questions, we can take them now.  And we've got stickers towards the exit when you leave  if you want any others.  Questions?  Sorry for speaking so fast, but it turns out 20 minutes  is pretty short.  Thank you very much.  Two quick questions.  What do you think about running Kubernetes inside the LexD  containers or LexD containers here?  Yeah.  So yes, we can do it either way around.  Some people have been, you can run Kubernetes,  especially things like API servers and stuff inside LexD  containers, no problem.  You can even run KubeNet inside LexD containers,  because we support nesting and we support running Docker  inside LexD containers.  So that's possible.  And people have done it before.  And you can actually do it the other way around as well,  where LXE is a community project that  implements a CRI for Kubernetes that then  drives LexD containers.  So you can do it either way.  But yeah, it's possible.  And then question, why cut a container exist if LexD is secure?  Why what?  Why cut a container exist if LexD is secure system  containers?  It's always, so that depends on people.  That depends what they trust.  We've seen that hardware is not particularly safe either.  Some people think that relying on the Linux  kernel for the entirety of the security story is quite fine.  Some people think that VMs are the only option.  Some people think that you need both.  In fact, on Chromebooks, Google is on purpose using both.  So they're using a virtual machine layer and then running  only on previous containers inside there, so that if the  kernel is busted, you're staying in a VM.  If the VM is busted, you're staying on a previous user  namespace.  Because we've seen exploits against both in the past.  Recently, we've actually seen more CVEs and security issues  around both the hardware bits of attrition and some of the  hypervisor stuff than we have against the Linux kernel as  far as escape of containers.  But I mean, there's always a risk.  That's kind of up to you what's fine with you, what's not.  Combining both is also the slowest option.  But it's there.  Some people have done it.  OK.  Two short questions.  First, do you support foreign containers on the host?  I mean, running ARM containers on 64 or something like this?  Sorry, I'm not asking a question.  Foreign containers.  Oh, yeah.  OK.  So you architecture emulation on the system,  effectively.  So like ARM on x86?  Yeah.  So we did that in the past.  I did implement support for that in LXC, almost, I don't  know, five, six, seven years ago, using QMU user static.  It is possible.  It is not pleasant.  And it's not something we want to ever have to support again.  The main issue being that the QMU user static layer cannot  handle properly threads or net link and some other things  like that.  So we had to do a very, very weird container where most of  the binaries were indeed ARM.  But the init system and the network tools were x86, which  works, but is really, really weird.  And as soon as you start doing updates and stuff against  those containers, just quickly get into a really weird state.  So not something we're particularly keen on  revisiting at this point.  It is possible.  You can make it work.  Like you could create a custom image that bundles QMU user  static at the right location and with the right bin  format configuration.  Lexi will let you do it and it will just work.  But not something we want to support.  Actually, I do just a bind of host system works.  Want to see a native implementation.  But anyway, second question.  You talk about clustering.  What about roaming clustering nodes?  If you remove your notebook, for example, with a node,  somewhere else.  Yeah.  So that's what is kind of tricky.  So Lexi has moved supports to move containers around.  That works fine, but usually you want to stop them.  If they're running, then you just create you, that Adrian  talked about earlier, which can work in some cases, but can  also tends to fail with a lot of modern software.  As far as the storage bits, one thing that's interesting is  that Lexi does support Ceph as a storage driver.  And so if your container is backed by Ceph, at least if  a node goes down, you can always start the containers  back up anywhere else you want, because the data is on the  network.  So that's kind of what we have there.  Anything we're out of time?  Yep.  We're out of time.  Thanks very much.  Thanks.  Thank you.
containers_lxd_0_167.png containers_lxd_168_263.png containers_lxd_264_323.png containers_lxd_324_443.png containers_lxd_444_575.png containers_lxd_576_707.png containers_lxd_708_2627.png containers_lxd_2628_3263.png containers_lxd_3264_4655.png containers_lxd_4656_5255.png containers_lxd_5256_5939.png containers_lxd_5940_6647.png containers_lxd_6648_8387.png containers_lxd_8388_10703.png containers_lxd_10704_10835.png containers_lxd_10836_11039.png containers_lxd_11040_11195.png containers_lxd_11196_12791.png containers_lxd_12792_13847.png containers_lxd_13848_14003.png containers_lxd_14004_15683.png containers_lxd_15684_18899.png containers_lxd_18900_18947.png containers_lxd_18948_21323.png containers_lxd_21324_21743.png containers_lxd_21744_22835.png containers_lxd_22836_22883.png containers_lxd_22884_23363.png containers_lxd_23364_23459.png containers_lxd_23460_23771.png containers_lxd_23772_24299.png containers_lxd_24300_31703.png containers_lxd_31704_31825.png