Portable Services are Ready to Use
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Poettering, Lennart
Portable Services bring certain aspects of containers to classic systemd service management. With systemd v239 Portable Services are for the first time complete and ready for users to take advantage of. In this talk we'll have a look on the underlying technical concepts, how things fit together and what the precise limitations and benefits are.
 Hi, I'm Lena Padaling, I work on systemd. I'm going to talk about portable services already to use. Portable services is a new concept in systemd. I'd like to introduce you to this. I did this talk before by the way. You might have seen it if you attended DefConf or if you said I've seen it all systems ago. There's not going to be much new in that if you saw it there and probably would be cool if there are still people outside. If you have seen it, you wouldn't miss anything. But apparently nobody has seen it yet. If you have any questions, completely, totally interrupt me right away. I much prefer my talks to be like conversations instead of just a Q&A at the end. So do not hesitate, interrupt me. I love that. Yeah, let's jump right in. Portable services. What are portable services? Portable services you can see two ways. One way to see it is they're system services with some container features. You can also see the other way around. They're kind of like containers but with some system service features. What does that mean? First of all, we have to understand what containers are. I don't really know what containers are. People have different ideas about containers, what they precisely consist of. I think the definition that I tend to agree on is that they combine two, like three concepts. One is resource bundles, right? That's what you do with containers. You pack them all up in a couple of towels, like shed libraries and all the dependencies. There's isolation, meaning that they generally run in some form of sandbox. Like, might be a better one, might be a worse one, but there's at least some form of isolation generally. And delivery, right? Like, you're delivering on the server you want to deploy them on and then you can run them there. So these three concepts is what at least I find most interesting about containers. I'm pretty sure that other people who care about containers probably would list more of that. Portable services try to take some of these features and add them to classic service management. Specifically, Portable services about making resource bundles available for regular system services. There are about integration, right? Like, if you look at the previous slide I had isolation there, I put integration here, right? Like, because that's what system services are really in comparison to containers. They're generally much more tightly integrated to the host system. And sandboxing, right? I do not put delivery on this slide because I don't really care about delivery. But yeah. So the difference really here is I don't care about delivery and instead of isolation I put integration and sandboxing here. Sandboxing for me is slightly different from isolation because like the way I see isolation, it's really about creating a new world that is separate from the host you live in. While sandboxing I kind of more see is like you're living still in the same world but you can't do everything you want to do. Another key aspect of Portable services that's supposed to be highly modular which like to show the opposition to classic container management is that like in container management you generally tend to have to buy into the whole idea, right? And then some people don't do that and for example end up with containers which they introduce the concept of super privileged containers which are basically containers where you just throw one-half of the concept away by turning off all the isolation features. But yeah, Portable services are not supposed to be like that. Portable services are supposed to be modular. You pick exactly what you want. Do you want the resource bundling? And if you want to have sandboxing you pick exactly how much isolation or integration you want. So it's really supposed to be modular and very, very fine grained and doesn't require you to buy into the whole idea but into parts only. Another way to look at it, consider range from integrated to isolated. If I was a good graphics artist I would actually have drawn a graph here but I'm lazy, very lazy so I just put a couple of words here with an arrow in the middle. Think about a range from integrated to isolated, right? On one side you have the classic system services, right? Like these are system five services or system D services. They tend to be very well integrated into the host, right? They live in the same world, they see the same network interface, they name file systems, they can establish file systems, mount them and do things like that. They see the same users, they see everything because they are part of the host and they have full integration. On the other extreme there's VMs or KVM, right? They tend to live in the entirely own world, right? Like they could be living on a different host after all and the way how they communicate with the rest of the stuff that runs on the local system is actually across the network, right? Like so you get a maximum isolation. Docker style microservices are probably somewhere in the middle, right? They, it's not entirely clear of their more like system services or more like VMs. I put them on my little range here in the middle. Four containers are Alexi, generally try to be something very similar to KVM, right? Like they expose the system that runs an inner system inside and that you can SSH into, as it's H into is probably something you wouldn't do into Docker containers, yeah. And now the concept that I'm going to introduce is portable system services which I put close to classic system services but somewhere on the range that goes towards Docker, right? So just to position this on a, on a, on a one dimensional access of integration versus isolation. Yeah, think about what's actually shared and not shared with these forms of virtualization, right? Like classic system services obviously have shared networking, right? Like they don't configure that. Like if I install engine X as a system five servers or as a system, you serve as on my system, you don't configure networking explicitly for it. You just use the host networking. While on the other extreme, of course, it's, it's, it's completely separate, right? Like they have to run their own network management solution inside of the VM for things to work. If you go along the axis, of course, yeah, Docker style microservices generally do not configure on the networking but they also don't share it with the host. So you have a relatively like, yeah, but it really depends on how you configure things. Alexi generally also it's like a little bit vague, but yeah. Think about file systems. Generally, Docker style microservices do not share the file system, right? They have a different route, right? So they are relatively separate in this regard. Yeah, classic system service, of course, are fully integrated. They see the same files and directories as everybody else. VMs live in a completely different world, right? Like they have their own block devices even and mount these block devices and see something that the host cannot even see, right? Think about PID namespaces, right? PID1 in classic system services is the same as the host, right? Like because it doesn't live in a new PID concept, right? Like in PID55 that the, that the classic system service sees is actually the same PID55 that, that's actually running on the host, right? In Docker style microservices and everything to the right, that's not the case, right? Like they generally live in their own little world where they have PID namespaces and yeah, PID1 inside of it or PID55 or 77 is going to be something very different from what the container sees and from the host, right? And then VMs, of course, PIDs are separate too, right? So you see, yeah, the PID namespace where the boundaries is somewhere there. The unit system, it's kind of similar, right? Like where is the cut there? If you have classic system service, they of course share the unit system with the host. I mean, that's what started there. If you do Docker style microservices, they, that's kind of weird because they generally don't have an unit system. Like there's no unit system visible from the containers, but they also don't have their own. If you do Alex C, then yes, we have an unit system inside of the container generally. I mean, yeah, people can disagree with me on this that yeah, you can configure differently, but yeah, I'm trying to position this in the general case how people tend to actually use this, right? Device access is also, yeah, if you do classic system services, you generally have raw device access, right? Like you can access the block devices like the, or the sound card or whatever physical devices your, your computer has directly because you're living on a system, right? This is very different, of course, in VMs. VMs are generally isolated completely. Yes, I know that you can do pass through and things like that, but that's kind of, that's magic manual working to make this happen. It's not how this works out of, out of the box, right? And logging on the other end tends to be much more integrated. Like even Docker style microservices tend to log, like provide the logging and it's not done by the payload itself. So, yeah. So the point of this lower thing is that it's actually, even though the access suggests it's a, it's a linear access of integration, it's actually more multi-dimensional, right? Like depending on what you look at, the cut where you get the integration and the isolation is more to the left and more to the right. I hope this makes any sense so far. Okay. Then portable services, one of the goals of portable service we had is leave no artifacts, right? Like in containers, it's kind of a given, but on system services, it's not, right? What do I mean by that actually? If on a, on a Linux system, you install Nginx or MySQL or whatever package you like on your server, right? Down at the depths of RPMs, install them. And then remove them again, right? This leaves artifacts around, major artifacts. For example, on Unix, users are generally, like you cannot sensibly delete system users or any kind of users actually, because if the user's created in some directory, some file, and you remove the user, these files will still be owned by the user ID of that user. So the file ownership is sticky and if the user doesn't exist anymore that the file ownership refers to, then you have a problem and then if the UID gets recycled later on, you have a security issue, right? So this is a problem, I think, with Unix since forever. Most distributions generally do never delete users, right? So that basically means you install Nginx once, you remove it again. Yeah, your UID is gone for good. With portable services, our attempt was to fix that problem and provide a way how user IDs can be used, but they also become inherently transient. We'll talk about that in more detail later on. What this specifically means, but it's not just about system users actually. There should be no artifacts left around, like if you ran a service and you remove it again, like for example temporary files should be gone as well, right? So yeah, it's about binding life cycles. When a system service starts up, it can allocate a couple of resources, when it shuts down, they are released and unlike classic system services, we don't leave stuff around. Another goal of portable services is to have everything in one place. This is kind of the bundling thing, right? Like, yeah, it's not a new concept of course because Teru's existed in Unix since forever. In fact, the whole of portable services can summarize as making Teru's useful. But yeah, it's one of the key goals. I'm a service guy of course because I've wrote most of SystemD. So for me, doing Teru's is awesome, but I want to make them viable to service managers in a very regular system service in a very powerful way and that's what portable services are. Another goal is I want this new concept that adds these couple of portable, of container features to service management, a lot like a native service, right? Because I want it to actually be a native service. Native by this, I mean like a regular system D service, like that has a dot service file or so on. So the ultimate goal after this is that if the portable services are used, then you end up having an inner system that supports three formats for services. The native ones, classic one system D service files, the old system five-init scripts, right, and these new portable services. And as you, like if you're using system D, you already noticed that the behavior of system five-init scripts and regular system D services, the behavior like, do you do system control start and stop on them and can reset resources and see the logs of them? It doesn't really matter, like this distinction is removed. The key with a portable service concept is to do the same here too. Okay, so there's so much about the goal, so much about the positioning of this new concept of portable services. Let's talk about the why, right? Like this is a question you always have to ask. Yeah, I already mentioned that I'm a service management guy, so we don't live in a vacuum, right? Like things happen around us, containers happen of course, right? They have a lot of mind share and they are in a form of form, in a way a form of service management, but we're not removed from the core system. But there are definitely a couple of good ideas that I think make a ton of sense to take and apply to system service management as well, right? It's about, it's not about coming up with anything new, it's just about looking what's good, what's out there and figuring out is this something that we want for service management, like for regular service management as well, and I think this, yeah, the bundling and the sandboxing is, so let's apply it there as well. Also, what's really interesting to notice is at this point in time, pretty much all packages of service that are viable tend to have native system-based service files, right? So in a way, most of the stuff that already exists on the Internet tends to have the service files, and that's actually kind of cool because it allows us to, if we add a little bit on top, we can do something that goes in the direction of containers without actually defining any new kind of metadata because we already have the service files and pretty much everything has the service files at this point already. Yeah, another concept is like, you know, containers in a separate world where you use different tools. Admins generally are used to system service already, so maybe we can just make them more powerful in some regards because, yeah, some of the features that you want to encounter from containers, you can just make a revival for regular system services as well. One primary use case for portable services, I mean, just to make this clear, this is not an attempt to reinvent containers or something like this. I explicitly want to position it as something that is more low level than this, and I explicitly want to position it for use cases where containers might not be the most appropriate way to do things. Like, if you want to use containers or something and containers are the right choice for you, what you do, continue doing this, this is supposed to be a little bit more low level. So one primary use case is what people have dubbed super privileged containers so far. It's, for example, storage, people like to do this, right? Like, they want to ship a lot of complex stack in one image onto your server machine, right? That's why they want to use containers, but on the other hand, they need a really strong integration into the whole system because they need to do device management, like block device management. They need to figure out what's being plugged in, what I SCSI does, and whatever else, right? So they earn this weird position that they would like to ship a lot of complex stack software with its own dependencies because it's all far from trivial onto existing machines, but they also want the full integration into the host because they do device management and if you do device management, that's where you really, really need it. So yeah, inside of Red Hat, there was this people working on super privileged containers when I saw that I said, this is horrible, like, because they ended up using Docker initially and then they turned off all the sandboxing features and created all these bridges that you could escape from inside of the Docker to the host so that they could do the manipulations of the device and everything is like, but yeah, portable servers are supposed to cover that use case perfectly, right? So that you can have your bundling and all these kind of things where you can pick exactly how much you want to see from the system or how much you want to be isolated from the system without being completely and terribly ugly. Yeah, also one of the key ideas is integration is a good thing often, not the bad thing, right? Like with containers and all these things, they're very strong about isolation. I think in many use cases, you want the integration, not in all, but in many you do, right? So yeah, it really depends on your use case, integration and the ability that you can introspect the rest of the system, and particularly for tracing tools and debugging functionality and metrics and these kind of things. It's a really, really good thing if you do not have to first play games with the sandboxing to escape it. Okay, I kind of mentioned this already. One of the goals is that, yeah, the system five services, the native service and the portable service are supposed to be next to each other and equally well supported with the same interfaces and same behavior, same resource management, same everything. The building blocks this all is built of are actually permitted more, right? Like the portable services code base is actually relatively separate. When we added that to system D, like the portable control command that makes all of this available, it didn't actually require any changes on the system deco itself. It just added a new utility that allows you to interface with the existing sandboxing options and bundling options in a nicer, more integrated way, right? So this is kind of, because it is implemented that way, it basically even would allow you to come up with your completely own service delivery framework, use all these basic concepts and build something completely different, right? Like for example, people have been working on making OCI stuff work and translate them dynamically into native system service and things like that. Yeah, one key idea about, any questions so far by the way? Nobody has interrupted me about all the stuff that, okay, let's talk a little bit about disk images, right? Like when we think resource bundling, we have to think about disk images in some form. Walker, as you know, uses tar balls and then weird layers and AUFS and these kind of things. And yeah, one of the goals of this portable service was no new metadata, right? Like I didn't want to sit down and come up with a new OCI spec. I have no interest in whatsoever in that. I didn't want to define a new image form or anything. I have no interest whatsoever in that and it's like a political massive job. So with system reportable services, the key really is we use the metadata we already have. So, in speaking to the disk images, this means we don't care how the bundled images come onto the system as long as the moment you actually want to start them, they are accessible as a Linux accessible file system in some way. We're happy. What does that mean? You can ship things as tar ball if you like and unpack them and then system you can use it as a portable services. But you don't have to. You can also use it as a disk image, right? UFI block device image that you can mount. In a system with the portable service concept, we support both equally. The key, what we just require from people is that they use it, provide images on the block layer, or on the file system unpacked layer. But in both cases, it needs to be something that the Linux kernel natively supports. Is the support native files that the main would just like to score with us? Yes, that's kind of what I'm, the point that I was making. It's like, I don't really care what you use, and we explicitly support both block device level stuff and unpacked table kind of stuff. Right? But not just any file system, you need a block device within the partition table. So the question was supposed to repeat the question, right? Not just, so the question was regarding whether whatever is mounted as a disk image has to be a proper block device, or if it can be just a regular file. The idea is it's just a regular file, right? It can be a block device, but the idea is it's normally a regular file, and the system will internally do the loopback mounting and stuff like that. So you will never actually see that there are loopback devices and block devices involved in the end, but the idea is generally, yeah, either take a table uncompressed and you operate on the file system level, or provide us with maybe a squashFS file and system will set it up as a loopback and mount it, and then it's kind of the same thing from that point on. You can point out a folder. Just say this folder is the file system. So the question was regarding, yeah, if we can point it to a folder, yeah, that's the table option, right, like that I mentioned, right? Like the fact that table is used, I don't give a damn about that, right? Like as long as it's there, in a format that the Linux kernel natively supports, which could be directory, or could be something that I can mount, I'm happy. Are you aware of anybody, I guess, maybe at Red Hat using OS tree with this? So the question was regarding whether I know of anybody using OS tree with this. I heard of people were interested in this, but I didn't follow up in details. But of course, you can totally use OS tree, like because as mentioned, if it's a directory, it's good enough for us. And OS tree stuff after you check it out is just a directory, so all is good. So. Let me come on from that. I was going to ask this later, but I was thinking about how this relates to the stuff that I was doing versus like using some data stuff and other things. And then I was wondering, is this partially you trying to park your tanks on the flaphack and the snaps, like, raw? I can't even take a delivery bit out, which I'm interested in, it's sort of simple, like the same territory, right? So yeah, the question was probably, let's, I guess I can summarize it, the relationship to OS tree and snaps, right? Yeah, to flatpacks and snap. So, I mean, the key really here is this stuff is system level stuff, right? So, flatpacks not system level stuff. Snap is though, right? This is actually in the design very close to snap, what it does. But I mean, I don't really care about the disc images, as mentioned, right? You can actually use even a snap disc image if you want. The focus that I have really is, is after it's there, how to make the stuff that's in that image, available as a regular system service, right? And this is what they generally don't care about or want to provide. So yeah, flatpacks, different story, desktop stuff, this stuff requires privileges. People have asked about making this available unprivileged, but it's kind of difficult because, at least if you do disc images, it's all about mounting and loopback devices and none of that is available unprivileged. I hope that answered the question a little bit. Yeah, key here really is, let's avoid not something new, right? I don't want to be in the business of defining image file formats. So yeah, let's just take simple directory trees or better, stop volume, or maybe a GPT containing squashFS, but actually we don't really care if it's squashFS and we don't really care about if it's GPT either. I just think it's a nice thing to do. The services run directly from these images, right? There has been this root image and root directory service setting since ages in system deep. We just make use of this here to say, yeah, now I run the service from this image and then the moment the service starts, this image is mounted, if it's not mounted yet, or bind mounted, and yeah, this is the moment the service shuts down, but yeah, the image is not used anymore. In a way, this is about like fixing charoot. I mean, charoot has been around since ages and some people have deployed things like that since the 90s, but charoot has a couple of serious problems. Like for example, one of the bigger ones we'll talk about later on is about like, ACPASSWD because you kind of have to synchronize the ACPASSWD on the host with the container. We'll talk about that what we're doing in this area. What's, by the way, interesting because we actually do, so like root image is basically where you specify a squash file system, like any kind of file system that currently mounts. Root directory is where you specify directory, but the root image thing is actually kind of nice because when we mount this stuff, we can actually take benefit of all the weird storage stuff that the Linux kernel has. Two things are particularly interesting. I think like one is you can luxe and kip, for example, an image, and then just make system the start the service from it so you can have a model of basically encrypted services. And variety, I find even more interesting. Variety for those who don't know is a kernel concept about that every access to the disk, or in this case to the image, is verified as the access happens cryptographically against some predefined hash value that can be signed. So with that, you can actually make trusted services in a way where you basically say, yeah, I have this computer here and it will only run software that is signed by me and then you deploy an image on it. And then system will start it, but system will only start it if the top level hash of that image is matches against some signature checking stuff that makes sure that it's only my stuff that runs. Don't wanna talk too much about that because that's probably a talk of its own. Just wanted to mention this simply by the fact that we rely on whatever the Linux kernel provides us with, we can make these things happen. And we did make this happen, it was all hooked up. So, yeah. It doesn't mean that we can use our distribution, for example, and start single service from it. So the question was if this means that we can use another distribution to start services on some host system. And yes, this is what it means. The idea is that much like for containers, the distribution that is used inside of this image doesn't have to match what's on the host and things should still work just fine because the Linux kernel people might not be perfect in maintaining compatibility with everything, but they're pretty good. Okay, yeah. The only thing that an image needs to have to qualify as a portable services is that it has to carry system to unit files, that it has to carry a file, use a LibOS release, and that's it. It has to carry system to unit files because we need to know what to actually start in it. Right, like so it has to have that. As mentioned, it's kind of cool that all software that currently exists generally has these already. And the use a LibOS release file is something that has been existing since five years already. We came up with this originally in the system context, but it's actually adopted universally even beyond, like even the system de-haters now ship this file. It's a very simple file that is supposed to describe the distribution you use, but it's actually extensible. So our idea was, yeah, we use that as a metadata information that if you want to declare what the image that you have there is about you just add a field there. So again, the key here is no new metadata, no new formats. We use Linux file systems. We use system to unit files. We use the OS release file. Nothing of that is new. All of this exists for at least five years and in many cases for 10, 20 years. Any questions about this so far? So in this case, if you don't want to use portable control to start the service, the portable service, you have to declare like a post-system service file to load the portable service, which then in turn carries the system service file. So that was a very complicated question and I would like to delay that to the end. If I hope we have enough time, well, I kind of introduce a little bit to the command line if we have enough time to, because that will I think explain your question. Somebody else had a question? I didn't basically just say the question. Okay, then let's talk about that a little bit later. One quick slide. I mean, there's lots of stuff on this, but I don't want to go too much into detail. The point I want to make here is like system before a long time had all these sandboxing options for system services, right? This is independent like everything else that I said from the actual portable service concept. These things existed since some longer, some shorter, but they're generally options how you can lock down your system services. Just as an example, private devices, for example, it's a Boolean that you can set on a system service. If you turn it on, it basically means that the service gets its own instance of less dev that doesn't contain any real devices, but devU, random dev, now devZero, and these kind of pseudo devices that are Unix API or Linux API, but don't actually reflect to real physical devices you could touch. And a couple of other options of this where they're generally designed to be super easy like Booleans or something very close to being just a Boolean how you can lock down your services. In the portable services concept, we just make use of the fact that this already exists. There's one major difference, though. You can turn this on on your classic system service already, but they generally are opt-in so far, right? Which is something, I mean, we would like to turn that around, of course, but at this point, we can't really for compatibility reasons, like because system services have been around, like I mean, they inherit everything from system five even. And since system five, and in the beginning, system services did not have sandboxing, if we would turn that on by default, now we would break everything, right? So for the classic stuff, it's opt-in. For the portable services, we have the luxury, though, because it's a new concept we just introduced, it's a reverse, it's opt-out, right? Like, so by default, you get a policy, but you can actually opt-out from everything. And if you do, then you get the full integration into the whole system, can do whatever you want. Yeah, these are a couple of that we already have. I did talk in the past, talking about all of them in detail, because they actually do fill like a talk of their own, and they're gonna be more. There's also nowadays, which is really interesting to know, it's like the per-service firewalling these days, where you can set inside of the unit file, you can do access control on IP level and things like that. This is an IP accounting, actually, this is so awesome, but yeah, it kind of fits into this whole sandboxing concept. Yeah, I already mentioned this, sandboxing is opt-out for portable services, rather than opt-in. How much time do I have? This is? I mean, kind of like that. 20 minutes, okay, let's talk a little bit about these hard problems. This is, I kind of already mentioned this. If you do charoots classic on Unix, you have this problem that, yeah, because the charoot environment generally doesn't see the user database of the host, both like the host and the charoot environment might have a different idea what UID 500 means, right, or UID 1000 means. So most of the how-tos that you find on the internet that tell you how to manually set up a charoot, are by, yeah, copy over at C past WD. Yeah, because portable servers are ultimately just a way to make charoots more useful to work, we try to figure out what we can do in this area. For this, we added a concept called dynamic users. Dynamic users is something that is particularly useful in the context of portable services, but you can use it already on your system independently of it, and yeah, it's the one building block, but the building block can be used in any context you like. It just happens to be one of the building blocks portable services are built on. What are dynamic users? Dynamic users is a concept where you basically can say for service that when the service starts, a system user is registered, and when the service shuts down, it's released again, right? I've already mentioned this problem. Now, you have this problem with the file stickiness, right? So that, yeah, when the service then goes down and the user ID is released, what happens to the files that the service created while it was running? Our solution to this is a couple of things. First of all, when the service writes something to slash temp or slash var temp, yeah, we, it will be able to do this. What it instead will do, it will get its own fake little slash temp and fake little var temp that actually is backed by the real one, but whatever the service writes into that is automatically removed when the service shuts down, right? This is what I call life cycle binding, right? Like the life cycle of the system is bound to the life cycle of the temporary files. So when the service goes down, the temporary files go with it, right? So that's one facet of it, but it's not particularly useful yet. So the other thing is, yeah, to deal with the sticky file ownership problem, our solution is we simply disallow the service to write anywhere, right? So it's a nice way to avoid the problem with sticky files by simply prohibiting files altogether, right? So this of course limits the usefulness because you then have a service that can write stuff to temp and var temp but can't do anything else. I mean, it's good enough for probably some use cases probably most use cases wanna be able to actually write stuff to disk and like whatever they generate. So our way out of this is in system need, it's actually also independently useful of dynamic use and independent of portable services. There's a concept where you can specify a state directory inside of the service file. If you do that, that basically means that if a directory in var lib gets choned, like the ownership gets changed to the services system user, the instant the service is started, right? So basically the idea here, hence is that system de-manages for you of specific directories, the ownership and these, the service then gets access to for writing, right? This is a little bit ugly, right? Because it basically means that you start up your service for the first time, system de-creates a new directory in var lib for that specific service, changes the ownership of that directory to the system user, it also like a catered for you. Then you run, you write some stuff to it, you shut down again. Now the user gets released, the data shall stick around and it does and then you start again. But now you might have gotten a different user ID. So what system has to do, it has to recursively chone everything. That is horrible, but it's actually not as bad as it sounds because Linux is very much optimized on that. Even if you have a directory tree that's a couple of gigabytes, at least on my machines, it never took more than a couple of seconds to chone recursively through it. If it's something smaller than a couple of gigabytes, it's probably actually not noticeable. Also, system tries really, really hard to assign the same services, the same dynamic use IDs as it can, by hashing them out of the name of the service and things like that. However, given that the name, like the UAD space, is a little bit too short, collisions will happen in that case it has to chone. So in the meantime, the system stopped after running in first time. Do you change it to rule normally? That's a very good question. So the problem is, of course, if the service started up, has its own directory, then our service goes down. Now these files are there, still owned by this user ID that now sees to exist. This is, of course, the problem we always wanted to avoid. So what do we do? We take a lesson out of how containers are managed. Because in containers, they generally say stored somewhere in valid containers or valid docker or something like that. And the way, because they have the very same problem, they also have a concept of users that only exist while the container is running. The way they avoid it is they have a top level directory where all the containers are stored. And this top level directory is not readable by anybody but root, basically. So they avoid this by adding a barrier in the middle so that it doesn't matter if the files that are stored below are owned by a user ID that is otherwise recycled potentially now, simply by just cutting it off in the middle and saying, yeah, and try a subtree. It's not available to you. That's exactly what we do here. Now this is actually harder than it sounds, because we want to make available var lib foo for a service foo. So what do we do? Do we change the ownership of var lib to 7.00? We can't really do that, right? So this is actually tricky in the background. What actually happens there, there's a directory var lib private. And var lib private, that thing is actually 7.00. I hope you guys still follow this. It sounds like without slides. Really nasty to follow. So that one is actually 0.7.00. And then there's a sim link automatically created from var lib foo into var lib private foo to make it invisible to the outside. And from the inside, because the inside shall have access to this directory, even though it's unprivileged, through bind mounts, we make the var lib private hidden, and instead mount it to the dopper. And I hope you kind of followed at all what I was babbling here. It took us a while to figure out that this is actually workable and is nice. And I'd really like to get rid of much of this code. And maybe one day we can if we have shift fs in the kernel, like the file system where you can actually change the use eddies so far we can't. I don't know. I try to come up with anything better and talk to a lot of people we couldn't. The general runtime behavior of this, even though we do the recursive churning, is kind of nice. It's not a costly operation because Inode updates, because we never actually write through files, we just change the Inode ownership, are surprisingly fast on Linux and on the file systems these days. There was somebody had a? What if you need to send it to a like, say, the same directory? That's a very good question. So the question was, what happens if you have two services that want access to the same directory? The thing is, if they have two dynamic users attached to it, that's not going to work. The Unix doesn't allow that. It would allow that if we have shift of s, which we don't. I hope that we eventually can fix it and make this happen. As long as that is not available, though, what you can do is you can, like, system D actually, when it creates these dynamic users, honors the user name you specify inside of the unit file. Now, if you have two unit files, both of them have dynamic user turned on, and both of them specify the same user name, system D will actually create one of the same user for this. And if you do this, then you can actually share this directory, right? But it's really on you. It certainly won't help you right now with this, because I kind of still hope that shift of s is a real thing eventually. Actually, what are the contain them? Many conferences before here where one of the guys actually talked about this, and then he said that's going to happen. But yeah, we'll see when that's going to happen. But yeah, you can do it, but you have to be careful to use the same user names for the services that want to access. Sorry? But you're not done something with groups instead. So the question is, what about couldn't you use something with groups instead? But I mean, the general problem is that the user IDs, yeah, well, I mean, the problem is that it's up to the programs then to make this happen, because they can't, for example, create files that have non-ridable things like that. So in my assumption, we thought about this, but using SCAles in particular to make something like this happen. But the problem I always solve with that is these solutions tend to end up being something that applications need to explicitly support, because many applications manage the access control manually. While this is a solution we went for, it's transparent to the applications. They don't know that there is, well, I mean, if they look from the outside, they will see the weird thim link. But from the inside, there's no difference from regular, like, yeah. So the question was, how does the file system look like inside of the services namespace, basically? If you provide an image, like, for example, a squash-res image or directory, and then you have a unit file inside of it, and then you're told portable control was the command that I'm going to show you later to attach this thing to the host, which basically means copy the unit file out of it, put it on the host, and update it slightly so that there's a root directory or root image setting that points it back to the image file, if you then start to stuff, what it sees from the inside is exactly what is inside of that image, except for the stuff where you punch the holes into. And the holes that you punch are generally something like state directory, the thing that I already mentioned. If you specify state directory in the unit file, it basically means that directory that you picked there in Vallib is shared between a host and the user. And if dynamic use is turned on, which is an optional feature, then it does the magic UID stuff. And then there are a couple of other settings likes. There's besides state directory, there is cache directory, which does the same in varkache. There is configuration directory, which you might guess does the same thing in Etsy. There's runtime directory, which does the same thing and you might guess slash run. There's one more. Log is directory. Log is directory, too. So we get the same thing in Vallog. But the model really is towards pushing people to be more declarative in their services by denoting exactly which of the directories are actually relative to the service. And then this actually doubles as the way how you can punch holes into this sandbox in a safe way because we do the dynamic user re-choning if necessary. The question is, if we support arbitrary bind mount from the host, we do. But if you do that, then of course you can't use the dynamic user ID thing so easily because we would have to chone magic. But if it's arbitrary stuff, we can't insert the Vallib private thing in between so everything explodes. But by all means, go and do that. Like for example, the storage people, they would run their stuff as root anyway, not as a dynamic user. And for them, yes, use as many explicit bind mounts as you want and make it viable, whatever you want to have for viable. But. AUDIENCE member, like distribution ship of tiny portable images and using stuff from the host? I mean, yeah. So the question is regarding if distributions could ship tiny portable images and use stuff from the host. Yes, they can. I'm not going to prohibit that from you. This is a completely generic tool. You can misuse it. Anyway, like I'm not going to rule into that. Would I do that? I'm not sure. Like I mean, part of the idea about portable servers, of course, is that you, much like for containers, that you distance yourself from the ABI of the host to make the stuff more portable. I'm sorry? You probably just use the existing sandbox and stuff directly, not the course. Yeah. So 10 minutes? OK. Like a couple of more slides, it's completely fine that I didn't cover this, because I got so many good questions. And even more here that's. Is it possible to start multiple instances of a service, of multiple services like this, and with normal systems of it? So the question was regarding whether it's possible to start multiple instances of a portable service, like it is possible with regular services. The answer to that is a clear and resounding yes, because these things aren't native services. So if you put a template file, like a template unit, inside of a portable services, then you can instantiate it as many times as you want, just like you could do it with a template unit that is installed on the host. The question was regarding whether if you do the multiple instances of the same service, whether they can share the same user ID, yes, absolutely. But they don't even have to be multiple instances of the same service in that case. As mentioned earlier, if you have two otherwise unrelated services that are not instances of the same one, you can do the same thing. What's key is that you turn on dynamic users in those cases, and that you set user equals the same string, which is the name. It's also possible. If you want to have different user IDs for every instance, yes, you can do this too, which is actually really really awesome for the dynamic user stuff, because you can actually implement, trivially now, a service that is socket activated. And for age incoming connection, a new instance is created, and each instance gets its own dynamic user that lists as long as this connection exists, and then goes away again. To me, this is actually just this concept of making user ID something cheap that you can have, and then you can return them, and they become this extremely expensive thing that stick around forever, and hence your package can only allocate like one or two, or maybe three, but never like 100 of these. Yeah, this is actually one of the, like, I see it as a breathing new life into the UNIX concept of user IDs, because suddenly you can use them for much more than you traditionally could. And that is so awesome, actually, because user IDs are like the core security feature of UNIX, after all. All the other stuff that we have these days was as a Linux and up armor, and whatever else that was second, came later in it specifically supported in only one concept. But the user IDs, as a security concept, have existed always, and are built into every piece of our software, after all. So adding the dynamic system user concept to that is kind of like, yeah, it turns something really established into a much more powerful concept. OK, so I'm going to talk about this slide here as my last slide then, just to give you a little bit of feeling. I don't have demo, because demos tend to go wrong. But I want to at least show you how the concept generally works. Yeah, the command you use for interfacing this portable service is called portable control. You invoke it, portable control, with the verb attach, for example. Fubar.raw, in this case, is a disk image, like could be, for example, a squashFS image in there. When you do this, when you invoke this, then this portable service image is attached to the host. What does it mean? It means that, yeah, portable control will do a little bit of a verification that it's the same image. And then it will just copy out a couple of unit files from this image. Which unit files will it copy? Actually, the ones that start with the same name as the image file itself. So if the image file is called fubar.raw, the unit files that copies out are fubar dash whatever you like, as well as fubar dot whatever you like. The unit files that are copied out don't have to be service unit files, by the way. They can be socket unit files as well. They can be target unit files. They can be whatever you else. Like not all of them, not all eight of them that we have in system D, but most of them. So you can, what this basically means is that you can package a couple of related units into one image. You just have to follow a little naming regime that you always call them some prefix dash, some suffix, and the prefix is always the same. You can use socket activation, time activation, all of this at the same time. And just by portable control attach, all of them become available in the system. And from that point on, they are regular services. So at that point, you can do system control start, system control stop, system control set property, system control whatever else you kill, whatever. You can do general control dash, you run with them, because after they're copied out like that, they are regular system services. There's nothing distinguishing them anymore, except for the fact that they originally got copied out of some image file. And then of course there's. Also across reboots? Yes, portable control attach. If you call it like this, it's actually across reboots. So the files are copied out into Etsy. There's actually portable control attach, dash, run time as well, and you might guess it copies them into slash run. So the attachment, the fact that the unit files exist on the host goes away when you reboot. There's obviously the other work that undoes all of this. It just removes the files that were copied out. And then at that point, the services are not available on the host. Key again is leave no traces. The idea really is that besides logs, like because we never should delete logs, after you do the detail, nothing remains in the system. Can you write to the template that the unit files still have? Can you use kind of templates? So the question is whether these can be template unit files? Yes, they can. The files that are copied out just have to have foobar as a prefix followed by either a dot or a dash. But what comes after that, it doesn't really matter. It can be a template. It can be an instance, even. It's very multiple kind of services with different things. You can template. You can use template files to do that. And then run multiple services on that. And by the way, by default, the idea is that you enable them all at the same time as attaching, but you don't have to. You can attach them and then enable them or you don't. It's completely up to you. Yeah? Last question. I assume that the drop in replacement is focused on. Sorry? I assume that the drop in replacement is like input. Or it's like in the such, more problems. Oh, you mean drop in. OK, the question was regarding the drop in files we support for unit files so that you can extend them on a host. Yes, these, because after they're copied out, they are regular unit files. You can also extend the contained unit files on the host by dropping stuff onto the host. And at sea and run, because they are native unit files at that point, they're not distinct anymore. You can do everything you can do. You can even do system control edit if you like. And then system control will drop in the unit file for you. If you do detach, however, these ones would not be removed. Because yeah, I mean we could probably add that. But it's probably needs an extra switch because admins might be pissed if we remove the configuration that the chain is meant to have. This was my last question. So thank you very much, everybody. If you have further questions, I'm going to be outside. Thank you. Thank you.
 Hi, I'm Lena Padaling, I work on systemd. I'm going to talk about portable services  already to use. Portable services is a new concept in systemd. I'd like to introduce  you to this. I did this talk before by the way. You might have seen it if you attended  DefConf or if you said I've seen it all systems ago. There's not going to be much new in that  if you saw it there and probably would be cool if there are still people outside. If  you have seen it, you wouldn't miss anything. But apparently nobody has seen it yet. If  you have any questions, completely, totally interrupt me right away. I much prefer my  talks to be like conversations instead of just a Q&A at the end. So do not hesitate,  interrupt me. I love that. Yeah, let's jump right in. Portable services. What are portable  services? Portable services you can see two ways. One way to see it is they're system  services with some container features. You can also see the other way around. They're  kind of like containers but with some system service features. What does that mean? First  of all, we have to understand what containers are. I don't really know what containers are.  People have different ideas about containers, what they precisely consist of. I think the  definition that I tend to agree on is that they combine two, like three concepts. One  is resource bundles, right? That's what you do with containers. You pack them all up in  a couple of towels, like shed libraries and all the dependencies. There's isolation, meaning  that they generally run in some form of sandbox. Like, might be a better one, might be a worse  one, but there's at least some form of isolation generally. And delivery, right? Like, you're  delivering on the server you want to deploy them on and then you can run them there. So  these three concepts is what at least I find most interesting about containers. I'm pretty  sure that other people who care about containers probably would list more of that. Portable  services try to take some of these features and add them to classic service management.  Specifically, Portable services about making resource bundles available for regular system  services. There are about integration, right? Like, if you look at the previous slide I  had isolation there, I put integration here, right? Like, because that's what system services  are really in comparison to containers. They're generally much more tightly integrated to the  host system. And sandboxing, right? I do not put delivery on this slide because I don't  really care about delivery. But yeah. So the difference really here is I don't care about  delivery and instead of isolation I put integration and sandboxing here. Sandboxing for me is  slightly different from isolation because like the way I see isolation, it's really about  creating a new world that is separate from the host you live in. While sandboxing I kind  of more see is like you're living still in the same world but you can't do everything  you want to do. Another key aspect of Portable services that's supposed to be highly modular  which like to show the opposition to classic container management is that like in container  management you generally tend to have to buy into the whole idea, right? And then some  people don't do that and for example end up with containers which they introduce the concept  of super privileged containers which are basically containers where you just throw one-half  of the concept away by turning off all the isolation features. But yeah, Portable services  are not supposed to be like that. Portable services are supposed to be modular. You pick  exactly what you want. Do you want the resource bundling? And if you want to have sandboxing  you pick exactly how much isolation or integration you want. So it's really supposed to be modular  and very, very fine grained and doesn't require you to buy into the whole idea but into parts  only. Another way to look at it, consider range from integrated to isolated. If I was  a good graphics artist I would actually have drawn a graph here but I'm lazy, very lazy  so I just put a couple of words here with an arrow in the middle. Think about a range  from integrated to isolated, right? On one side you have the classic system services,  right? Like these are system five services or system D services. They tend to be very  well integrated into the host, right? They live in the same world, they see the same network  interface, they name file systems, they can establish file systems, mount them and do  things like that. They see the same users, they see everything because they are part  of the host and they have full integration. On the other extreme there's VMs or KVM,  right? They tend to live in the entirely own world, right? Like they could be living on  a different host after all and the way how they communicate with the rest of the stuff  that runs on the local system is actually across the network, right? Like so you get  a maximum isolation. Docker style microservices are probably somewhere in the middle, right?  They, it's not entirely clear of their more like system services or more like VMs. I  put them on my little range here in the middle. Four containers are Alexi, generally try to  be something very similar to KVM, right? Like they expose the system that runs an inner system  inside and that you can SSH into, as it's H into is probably something you wouldn't  do into Docker containers, yeah. And now the concept that I'm going to introduce is portable  system services which I put close to classic system services but somewhere on the range  that goes towards Docker, right? So just to position this on a, on a, on a one dimensional  access of integration versus isolation. Yeah, think about what's actually shared and not  shared with these forms of virtualization, right? Like classic system services obviously  have shared networking, right? Like they don't configure that. Like if I install engine X  as a system five servers or as a system, you serve as on my system, you don't configure  networking explicitly for it. You just use the host networking. While on the other extreme,  of course, it's, it's, it's completely separate, right? Like they have to run their own network  management solution inside of the VM for things to work. If you go along the axis, of course,  yeah, Docker style microservices generally do not configure on the networking but they  also don't share it with the host. So you have a relatively like, yeah, but it really  depends on how you configure things. Alexi generally also it's like a little bit vague,  but yeah. Think about file systems. Generally, Docker style microservices do not share the  file system, right? They have a different route, right? So they are relatively separate  in this regard. Yeah, classic system service, of course, are fully integrated. They see  the same files and directories as everybody else. VMs live in a completely different world,  right? Like they have their own block devices even and mount these block devices and see  something that the host cannot even see, right? Think about PID namespaces, right? PID1  in classic system services is the same as the host, right? Like because it doesn't live  in a new PID concept, right? Like in PID55 that the, that the classic system service  sees is actually the same PID55 that, that's actually running on the host, right? In Docker  style microservices and everything to the right, that's not the case, right? Like they  generally live in their own little world where they have PID namespaces and yeah, PID1 inside  of it or PID55 or 77 is going to be something very different from what the container sees  and from the host, right? And then VMs, of course, PIDs are separate too, right? So you  see, yeah, the PID namespace where the boundaries is somewhere there. The unit system, it's  kind of similar, right? Like where is the cut there? If you have classic system service,  they of course share the unit system with the host. I mean, that's what started there.  If you do Docker style microservices, they, that's kind of weird because they generally  don't have an unit system. Like there's no unit system visible from the containers,  but they also don't have their own. If you do Alex C, then yes, we have an unit system  inside of the container generally. I mean, yeah, people can disagree with me on this  that yeah, you can configure differently, but yeah, I'm trying to position this in the  general case how people tend to actually use this, right? Device access is also, yeah,  if you do classic system services, you generally have raw device access, right? Like you can  access the block devices like the, or the sound card or whatever physical devices your,  your computer has directly because you're living on a system, right? This is very different,  of course, in VMs. VMs are generally isolated completely. Yes, I know that you can do pass  through and things like that, but that's kind of, that's magic manual working to make this  happen. It's not how this works out of, out of the box, right? And logging on the other  end tends to be much more integrated. Like even Docker style microservices tend to log,  like provide the logging and it's not done by the payload itself. So, yeah. So the point  of this lower thing is that it's actually, even though the access suggests it's a, it's  a linear access of integration, it's actually more multi-dimensional, right? Like depending  on what you look at, the cut where you get the integration and the isolation is more  to the left and more to the right. I hope this makes any sense so far. Okay. Then portable  services, one of the goals of portable service we had is leave no artifacts, right? Like in  containers, it's kind of a given, but on system services, it's not, right? What do I mean by  that actually? If on a, on a Linux system, you install Nginx or MySQL or whatever package  you like on your server, right? Down at the depths of RPMs, install them. And then remove  them again, right? This leaves artifacts around, major artifacts. For example, on Unix, users  are generally, like you cannot sensibly delete system users or any kind of users actually,  because if the user's created in some directory, some file, and you remove the user, these  files will still be owned by the user ID of that user. So the file ownership is sticky  and if the user doesn't exist anymore that the file ownership refers to, then you have  a problem and then if the UID gets recycled later on, you have a security issue, right?  So this is a problem, I think, with Unix since forever. Most distributions generally do never  delete users, right? So that basically means you install Nginx once, you remove it again.  Yeah, your UID is gone for good. With portable services, our attempt was to fix that problem  and provide a way how user IDs can be used, but they also become inherently transient.  We'll talk about that in more detail later on. What this specifically means, but it's  not just about system users actually. There should be no artifacts left around, like if  you ran a service and you remove it again, like for example temporary files should be  gone as well, right? So yeah, it's about binding life cycles. When a system service  starts up, it can allocate a couple of resources, when it shuts down, they are released and  unlike classic system services, we don't leave stuff around. Another goal of portable services  is to have everything in one place. This is kind of the bundling thing, right? Like, yeah,  it's not a new concept of course because Teru's existed in Unix since forever. In fact, the  whole of portable services can summarize as making Teru's useful. But yeah, it's one  of the key goals. I'm a service guy of course because I've wrote most of SystemD. So for  me, doing Teru's is awesome, but I want to make them viable to service managers in a  very regular system service in a very powerful way and that's what portable services are.  Another goal is I want this new concept that adds these couple of portable, of container  features to service management, a lot like a native service, right? Because I want it  to actually be a native service. Native by this, I mean like a regular system D service,  like that has a dot service file or so on. So the ultimate goal after this is that if  the portable services are used, then you end up having an inner system that supports three  formats for services. The native ones, classic one system D service files, the old system  five-init scripts, right, and these new portable services. And as you, like if you're using  system D, you already noticed that the behavior of system five-init scripts and regular system  D services, the behavior like, do you do system control start and stop on them and can reset  resources and see the logs of them? It doesn't really matter, like this distinction is removed.  The key with a portable service concept is to do the same here too. Okay, so there's so  much about the goal, so much about the positioning of this new concept of portable services.  Let's talk about the why, right? Like this is a question you always have to ask. Yeah,  I already mentioned that I'm a service management guy, so we don't live in a vacuum, right? Like  things happen around us, containers happen of course, right? They have a lot of mind  share and they are in a form of form, in a way a form of service management, but we're  not removed from the core system. But there are definitely a couple of good ideas that  I think make a ton of sense to take and apply to system service management as well, right?  It's about, it's not about coming up with anything new, it's just about looking what's  good, what's out there and figuring out is this something that we want for service management,  like for regular service management as well, and I think this, yeah, the bundling and the  sandboxing is, so let's apply it there as well. Also, what's really interesting to notice  is at this point in time, pretty much all packages of service that are viable tend to  have native system-based service files, right? So in a way, most of the stuff that already  exists on the Internet tends to have the service files, and that's actually kind of cool because  it allows us to, if we add a little bit on top, we can do something that goes in the  direction of containers without actually defining any new kind of metadata because we already  have the service files and pretty much everything has the service files at this point already.  Yeah, another concept is like, you know, containers in a separate world where you use different  tools. Admins generally are used to system service already, so maybe we can just make  them more powerful in some regards because, yeah, some of the features that you want to  encounter from containers, you can just make a revival for regular system services as well.  One primary use case for portable services, I mean, just to make this clear, this is not  an attempt to reinvent containers or something like this. I explicitly want to position it  as something that is more low level than this, and I explicitly want to position it for use  cases where containers might not be the most appropriate way to do things. Like, if you  want to use containers or something and containers are the right choice for you, what you do,  continue doing this, this is supposed to be a little bit more low level.  So one primary use case is what people have dubbed super privileged containers so far.  It's, for example, storage, people like to do this, right? Like, they want to ship a  lot of complex stack in one image onto your server machine, right? That's why they want  to use containers, but on the other hand, they need a really strong integration into  the whole system because they need to do device management, like block device management.  They need to figure out what's being plugged in, what I SCSI does, and whatever else, right?  So they earn this weird position that they would like to ship a lot of complex stack  software with its own dependencies because it's all far from trivial onto existing machines,  but they also want the full integration into the host because they do device management  and if you do device management, that's where you really, really need it.  So yeah, inside of Red Hat, there was this people working on super privileged containers  when I saw that I said, this is horrible, like, because they ended up using Docker initially  and then they turned off all the sandboxing features and created all these bridges that  you could escape from inside of the Docker to the host so that they could do the manipulations  of the device and everything is like, but yeah, portable servers are supposed to cover  that use case perfectly, right?  So that you can have your bundling and all these kind of things where you can pick exactly  how much you want to see from the system or how much you want to be isolated from the  system without being completely and terribly ugly.  Yeah, also one of the key ideas is integration is a good thing often, not the bad thing,  right?  Like with containers and all these things, they're very strong about isolation.  I think in many use cases, you want the integration, not in all, but in many you do, right?  So yeah, it really depends on your use case, integration and the ability that you can introspect  the rest of the system, and particularly for tracing tools and debugging functionality  and metrics and these kind of things.  It's a really, really good thing if you do not have to first play games with the sandboxing  to escape it.  Okay, I kind of mentioned this already.  One of the goals is that, yeah, the system five services, the native service and the  portable service are supposed to be next to each other and equally well supported with  the same interfaces and same behavior, same resource management, same everything.  The building blocks this all is built of are actually permitted more, right?  Like the portable services code base is actually relatively separate.  When we added that to system D, like the portable control command that makes all of this available,  it didn't actually require any changes on the system deco itself.  It just added a new utility that allows you to interface with the existing sandboxing  options and bundling options in a nicer, more integrated way, right?  So this is kind of, because it is implemented that way, it basically even would allow you  to come up with your completely own service delivery framework, use all these basic concepts  and build something completely different, right?  Like for example, people have been working on making OCI stuff work and translate them  dynamically into native system service and things like that.  Yeah, one key idea about, any questions so far by the way?  Nobody has interrupted me about all the stuff that, okay, let's talk a little bit about  disk images, right?  Like when we think resource bundling, we have to think about disk images in some form.  Walker, as you know, uses tar balls and then weird layers and AUFS and these kind of things.  And yeah, one of the goals of this portable service was no new metadata, right?  Like I didn't want to sit down and come up with a new OCI spec.  I have no interest in whatsoever in that.  I didn't want to define a new image form or anything.  I have no interest whatsoever in that and it's like a political massive job.  So with system reportable services, the key really is we use the metadata we already  have.  So, in speaking to the disk images, this means we don't care how the bundled images come  onto the system as long as the moment you actually want to start them, they are accessible  as a Linux accessible file system in some way.  We're happy.  What does that mean?  You can ship things as tar ball if you like and unpack them and then system you can use  it as a portable services.  But you don't have to.  You can also use it as a disk image, right?  UFI block device image that you can mount.  In a system with the portable service concept,  we support both equally.  The key, what we just require from people is that  they use it, provide images on the block layer,  or on the file system unpacked layer.  But in both cases, it needs to be  something that the Linux kernel natively supports.  Is the support native files that  the main would just like to score with us?  Yes, that's kind of what I'm,  the point that I was making.  It's like, I don't really care what you use,  and we explicitly support both block device level stuff  and unpacked table kind of stuff.  Right?  But not just any file system,  you need a block device within the partition table.  So the question was supposed to repeat the question, right?  Not just, so the question was regarding  whether whatever is mounted as a disk image  has to be a proper block device,  or if it can be just a regular file.  The idea is it's just a regular file, right?  It can be a block device, but the idea is  it's normally a regular file,  and the system will internally do the loopback mounting  and stuff like that.  So you will never actually see that  there are loopback devices and block devices involved  in the end, but the idea is generally,  yeah, either take a table uncompressed  and you operate on the file system level,  or provide us with maybe a squashFS file  and system will set it up as a loopback and mount it,  and then it's kind of the same thing from that point on.  You can point out a folder.  Just say this folder is the file system.  So the question was regarding, yeah,  if we can point it to a folder, yeah,  that's the table option, right, like that I mentioned, right?  Like the fact that table is used,  I don't give a damn about that, right?  Like as long as it's there,  in a format that the Linux kernel natively supports,  which could be directory,  or could be something that I can mount, I'm happy.  Are you aware of anybody, I guess,  maybe at Red Hat using OS tree with this?  So the question was regarding whether I know of anybody  using OS tree with this.  I heard of people were interested in this,  but I didn't follow up in details.  But of course, you can totally use OS tree,  like because as mentioned,  if it's a directory, it's good enough for us.  And OS tree stuff after you check it out  is just a directory, so all is good.  So.  Let me come on from that.  I was going to ask this later, but I was thinking about  how this relates to the stuff that I was doing  versus like using some data stuff and other things.  And then I was wondering,  is this partially you trying to park your tanks  on the flaphack and the snaps, like, raw?  I can't even take a delivery bit out,  which I'm interested in, it's sort of simple,  like the same territory, right?  So yeah, the question was probably,  let's, I guess I can summarize it,  the relationship to OS tree and snaps, right?  Yeah, to flatpacks and snap.  So, I mean, the key really here is this stuff  is system level stuff, right?  So, flatpacks not system level stuff.  Snap is though, right?  This is actually in the design very close to snap,  what it does.  But I mean, I don't really care about the disc images,  as mentioned, right?  You can actually use even a snap disc image if you want.  The focus that I have really is,  is after it's there, how to make the stuff  that's in that image,  available as a regular system service, right?  And this is what they generally don't care about  or want to provide.  So yeah, flatpacks, different story, desktop stuff,  this stuff requires privileges.  People have asked about making this available unprivileged,  but it's kind of difficult because,  at least if you do disc images,  it's all about mounting and loopback devices  and none of that is available unprivileged.  I hope that answered the question a little bit.  Yeah, key here really is,  let's avoid not something new, right?  I don't want to be in the business of defining image file  formats.  So yeah, let's just take simple directory trees  or better, stop volume,  or maybe a GPT containing squashFS,  but actually we don't really care if it's squashFS  and we don't really care about if it's GPT either.  I just think it's a nice thing to do.  The services run directly from these images, right?  There has been this root image and root directory service  setting since ages in system deep.  We just make use of this here to say,  yeah, now I run the service from this image  and then the moment the service starts,  this image is mounted, if it's not mounted yet,  or bind mounted,  and yeah, this is the moment the service shuts down,  but yeah, the image is not used anymore.  In a way, this is about like fixing charoot.  I mean, charoot has been around since ages  and some people have deployed things like that  since the 90s,  but charoot has a couple of serious problems.  Like for example,  one of the bigger ones we'll talk about later on  is about like,  ACPASSWD because you kind of have to synchronize  the ACPASSWD on the host with the container.  We'll talk about that what we're doing in this area.  What's, by the way, interesting because we actually do,  so like root image is basically where you specify  a squash file system,  like any kind of file system that currently mounts.  Root directory is where you specify directory,  but the root image thing is actually kind of nice  because when we mount this stuff,  we can actually take benefit of all the weird storage stuff  that the Linux kernel has.  Two things are particularly interesting.  I think like one is you can luxe and kip,  for example, an image,  and then just make system the start the service from it  so you can have a model of basically encrypted services.  And variety, I find even more interesting.  Variety for those who don't know is a kernel concept  about that every access to the disk,  or in this case to the image,  is verified as the access happens  cryptographically against some predefined hash value  that can be signed.  So with that,  you can actually make trusted services in a way  where you basically say,  yeah, I have this computer here  and it will only run software that is signed by me  and then you deploy an image on it.  And then system will start it,  but system will only start it  if the top level hash of that image  is matches against some signature checking stuff  that makes sure that it's only my stuff that runs.  Don't wanna talk too much about that  because that's probably a talk of its own.  Just wanted to mention this simply by the fact  that we rely on whatever the Linux kernel provides us with,  we can make these things happen.  And we did make this happen, it was all hooked up.  So, yeah.  It doesn't mean that we can use our distribution,  for example, and start single service from it.  So the question was if this means  that we can use another distribution to start services  on some host system.  And yes, this is what it means.  The idea is that much like for containers,  the distribution that is used inside of this image  doesn't have to match what's on the host  and things should still work just fine  because the Linux kernel people might not be perfect  in maintaining compatibility with everything,  but they're pretty good.  Okay, yeah.  The only thing that an image needs to have  to qualify as a portable services  is that it has to carry system to unit files,  that it has to carry a file, use a LibOS release,  and that's it.  It has to carry system to unit files  because we need to know what to actually start in it.  Right, like so it has to have that.  As mentioned, it's kind of cool that all software  that currently exists generally has these already.  And the use a LibOS release file is something  that has been existing since five years already.  We came up with this originally in the system context,  but it's actually adopted universally even beyond,  like even the system de-haters now ship this file.  It's a very simple file that is supposed  to describe the distribution you use,  but it's actually extensible.  So our idea was, yeah, we use that as a metadata information  that if you want to declare what the image  that you have there is about you just add a field there.  So again, the key here is no new metadata,  no new formats.  We use Linux file systems.  We use system to unit files.  We use the OS release file.  Nothing of that is new.  All of this exists for at least five years  and in many cases for 10, 20 years.  Any questions about this so far?  So in this case, if you don't want to use portable control  to start the service, the portable service,  you have to declare like a post-system service file  to load the portable service,  which then in turn carries the system service file.  So that was a very complicated question  and I would like to delay that to the end.  If I hope we have enough time,  well, I kind of introduce a little bit to the command line  if we have enough time to,  because that will I think explain your question.  Somebody else had a question?  I didn't basically just say the question.  Okay, then let's talk about that a little bit later.  One quick slide.  I mean, there's lots of stuff on this,  but I don't want to go too much into detail.  The point I want to make here is like system before a long  time had all these sandboxing options  for system services, right?  This is independent like everything else that I said  from the actual portable service concept.  These things existed since some longer, some shorter,  but they're generally options  how you can lock down your system services.  Just as an example, private devices, for example,  it's a Boolean that you can set on a system service.  If you turn it on, it basically means that the service  gets its own instance of less dev  that doesn't contain any real devices,  but devU, random dev, now devZero,  and these kind of pseudo devices that are Unix API  or Linux API, but don't actually reflect  to real physical devices you could touch.  And a couple of other options of this  where they're generally designed to be super easy  like Booleans or something very close to being just a Boolean  how you can lock down your services.  In the portable services concept,  we just make use of the fact that this already exists.  There's one major difference, though.  You can turn this on on your classic system service already,  but they generally are opt-in so far, right?  Which is something, I mean,  we would like to turn that around, of course,  but at this point, we can't really  for compatibility reasons,  like because system services have been around,  like I mean, they inherit everything from system five even.  And since system five, and in the beginning,  system services did not have sandboxing,  if we would turn that on by default,  now we would break everything, right?  So for the classic stuff, it's opt-in.  For the portable services, we have the luxury, though,  because it's a new concept we just introduced,  it's a reverse, it's opt-out, right?  Like, so by default, you get a policy,  but you can actually opt-out from everything.  And if you do, then you get the full integration  into the whole system, can do whatever you want.  Yeah, these are a couple of that we already have.  I did talk in the past, talking about all of them in detail,  because they actually do fill like a talk of their own,  and they're gonna be more.  There's also nowadays, which is really interesting to know,  it's like the per-service firewalling these days,  where you can set inside of the unit file,  you can do access control on IP level and things like that.  This is an IP accounting, actually,  this is so awesome, but yeah,  it kind of fits into this whole sandboxing concept.  Yeah, I already mentioned this,  sandboxing is opt-out for portable services,  rather than opt-in.  How much time do I have?  This is?  I mean, kind of like that.  20 minutes, okay, let's talk a little bit  about these hard problems.  This is, I kind of already mentioned this.  If you do charoots classic on Unix,  you have this problem that, yeah,  because the charoot environment generally doesn't see  the user database of the host,  both like the host and the charoot environment  might have a different idea what UID 500 means, right,  or UID 1000 means.  So most of the how-tos that you find on the internet  that tell you how to manually set up a charoot,  are by, yeah, copy over at C past WD.  Yeah, because portable servers are ultimately  just a way to make charoots more useful to work,  we try to figure out what we can do in this area.  For this, we added a concept called dynamic users.  Dynamic users is something that is particularly useful  in the context of portable services,  but you can use it already on your system independently of it,  and yeah, it's the one building block,  but the building block can be used in any context you like.  It just happens to be one of the building blocks  portable services are built on.  What are dynamic users?  Dynamic users is a concept where you basically can say  for service that when the service starts,  a system user is registered,  and when the service shuts down, it's released again, right?  I've already mentioned this problem.  Now, you have this problem with the file stickiness, right?  So that, yeah, when the service then goes down  and the user ID is released,  what happens to the files that the service  created while it was running?  Our solution to this is a couple of things.  First of all, when the service writes something  to slash temp or slash var temp,  yeah, we, it will be able to do this.  What it instead will do,  it will get its own fake little slash temp  and fake little var temp that actually is backed  by the real one, but whatever the service writes into that  is automatically removed when the service shuts down, right?  This is what I call life cycle binding, right?  Like the life cycle of the system  is bound to the life cycle of the temporary files.  So when the service goes down,  the temporary files go with it, right?  So that's one facet of it,  but it's not particularly useful yet.  So the other thing is, yeah,  to deal with the sticky file ownership problem,  our solution is we simply disallow the service  to write anywhere, right?  So it's a nice way to avoid the problem with sticky files  by simply prohibiting files altogether, right?  So this of course limits the usefulness  because you then have a service  that can write stuff to temp and var temp  but can't do anything else.  I mean, it's good enough for probably some use cases  probably most use cases wanna be able to actually  write stuff to disk and like whatever they generate.  So our way out of this is in system need,  it's actually also independently useful  of dynamic use and independent of portable services.  There's a concept where you can specify a state directory  inside of the service file.  If you do that, that basically means  that if a directory in var lib gets choned,  like the ownership gets changed to the services system user,  the instant the service is started, right?  So basically the idea here, hence is that system  de-manages for you of specific directories, the ownership  and these, the service then gets access to for writing, right?  This is a little bit ugly, right?  Because it basically means that you start up your service  for the first time, system de-creates a new directory  in var lib for that specific service,  changes the ownership of that directory  to the system user, it also like a catered for you.  Then you run, you write some stuff to it,  you shut down again.  Now the user gets released, the data shall stick around  and it does and then you start again.  But now you might have gotten a different user ID.  So what system has to do, it has to recursively  chone everything.  That is horrible, but it's actually not as bad as it sounds  because Linux is very much optimized on that.  Even if you have a directory tree that's a couple of gigabytes,  at least on my machines, it never took more than a couple  of seconds to chone recursively through it.  If it's something smaller than a couple of gigabytes,  it's probably actually not noticeable.  Also, system tries really, really hard to assign  the same services, the same dynamic use IDs as it can,  by hashing them out of the name of the service  and things like that.  However, given that the name, like the UAD space,  is a little bit too short, collisions will happen  in that case it has to chone.  So in the meantime, the system stopped  after running in first time.  Do you change it to rule normally?  That's a very good question.  So the problem is, of course, if the service started up,  has its own directory, then our service goes down.  Now these files are there, still owned by this user ID that  now sees to exist.  This is, of course, the problem we always wanted to avoid.  So what do we do?  We take a lesson out of how containers are managed.  Because in containers, they generally  say stored somewhere in valid containers  or valid docker or something like that.  And the way, because they have the very same problem,  they also have a concept of users  that only exist while the container is running.  The way they avoid it is they have a top level directory  where all the containers are stored.  And this top level directory is not  readable by anybody but root, basically.  So they avoid this by adding a barrier in the middle  so that it doesn't matter if the files that are stored  below are owned by a user ID that is otherwise  recycled potentially now, simply by just cutting it off  in the middle and saying, yeah, and try a subtree.  It's not available to you.  That's exactly what we do here.  Now this is actually harder than it sounds,  because we want to make available var lib foo for a service foo.  So what do we do?  Do we change the ownership of var lib to 7.00?  We can't really do that, right?  So this is actually tricky in the background.  What actually happens there, there's a directory var lib  private.  And var lib private, that thing is actually 7.00.  I hope you guys still follow this.  It sounds like without slides.  Really nasty to follow.  So that one is actually 0.7.00.  And then there's a sim link automatically created from var  lib foo into var lib private foo to make it  invisible to the outside.  And from the inside, because the inside shall have access  to this directory, even though it's unprivileged,  through bind mounts, we make the var lib private hidden,  and instead mount it to the dopper.  And I hope you kind of followed at all what I was babbling here.  It took us a while to figure out that this is actually  workable and is nice.  And I'd really like to get rid of much of this code.  And maybe one day we can if we have shift fs in the kernel,  like the file system where you can actually change the use  eddies so far we can't.  I don't know.  I try to come up with anything better and talk to a lot of  people we couldn't.  The general runtime behavior of this, even though we do the  recursive churning, is kind of nice.  It's not a costly operation because Inode updates, because  we never actually write through files, we just change the  Inode ownership, are surprisingly fast on Linux  and on the file systems these days.  There was somebody had a?  What if you need to send it to a  like, say, the same directory?  That's a very good question.  So the question was, what happens if you have two  services that want access to the same directory?  The thing is, if they have two dynamic users attached to  it, that's not going to work.  The Unix doesn't allow that.  It would allow that if we have shift of s, which we don't.  I hope that we eventually can fix it and make this happen.  As long as that is not available, though, what you  can do is you can, like, system D actually, when it creates  these dynamic users, honors the user name you specify  inside of the unit file.  Now, if you have two unit files, both of them have dynamic  user turned on, and both of them specify the same user name,  system D will actually create one of the same user for this.  And if you do this, then you can actually share this  directory, right?  But it's really on you.  It certainly won't help you right now with this, because I  kind of still hope that shift of s is a real thing  eventually.  Actually, what are the contain them?  Many conferences before here where one of the guys actually  talked about this, and then he said that's going to happen.  But yeah, we'll see when that's going to happen.  But yeah, you can do it, but you have to be careful to use  the same user names for the services that want to access.  Sorry?  But you're not done something with groups instead.  So the question is, what about couldn't you use something  with groups instead?  But I mean, the general problem is that the user IDs,  yeah, well, I mean, the problem is that it's up to the  programs then to make this happen, because they can't,  for example, create files that have non-ridable things  like that.  So in my assumption, we thought about this, but using  SCAles in particular to make something like this happen.  But the problem I always solve with that is these  solutions tend to end up being something that applications  need to explicitly support, because many applications  manage the access control manually.  While this is a solution we went for, it's transparent to  the applications.  They don't know that there is, well, I mean, if they look  from the outside, they will see the weird thim link.  But from the inside, there's no difference from  regular, like, yeah.  So the question was, how does the file system look like  inside of the services namespace, basically?  If you provide an image, like, for example, a  squash-res image or directory, and then you have a unit file  inside of it, and then you're told portable control was  the command that I'm going to show you later to attach  this thing to the host, which basically means copy the unit  file out of it, put it on the host, and update it slightly  so that there's a root directory or root image  setting that points it back to the image file, if you then  start to stuff, what it sees from the inside is exactly  what is inside of that image, except for the stuff where you  punch the holes into.  And the holes that you punch are generally something like  state directory, the thing that I already mentioned.  If you specify state directory in the unit file, it  basically means that directory that you picked there in  Vallib is shared between a host and the user.  And if dynamic use is turned on, which is an optional  feature, then it does the magic UID stuff.  And then there are a couple of other settings likes.  There's besides state directory, there is cache  directory, which does the same in varkache.  There is configuration directory, which you might  guess does the same thing in Etsy.  There's runtime directory, which does the same thing and  you might guess slash run.  There's one more.  Log is directory.  Log is directory, too.  So we get the same thing in Vallog.  But the model really is towards pushing people to be  more declarative in their services by denoting exactly  which of the directories are actually relative to the  service.  And then this actually doubles as the way how you can  punch holes into this sandbox in a safe way because we do  the dynamic user re-choning if necessary.  The question is, if we support arbitrary bind  mount from the host, we do.  But if you do that, then of course you can't use the  dynamic user ID thing so easily because we would have to  chone magic.  But if it's arbitrary stuff, we can't insert the Vallib  private thing in between so everything explodes.  But by all means, go and do that.  Like for example, the storage people, they would run  their stuff as root anyway, not as a dynamic user.  And for them, yes, use as many explicit bind  mounts as you want and make it viable, whatever you want to  have for viable.  But.  AUDIENCE member, like distribution ship of tiny  portable images and using stuff from the host?  I mean, yeah.  So the question is regarding if distributions could ship  tiny portable images and use stuff from the host.  Yes, they can.  I'm not going to prohibit that from you.  This is a completely generic tool.  You can misuse it.  Anyway, like I'm not going to rule into that.  Would I do that?  I'm not sure.  Like I mean, part of the idea about portable servers, of  course, is that you, much like for containers, that you  distance yourself from the ABI of the host to make the  stuff more portable.  I'm sorry?  You probably just use the existing sandbox and stuff  directly, not the course.  Yeah.  So 10 minutes?  OK.  Like a couple of more slides, it's completely fine that I  didn't cover this, because I got so many good questions.  And even more here that's.  Is it possible to start multiple instances of a service,  of multiple services like this, and with  normal systems of it?  So the question was regarding whether it's possible to start  multiple instances of a portable service, like it is  possible with regular services.  The answer to that is a clear and resounding yes, because  these things aren't native services.  So if you put a template file, like a template unit, inside  of a portable services, then you can instantiate it as many  times as you want, just like you could do it with a template  unit that is installed on the host.  The question was regarding whether if you do the multiple  instances of the same service, whether they can share the  same user ID, yes, absolutely.  But they don't even have to be multiple instances of the same  service in that case.  As mentioned earlier, if you have two otherwise unrelated  services that are not instances of the same one, you can do  the same thing.  What's key is that you turn on dynamic users in those cases,  and that you set user equals the same string, which is the  name.  It's also possible.  If you want to have different user IDs for every  instance, yes, you can do this too, which is actually really  really awesome for the dynamic user stuff, because you can  actually implement, trivially now, a service that is socket  activated.  And for age incoming connection, a new instance is  created, and each instance gets its own dynamic user that  lists as long as this connection exists, and then  goes away again.  To me, this is actually just this concept of making user ID  something cheap that you can have, and then you can return  them, and they become this extremely expensive thing that  stick around forever, and hence your package can only  allocate like one or two, or maybe three, but never like  100 of these.  Yeah, this is actually one of the, like, I see it as a  breathing new life into the UNIX concept of user IDs,  because suddenly you can use them for much more than you  traditionally could.  And that is so awesome, actually, because user IDs are  like the core security feature of UNIX, after all.  All the other stuff that we have these days was as a  Linux and up armor, and whatever else that was second,  came later in it specifically supported in only one concept.  But the user IDs, as a security concept, have existed  always, and are built into every piece of our software,  after all.  So adding the dynamic system user concept to that is kind of  like, yeah, it turns something really established into a much  more powerful concept.  OK, so I'm going to talk about this slide here as my last  slide then, just to give you a little bit of feeling.  I don't have demo, because demos tend to go wrong.  But I want to at least show you how the concept generally  works.  Yeah, the command you use for interfacing this portable  service is called portable control.  You invoke it, portable control, with the  verb attach, for example.  Fubar.raw, in this case, is a disk image, like could be,  for example, a squashFS image in there.  When you do this, when you invoke this, then this  portable service image is attached to the host.  What does it mean?  It means that, yeah, portable control will do a little bit  of a verification that it's the same image.  And then it will just copy out a couple of unit files from  this image.  Which unit files will it copy?  Actually, the ones that start with the same name as the  image file itself.  So if the image file is called fubar.raw, the unit files  that copies out are fubar dash whatever you like, as well as  fubar dot whatever you like.  The unit files that are copied out don't have to be service  unit files, by the way.  They can be socket unit files as well.  They can be target unit files.  They can be whatever you else.  Like not all of them, not all eight of them that we have in  system D, but most of them.  So you can, what this basically means is that you can  package a couple of related units into one image.  You just have to follow a little naming regime that you  always call them some prefix dash, some suffix, and the  prefix is always the same.  You can use socket activation, time activation, all of this  at the same time.  And just by portable control attach, all of them become  available in the system.  And from that point on, they are regular services.  So at that point, you can do system control start, system  control stop, system control set property, system control  whatever else you kill, whatever.  You can do general control dash, you run with them,  because after they're copied out like that, they are  regular system services.  There's nothing distinguishing them anymore, except for  the fact that they originally got copied out of some image  file.  And then of course there's.  Also across reboots?  Yes, portable control attach.  If you call it like this, it's actually across reboots.  So the files are copied out into Etsy.  There's actually portable control attach, dash, run time  as well, and you might guess it copies them into slash run.  So the attachment, the fact that the unit files exist on  the host goes away when you reboot.  There's obviously the other work that undoes all of this.  It just removes the files that were copied out.  And then at that point, the services are not  available on the host.  Key again is leave no traces.  The idea really is that besides logs, like because we  never should delete logs, after you do the  detail, nothing remains in the system.  Can you write to the template that the unit files  still have?  Can you use kind of templates?  So the question is whether these can be template unit  files?  Yes, they can.  The files that are copied out just have to have foobar as a  prefix followed by either a dot or a dash.  But what comes after that, it doesn't really matter.  It can be a template.  It can be an instance, even.  It's very multiple kind of services with different things.  You can template.  You can use template files to do that.  And then run multiple services on that.  And by the way, by default, the idea is that you enable them  all at the same time as attaching, but you don't have  to.  You can attach them and then enable them or you don't.  It's completely up to you.  Yeah?  Last question.  I assume that the drop in replacement is focused on.  Sorry?  I assume that the drop in replacement is like input.  Or it's like in the such, more problems.  Oh, you mean drop in.  OK, the question was regarding the drop in files we  support for unit files so that you can extend them on a host.  Yes, these, because after they're copied out, they are  regular unit files.  You can also extend the contained unit files on the  host by dropping stuff onto the host.  And at sea and run, because they are native unit files at  that point, they're not distinct anymore.  You can do everything you can do.  You can even do system control edit if you like.  And then system control will drop in the unit file for you.  If you do detach, however, these ones would not be  removed.  Because yeah, I mean we could probably add that.  But it's probably needs an extra switch because admins  might be pissed if we remove the configuration that the  chain is meant to have.  This was my last question.  So thank you very much, everybody.  If you have further questions, I'm going to be outside.  Thank you.  Thank you.
Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__0_131.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__132_1379.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__1380_1499.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__1500_1703.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__1704_1895.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__1896_2015.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__2016_2183.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__2184_3155.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__3156_3227.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__3228_3611.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__3612_3659.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__3660_3947.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__3948_4007.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__4008_4055.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__4056_4691.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__4692_6023.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__6024_6467.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__6468_7079.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__7080_8567.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__8568_13667.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__13668_13751.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__13752_15851.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__15852_16547.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__16548_17519.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__17520_18947.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__18948_18995.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__18996_19103.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__19104_19163.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__19164_20291.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__20292_21227.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__21228_21719.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__21720_24767.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__24768_25667.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__25668_25727.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__25728_26003.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__26004_27371.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__27372_32783.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__32784_34871.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__34872_35327.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__35328_35591.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__35592_35987.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__35988_36635.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__36636_39839.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__39840_41651.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__41652_42527.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__42528_44423.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__44424_45167.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__45168_45419.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__45420_45839.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__45840_45983.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__45984_46247.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__46248_50207.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__50208_59927.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__59928_64811.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__64812_64955.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__64956_65003.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__65004_65843.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__65844_65891.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__65892_65939.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__65940_69071.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__69072_69119.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__69120_69167.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__69168_72275.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__72276_72359.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__72360_72815.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__72816_76235.png Portable_Services_are_Ready_to_Use_Portable_Services_bring_some_container_features_to_classic_servi__76236_76355.png