Mgmt Config:
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Shubin, James
Mgmt is a next gen config management tool that takes a fresh look at existing automation problems. The tool has two main parts: the engine, and the language. The engine has a number of resources, including a container resource. We will feature a number of demos of this resource, and discuss how the language can be used to build a custom container automation solution. The language is a minimalistic, functional, reactive DSL. It can be used to build a real-time, distributed container automation solution, or something else entirely! We'll also use it to demo, service discovery, auto-scaling, and scheduling! Finally we'll talk about some of the future designs we're planning and make it easy for new users to get involved and help shape the project.
 So just for who doesn't know, this is Mr. James and he's going to present MGMT config for containers. So. Thank you so much. Applause. So I'm just going to talk a little bit about containers because it's such a hot topic and a little bit about why we don't treat containers exactly like other resources. And by resources I mean like virtual machines or other things. We're going to talk about that. I'm going to do some videos. Who am I? I'm a hacker. I talk really fast because I have a lot of stuff on our Facebook live demos. So if you really miss something and you really desperately don't understand, raise your hand and I'll try and answer quick questions. But I'm going to try and show you a lot of demos and it's recorded so you can watch it and slow at the end. I write a technical blog called the Technical Blog of James. Who's seen it? Just raise your hand. If you haven't seen it, just raise your hand so I seem really popular. Thank you. Excellent. I work on, I used to be a physiologist. That's what I did by training in and I don't really do much anymore. But if you have cardiology questions I can try and help you out. Big into DevOps. I used to do a lot of puppets. Who's done some puppet before? No? Okay. So you know what I'm talking about a bit? A lot of infrastructure automation. I learned a lot of puppet but I thought there were a lot of things wrong with it. And I realized very recently that I didn't really want to live in this Kubernetes monoculture. Do you want to live in this Kubernetes monoculture? Who said, yeah, get out of this room now. So this is my answer to whether I really want to live in that world. Because it's just silly. It might be useful for some things. So this is my nope guy. And it's just, everything is nope. So instead, long story short, some time back I started, I sat down and I started working on a project called MGMT. It was really intended as a solution to the problems in puppet. But I think there's some overlap with what people are doing with Kubernetes and how they're doing it wrong. So I'm going to talk about that and hopefully convince you. So in this project, open source, free software, whatever, there's two parts. There's the engine and the language. The engine actually does the work on the computer. So it sets up containers, VMs, files, packages, services, et cetera. And there's the language, which is actually what we use to describe everything to the containers. So, oops, sorry about that. So there's three main parts to the engine. It can run in parallel. There's events, which I'll show you in a moment. And the whole thing works as a distributed system. So it's in contrast to something like Kubernetes for example, which is really essential orchestration. And then there's the computer. The graphs that we run, we run graphs. That's our output. And these things are resources. So this might be install a package, start a service, and so on. And these black arrows are dependencies. So something might really, ah, similar to what you might be familiar with puppet and so on. Except we can run everything in parallel. And the events can be anything. They can be files, service, the resources can be anything. They can be files, services, packages, end spawn resources, kind of container thing. Docker containers, if you're into Docker. I'm personally not, but I'll show you a little demo of what I mean by event driven. So let me just actually, so I have to actually turn off. Is this big enough? Can you see? No. No? Bigger? Yes, no? Don't be shy. Is it big enough? Bigger? Okay. So what I'm going to actually do is I'm going to actually just start up MGMT. I'm going to show you just, I'm going to start with a little example where I go, hello, FOSSTEM. So I'm going to run this resource that basically says, please create one container, one file. And if we go into this directory here on the left, you can see, oops, I'm going to remove this mess. I can show you that it's created this file called hello world. And if I remove the file, you can see it comes right back. So I remove it and it comes right back. And this is what I mean by event. So each resource actually is being watched by the engine. And whenever something goes wrong, you can actually just remove it and cat, it's so fast it even comes back before it gets to the end there. And this is not fake. You can see the engine running as this happens. So it sleeps and it wakes up. And it's so fast you can even run it in this watch command, which just loops something over and over again really, really quickly. And you'll see that it just always sort of keeps you in the correct state. And I have actually an example with an N spawn resource. If you'd like to see that. So N spawn 0. So this one actually requires root because N spawn requires root. And this is actually kind of new. So I have to see if this is going to work because there's a bug actually in this resource. But I want to show you it anyways. I think it's actually not working right now. I have one second to kill this. It's N spawn. So if you want to do an N spawn resource, you can do it just like this. Just basically declare the resource. And I have this demo. So I was working on a fancier demo with N spawn. I'm not actually going to show you that right now. So I can actually just show you. I'll show you this one because I was really working to get this ready. Sorry, one second. What's my password? Who knows my root password? Anybody? So I used Fedoro on my laptop. I actually wanted to do a Debian demo. So this is what it is. So it's actually, you just need the name of the machine. Let's see if this works. Okay. There we go. So these are just the declarative resources. And if I go back here, and I run this one here, I think that's going to work. All right. So machine CTL, I think it's list. And there you can see it's running. And the same sort of thing happens. Like if you, I forget if it's terminate, stretch. Oops. If I kill the container, I'll type my password. You see it actually, MGMT wakes up and it brings it right back. Does that make sense? Cool. So I like demo a little bit. I just had to get it ready for you. So this is the whole point. So we have these primitives. They all have events. Now let's do some fancier stuff. So and this works for everything, virtual machines, files, packages, services. And I think this is actually kind of like monitoring. So typically we had to set up infrastructure and then monitoring would be a second step. But if we build this in this event driven nature into each resource, it's already partially done for you. So now we have to describe how you actually tell the computers what to do. So normally like in the Kubernetes land, Ansible land, it's blasting these giant YAML files, which to me makes no sense because it's not expressive. You can only play with the knobs that they've programmed for you. So I wanted to build a language. It's a DSL, so a very small language that lets you actually build your own custom creative things. The language has to be really safe because you don't want to make an off by one error and blow away a data center. It has to be very powerful. So it's a reactive language, kind of like an FRP. If you know what FRPs are, that's OK. You'll understand in a minute. And here's the demo I'm going to show you. So what I do here, I'm just going to actually start off this demo. I'm going to start off this demo on the left. And on the right here, I'm just going to watch this file. There we go. So what's happening? I'm running this code. And here's what it basically looks like. I have this datetime function at the top. And it adds the value from this output to a year, which is some variable that has a whole bunch of integers multiplied together. Then there's this load value down here. And it goes into this variable. And then both of those variables go into this struct, which get put into this big print tap string, which gets set as the content of this file. Right? So what's going on? But here's what's happening when it's running. Oops. Sorry. It's running over here. Here's what's happening when it's running. On the left, you have MGMT running. On the right, I'm just watching this file just so you can see what the output is. And you can see it's actually changing about every second. Because MGMT is actually constantly running, constantly deciding that the value should be different in the file. And it's updating the file every second. So what's actually happening here is this date time function is actually a stream of values. So it's the stream of number of seconds every second that goes by. Right? So the data is always going up. And it's putting all these values. Here's the load on the actual system, which changes constantly. Putting all those values. And it's reevaluating just the parts of the code that it needs to, which ultimately results in this file resource that has different contents. See that? And then that all gets put into this file. And you can see that it actually changes. Does that make sense? And just to make it a little bit more fun, I actually built a function, a custom function called VU Meter. And what it's doing, it's running. It's actually listening to my laptop microphone right now. And so if you make a lot of noise, you see it actually goes up. And just to prove you this is real, we're going to be very quiet. And when I point to you, I want you to make a whole lot of noise and see that it's working. Okay? All right? So nothing I could do. And so this is the kind of thing. We're taking real-time event sources and we're integrating them in this small, simple, safe language that lets you do very powerful things. So the kind of things you could do, for example, is you could take this and maybe you have this in your server room or in your little office. And when there's lots of loud noises and people are screaming and fighting, you automatically set all your systems read-only so that nothing is done in anger, right? So that's the kind of joke I make. So that's the sort of idea. But you want to see some more stuff? Do you want to hear some more stuff? These people leaving are like, nope, I don't want to see some more stuff. All right. So we're going to do some fancy scheduling stuff. So in addition to just these silly examples, we actually, as I said, work as a distributed system and we've built into the core of MGMT some functions and one of the special functions is called schedule. And what it does is it actually takes all the machines that run it and produces a list of machines that are selected by a distributed scheduler. So what it's going to do, it's going to do this and it's going to run in each machine. So I'm going to just run it. It's actually a new demo. So let's see if this works well. So I'm going to run it. I'm going to run it at a time. And oops, schedule. Schedule dash Docker. So I decided to do this with a Docker. So I'm going to run the first one and over here, run a watch. I'm going to run this big fancy watch command. What it does is it's just going to sleep and it's going to tail some files and run Docker PS and sleep and so on. So I'm just going to run that and you actually can see at the moment this is running. Did it work? The Docker thing is quite new. So let's see what's going on. Just kill this for a second. Okay. Docker PS. Docker images. Is it going? Is it starting up or is it unhappy? Oh, I don't have internet. That's why. I need some internet. I need some internet. Sorry, folks. If you're torrenting, please turn off your torrents. It doesn't need a lot of internet because I've pre-cached the images but Docker is sneaky about this stuff. Okay, we have internet. Just kill this. Okay, let's start this up. I need to system. CTL start. Okay, it's working. So on the left I started up the first machine and what it did is it basically had it right to a text file. Hey, if you're scheduled, write it off. So it's saying the first machine says I'm scheduled and it decided out of the available host of currently there's only one. It said this is the only one. Therefore, I'm only a schedule on the host, the first one. I don't know if you can actually see down here is one container was automatically created 24 seconds ago. And just instead of running four different VMs to do all this, I'm just doing it all locally. So each machine is named D-hostname. So H1, H2, H3. So let's run a second one and see what happens. So I'm going to run a second Docker host, the second MDMT. They'll automatically cluster together. And if we're lucky, you can see now that it starts up. And in a second, now we see a second container running because MDMT decided, okay, well, I have two hosts available, I asked for two containers to be running, and I put one on H1, one on H2, and you can see them running here. H2 right here and H1 down below. Do you see that? And each host is actually creating a text file just so it can visualize what's happening. This is the text file from the first host that says I am scheduled and H1 and H2 are the two hosts in the schedule set. Does that make sense? Just a cheap little schedule. You want to start a third one? Come on. Do you want to see a third one? Yeah. All right. So, we have one. Now, I've only asked for two things to be scheduled, right? So, if you look at the example, so watch what happens. Nothing happens, right? Because there's only two. And we've asked for two, a max of two, and a TTL of 10. So now, if we were to actually shut down the second one, we're going to shut down nicely. Watch what happens. We shut down that. Oh, two went away. And automatically, the first one has said now we've got to reschedule to host three. So H3, and then the third one finally came up when we have H3. And now you see we've got container H3 and container H1 running. You see that? And now, what if we restart up H2? It's not going to flop, because we don't want it to flop back and forth. So it should start up. And actually, it decided in this case to actually switch it. So, normally, it's fairly static, but you get the idea. And so, what we've basically done is with a very simple built-in function, we've allowed, I built in an hour, a custom little tiny play scheduler. Now, this is just a silly schedule. You could build something much more complicated that has a fancier scheduling algorithm. I just implemented around Robin strategy and so on. But that's the basic idea. Instead of building this huge monolith that has all these built-in components and built-in knobs, I decided to build this engine in this language that allows you to put in whatever resource you want, whether it's container, Docker, N-Spawn, whatever, and then other pieces that glue together to build what you want. You might not have to write the code for this. In the future, someone just might write a scheduling package that does all this stuff for you and you just say what images you want to run. But that's the basic idea. Any quick questions? Want to see some more stuff? Yeah? Okay. You're like somewhat enthusiastic. It's very warm in this room. It's like 300 Celsius. So that's the scheduling thing. Can kill this. I'm just going to actually kill it violently. I want it to go away. So we can do another demo. And you can, all this code is online, so you can play with this at home. So as I said, you don't need to have N-Spawn. This could be a vert resource or anything else. I think the Kubernetes project actually made a mistake by sort of hard coding the idea of containers as their primary thing. But their opinion ended that way, and I think that as a result, because they were so specific about that one solution, and that's the thing forever, it's caused a lot of problems in the code base. Very difficult to change the project for the future. So what is this? This is a great example because this room is a million Celsius. What is this? Yeah, it's a thermostat. It's from Canada. It's a photo I took in my parents' house. You can see it's in the international units of Celsius. It's very warm. I found a clear picture in some sort of arbitrary sauna units on the internet. And these things have actually a very interesting property. Does anyone know what it is? Historizes. Historizes. You saw my sock before? No. Okay, that's great. You're just a smart gentleman. So it has, it's hysteresis. And what hysteresis is, is it's also smart if you saw my talk, but that's okay. So let's say you get up to 20, you ask, please set your temperature to 20 Celsius and the temperature goes up and it hits 20, then it immediately clicks off and then heats up again. It goes back and forth, tick, tick, tick, tick really fast. This would be a big disaster. We don't want this to happen. So hysteresis actually, I see, we have the most enthusiastic sign wavers. Like 10 minutes left. I know, don't worry. Thank you. Higher this man. He needs a sign waving job. So hysteresis stops that flopping back and forth. So you can actually wait some amount of time or some distance below a threshold. And I'm going to show you, you want to see a demo of that? Yeah. Come on. Do you want to see a demo? Yeah. Well, it says live demoing for you. Okay, so I'm going to run this hysteresis example. Instead of containers, I'm going to use virtual machines just because it's a little change. And I'm going to run, so I'm going to run this watch command over here. It's going to print out the contents of this text file and run verse list. And what you can see, I'm actually, it's, MGMT on the left has started up these two virtual machines, one and two. And what I've actually done is it's also printing out the system load. So the kernel regenerates a new system load value every five seconds. And I've built in a threshold of 1.5. And the idea is that once we hit 1.5, we're asking MGMT to automatically shut down one of the VMs, right? There's too much load, too much traffic, so we're going to move that to some other machine somewhere and schedule it elsewhere. And so we're going to do that. And when the load drops below 1.5, we don't want to reschedule that VM, in this case, for 10 seconds. So let's try that. Does that make sense? So I'm just going to artificially increase the load just so that imagine the VM is doing a whole bunch of work. And I just got to trick Linux into thinking it's busy. Oops. So we're going to do that. My poor little laptop is hot enough as it is. It's going to be on fire after this. So CPUs are going, okay, poof. So you saw this first machine was there and the second machine disappeared? You saw that? So we're just watching it. And it's past 1.5. Now we're going to watch it go down. So it's still there. And below 1.5, it's going to take 10 seconds. So we're still not there. 1.8. MGMT is still running. It's just waiting, watching the threshold, reevaluating the code every time something changes. 1.6, 1.62. Any second? Keep watching for it. So it's blowing. Nothing happened. Right? We're waiting 10 seconds and then it should hopefully start it up again. 1.3. So five seconds left. Three, two, one. Boom. And it started right back up. You like that? Thank you. So again, if you wanted to build something like this with existing tools like Kubernetes, you'd have to find some special knob that some programmer spent tons of time building. And that's just absurd because there's always going to be some special infrastructure that's different that you want to do something custom your way without making it too complicated. And again, you don't have to necessarily build all this stuff. You can actually have someone else who wrote a module that does the general things that you use. So a bunch of companies use the same scheduling container module up to you. But the point is it's possible and it's very easy. You want to see some more stuff? Yeah. How much time do we have left? Let's see. I got to actually shut this down. 10 minutes left. Awesome. So I have some time. I'm just going to actually destroy these VMs. I get some, my poor laptop gets some love. So just some interesting properties about this stuff. So the language in the engine, these two parts to MGMT, there's actually a bit of a separation between the two. So in theory, the code actually is safe and it shouldn't crash. But if there is some disaster that happens, the engine can keep running even if the code actually hits an error. It's actually a safe language so there's no nils. You can't have like a nil pointer exception in these kinds of bugs. It's an immutable language. So if you did something like x equals 5, then x equals 6, it would be a compile time error. And the whole idea is to eliminate classes of errors before you run the code in production. So at compile time. So it's pretty cool. You might remember that earlier example I showed where things were kind of out of order. And you're like, what is this guy doing date times up here and the variables down there? And the reason I did this actually kind of as an example because the code actually doesn't need to be in a certain order. The code is actually a graph. It's just streams going from vertex to vertex of data. So the code can actually be written in out of order. Now you shouldn't do this in practice. You would be insane. But it's just an example to show you that the language engine doesn't actually care. Just looks at the code and it follows the data paths. Does that make sense? Yeah? It's a bit of a strange concept. I talked to you about immutable variables. I've shown you some hysteresis. So there's different kinds of hysteresis that based on a time limit or distance, things like that. So all sorts of fun stuff. Hysteresis is actually very important because hysteresis is actually a real world thing that our brains are sort of programmed to understand. For example, if your toaster is not working, you push the toaster thing in and the toaster doesn't work. You might like, you don't try right away again or you try once right away and then you wait, hmm, you think, why isn't the toaster working? And then you try again. So these delays and these backoffs and these logical things that humans do are actually possible because we think in time. So this reactive language is a time-based language for this reason. It models real life. And the idea is basically for error scenarios in real life, instead of getting a page and having to wake up and fix something, you just program in advance what the language should do for every second of the day and every situation possible. And then you have built an automatic system in, hopefully. Good idea, right? Yeah. Some people are like, yes, automatic. So this is some stuff we talked about. This is the important module system I'm going to be talking about at a different conference and I'll show you about it at the end. And there's still a lot to do with this project. It's still a relatively new project. It's not big and fancy. There's still some new stuff we need to do in the standard library. There's a whole bunch of new functions that would be great to add. New features and stuff are always something we need to do. There's a bit of code we need to clean up that's a bit messy. There's a few bugs and annoying things from some legacy stuff that I didn't know better at the time. And this is all about you. How can you help? Do you want to help? Yeah. I'm going to leave. Do you want to help? Yeah. All right. So what can you do? You can use this. Take a minute. Start it. Blog it. Tweet it. Discuss it. Just hack on this stuff. This is like hacking stuff. MGMT, I worked at a mostly cool tech company and I left a little while back to sort of work on this full time because I'm trying to do the free software thing. But the bad thing is I don't have really any money to do this. I'm just living off my savings. So if you want to help, send patches or money or both, that's cool. I have Patreon. It's just an attempt to see if it's helpful. And funding a hacker, it's very sexy. So don't be shy if you want to be sexy. This is the easy way. Let's just recap. I want to recap. There's no audio. This is just a bad joke of Arthur Benjamin putting the cat back on his pen. It's a bad joke. I reuse all my jokes and I apologize. So if you want to get involved, we have an IRC channel, MGMT config on FreeNode. We hang out. I see at least one MGMT contributor in the room. So join us. We have a Twitter account and a mailing list which you can hang out and listen to. There's a technical blog of James you all know about it now so you can check out RSS and read all the great blog posts I write. I'm purple idea at IRC and Twitter and GitHub and so on. So you can find me there. More stuff today. I'm actually somehow, Foste, I've accepted all of my talks except for my main track talk. It's just too bad. So I gave a whole bunch of talks already which is just absurd. That was yesterday. And then today I have one more talk in the config management and monitoring room which is basically just a five minute lightning talk. So you probably won't see anything new but it never ends. And these old talks you can go see the recordings of. I showed some different demos and different stuff and talked faster. So if you want to... The best fun thing is that there's this really cool conference called Config Management Camp which is happening tomorrow and the day after in Ghent. I'm giving two talks and Felix is also giving another talk who's a great contributor as well. And on the sixth right after that conference in Ghent as part of it, there's a hackathon. So if you want to actually get your hands on MGMT with me and some other people, writing code, writing your own function, writing your own resource, doing stuff, come check it out. Totally free, totally cool. If you like this talk, please go find a Foste organizer in those yellow shirts and say, hey, I really like James's talk, James Schubin purple idea. If everyone does that, it'll be like a DDoS and it'll be really funny. And if you don't want to do that, that's okay. If you go to the schedule link on the Foste website, if you look, there's a secret link right here. I didn't know about this for many years called submit feedback. So you find my talk, you click submit feedback, say this was dope, it was awesome, it was fun. If you give me some good advice, it costs you nothing, it would really help. Deal? Just do this. Imagine one guy gets like 200 submissions of awesome. They're going to eventually have me for a main track talk, right? All right. So I have some free stickers. There's a sticker cartel, so they're incredibly expensive. But if you promise to use them on your laptops, come at the end out there in the corner, I'll give you some away if you'd like and show them to your friends. You're going to have a cool laptop like me. And yeah, thank you very much. So before you leave, I have probably a few more minutes left. For questions? Maybe someone has questions? Yes. Questions. This is an incredible gentleman up here. I'm going to start putting stuff away so I'm out of the way. Yeah. Hi. Thanks for your talk. Here, run for our view. Oh, there you are. Hi. Like I'm a movie, so sorry if that's a completely absurd question that's been answered like many times before, like in the first page of the FAC or something. But I imagine that the whole dealing with being responsive when you're dealing with stuff that can send you like interrupts and tell you, oh, I changed this relatively easy. But what about like being reactive about stuff that doesn't announce it? Like do you have fears of like newbies starting to write code and then it's suddenly getting MGMT getting 100% CPU just because it's like polling stuff that isn't reactive? Yeah. So the way we work, every event source, whether the functions that actually look at data or the resources that actually look at changes, they all are based on events. So select calls, I notify her files, there's no polling. So if there's something we cannot do and we must poll, then the solution is to ask the kernel people, hey, we're missing something, we need an event source for this. So there's almost nothing, there's a few things left that don't have things and in those cases we could poll, but almost everything does have an event system or it's a bug. So really good question. Any more questions? Got another question over here. Hi, great talk. You talked about monitoring. Does it only work for the system itself or can we use it for logging or stuff like that and can we replace everything and just rely on the monitoring system? I don't, if you're leaving, try and be a little quiet. I don't understand 100% of what you're talking about, but if you can explain to me better, I think you're asking about can we log and do other things besides just automation? The answer is yes. So this is a general purpose engine for infrastructure automation stuff, whether you use it to get sources from events that come from monitoring systems and integrate them into your decisions or not, that's up to you. For example, you could write a Prometheus function which read from a Prometheus server that was sending data from something else and then respond to those things if you wanted to. Patch is welcome for sure. We have space for one more question. One more question. A gentleman in the front here maybe? And come get a sticker if you want. Mine goes with the first one. What's the overhead? If I'm looking for, I don't know, 1000 files, 100 virtual machines, how painful it is? It's incredibly fast. So it's probably the fastest tool out there if you were to compare it to Puppet and all these other equivalents. If you had some absurd requirements, there actually is a feature called auto grouping which will take multiple resources and group them together so it's more efficient. But here's one second. This guy really wants a sticker. He's like, hey. Yeah, so we actually can do some efficiency things like that. But if you did have something that was totally absurd and it's not possible, then chances are you're probably doing something that might not make sense. So if you find a use case that doesn't work for, tell us. But I think it's probably going to be good. It's in goal lying. It's very fast. And yeah. Thank you so much. Thank you. Thank you.
 So just for who doesn't know, this is Mr. James and he's going to present MGMT config  for containers.  So.  Thank you so much.  Applause.  So I'm just going to talk a little bit about containers because it's such a hot topic and  a little bit about why we don't treat containers exactly like other resources.  And by resources I mean like virtual machines or other things.  We're going to talk about that.  I'm going to do some videos.  Who am I?  I'm a hacker.  I talk really fast because I have a lot of stuff on our Facebook live demos.  So if you really miss something and you really desperately don't understand, raise your hand  and I'll try and answer quick questions.  But I'm going to try and show you a lot of demos and it's recorded so you can watch  it and slow at the end.  I write a technical blog called the Technical Blog of James.  Who's seen it?  Just raise your hand.  If you haven't seen it, just raise your hand so I seem really popular.  Thank you.  Excellent.  I work on, I used to be a physiologist.  That's what I did by training in and I don't really do much anymore.  But if you have cardiology questions I can try and help you out.  Big into DevOps.  I used to do a lot of puppets.  Who's done some puppet before?  No?  Okay.  So you know what I'm talking about a bit?  A lot of infrastructure automation.  I learned a lot of puppet but I thought there were a lot of things wrong with it.  And I realized very recently that I didn't really want to live in this Kubernetes monoculture.  Do you want to live in this Kubernetes monoculture?  Who said, yeah, get out of this room now.  So this is my answer to whether I really want to live in that world.  Because it's just silly.  It might be useful for some things.  So this is my nope guy.  And it's just, everything is nope.  So instead, long story short, some time back I started, I sat down and I started working  on a project called MGMT.  It was really intended as a solution to the problems in puppet.  But I think there's some overlap with what people are doing with Kubernetes and how they're  doing it wrong.  So I'm going to talk about that and hopefully convince you.  So in this project, open source, free software, whatever, there's two parts.  There's the engine and the language.  The engine actually does the work on the computer.  So it sets up containers, VMs, files, packages, services, et cetera.  And there's the language, which is actually what we use to describe everything to the  containers.  So, oops, sorry about that.  So there's three main parts to the engine.  It can run in parallel.  There's events, which I'll show you in a moment.  And the whole thing works as a distributed system.  So it's in contrast to something like Kubernetes for example, which is really essential orchestration.  And then there's the computer.  The graphs that we run, we run graphs.  That's our output.  And these things are resources.  So this might be install a package, start a service, and so on.  And these black arrows are dependencies.  So something might really, ah, similar to what you might be familiar with puppet and  so on.  Except we can run everything in parallel.  And the events can be anything.  They can be files, service, the resources can be anything.  They can be files, services, packages, end spawn resources, kind of container thing.  Docker containers, if you're into Docker.  I'm personally not, but I'll show you a little demo of what I mean by event driven.  So let me just actually, so I have to actually turn off.  Is this big enough?  Can you see?  No.  No?  Bigger?  Yes, no?  Don't be shy.  Is it big enough?  Bigger?  Okay.  So what I'm going to actually do is I'm going to actually just start up MGMT.  I'm going to show you just, I'm going to start with a little example where I go, hello,  FOSSTEM.  So I'm going to run this resource that basically says, please create one container, one file.  And if we go into this directory here on the left, you can see, oops, I'm going to remove  this mess.  I can show you that it's created this file called hello world.  And if I remove the file, you can see it comes right back.  So I remove it and it comes right back.  And this is what I mean by event.  So each resource actually is being watched by the engine.  And whenever something goes wrong, you can actually just remove it and cat, it's so fast  it even comes back before it gets to the end there.  And this is not fake.  You can see the engine running as this happens.  So it sleeps and it wakes up.  And it's so fast you can even run it in this watch command, which just loops something  over and over again really, really quickly.  And you'll see that it just always sort of keeps you in the correct state.  And I have actually an example with an N spawn resource.  If you'd like to see that.  So N spawn 0.  So this one actually requires root because N spawn requires root.  And this is actually kind of new.  So I have to see if this is going to work because there's a bug actually in this resource.  But I want to show you it anyways.  I think it's actually not working right now.  I have one second to kill this.  It's N spawn.  So if you want to do an N spawn resource, you can do it just like this.  Just basically declare the resource.  And I have this demo.  So I was working on a fancier demo with N spawn.  I'm not actually going to show you that right now.  So I can actually just show you.  I'll show you this one because I was really working to get this ready.  Sorry, one second.  What's my password?  Who knows my root password?  Anybody?  So I used Fedoro on my laptop.  I actually wanted to do a Debian demo.  So this is what it is.  So it's actually, you just need the name of the machine.  Let's see if this works.  Okay.  There we go.  So these are just the declarative resources.  And if I go back here, and I run this one here, I think that's going to work.  All right.  So machine CTL, I think it's list.  And there you can see it's running.  And the same sort of thing happens.  Like if you, I forget if it's terminate, stretch.  Oops.  If I kill the container, I'll type my password.  You see it actually, MGMT wakes up and it brings it right back.  Does that make sense?  Cool.  So I like demo a little bit.  I just had to get it ready for you.  So this is the whole point.  So we have these primitives.  They all have events.  Now let's do some fancier stuff.  So and this works for everything, virtual machines, files, packages, services.  And I think this is actually kind of like monitoring.  So typically we had to set up infrastructure and then monitoring would be a second step.  But if we build this in this event driven nature into each resource, it's already partially  done for you.  So now we have to describe how you actually tell the computers what to do.  So normally like in the Kubernetes land, Ansible land, it's blasting these giant YAML  files, which to me makes no sense because it's not expressive.  You can only play with the knobs that they've programmed for you.  So I wanted to build a language.  It's a DSL, so a very small language that lets you actually build your own custom creative  things.  The language has to be really safe because you don't want to make an off by one error  and blow away a data center.  It has to be very powerful.  So it's a reactive language, kind of like an FRP.  If you know what FRPs are, that's OK.  You'll understand in a minute.  And here's the demo I'm going to show you.  So what I do here, I'm just going to actually start off this demo.  I'm going to start off this demo on the left.  And on the right here, I'm just going to watch this file.  There we go.  So what's happening?  I'm running this code.  And here's what it basically looks like.  I have this datetime function at the top.  And it adds the value from this output to a year, which is some variable that has a whole  bunch of integers multiplied together.  Then there's this load value down here.  And it goes into this variable.  And then both of those variables go into this struct, which get put into this big print  tap string, which gets set as the content of this file.  Right?  So what's going on?  But here's what's happening when it's running.  Oops.  Sorry.  It's running over here.  Here's what's happening when it's running.  On the left, you have MGMT running.  On the right, I'm just watching this file just so you can see what the output is.  And you can see it's actually changing about every second.  Because MGMT is actually constantly running, constantly deciding that the value should be  different in the file.  And it's updating the file every second.  So what's actually happening here is this date time function is actually a stream of  values.  So it's the stream of number of seconds every second that goes by.  Right?  So the data is always going up.  And it's putting all these values.  Here's the load on the actual system, which changes constantly.  Putting all those values.  And it's reevaluating just the parts of the code that it needs to, which ultimately results  in this file resource that has different contents.  See that?  And then that all gets put into this file.  And you can see that it actually changes.  Does that make sense?  And just to make it a little bit more fun, I actually built a function, a custom function  called VU Meter.  And what it's doing, it's running.  It's actually listening to my laptop microphone right now.  And so if you make a lot of noise, you see it actually goes up.  And just to prove you this is real, we're going to be very quiet.  And when I point to you, I want you to make a whole lot of noise and see that it's working.  Okay?  All right?  So nothing I could do.  And so this is the kind of thing.  We're taking real-time event sources and we're integrating them in this small, simple, safe  language that lets you do very powerful things.  So the kind of things you could do, for example, is you could take this and maybe you have this  in your server room or in your little office.  And when there's lots of loud noises and people are screaming and fighting, you automatically  set all your systems read-only so that nothing is done in anger, right?  So that's the kind of joke I make.  So that's the sort of idea.  But you want to see some more stuff?  Do you want to hear some more stuff?  These people leaving are like, nope, I don't want to see some more stuff.  All right.  So we're going to do some fancy scheduling stuff.  So in addition to just these silly examples, we actually, as I said, work as a distributed  system and we've built into the core of MGMT some functions and one of the special functions  is called schedule.  And what it does is it actually takes all the machines that run it and produces a list  of machines that are selected by a distributed scheduler.  So what it's going to do, it's going to do this and it's going to run in each machine.  So I'm going to just run it.  It's actually a new demo.  So let's see if this works well.  So I'm going to run it.  I'm going to run it at a time.  And oops, schedule.  Schedule dash Docker.  So I decided to do this with a Docker.  So I'm going to run the first one and over here, run a watch.  I'm going to run this big fancy watch command.  What it does is it's just going to sleep and it's going to tail some files and run Docker  PS and sleep and so on.  So I'm just going to run that and you actually can see at the moment this is running.  Did it work?  The Docker thing is quite new.  So let's see what's going on.  Just kill this for a second.  Okay.  Docker PS.  Docker images.  Is it going?  Is it starting up or is it unhappy?  Oh, I don't have internet.  That's why.  I need some internet.  I need some internet.  Sorry, folks.  If you're torrenting, please turn off your torrents.  It doesn't need a lot of internet because I've pre-cached the images but Docker is sneaky  about this stuff.  Okay, we have internet.  Just kill this.  Okay, let's start this up.  I need to system.  CTL start.  Okay, it's working.  So on the left I started up the first machine and what it did is it basically had it right  to a text file.  Hey, if you're scheduled, write it off.  So it's saying the first machine says I'm scheduled and it decided out of the available  host of currently there's only one.  It said this is the only one.  Therefore, I'm only a schedule on the host, the first one.  I don't know if you can actually see down here is one container was automatically created  24 seconds ago.  And just instead of running four different VMs to do all this, I'm just doing it all  locally.  So each machine is named D-hostname.  So H1, H2, H3.  So let's run a second one and see what happens.  So I'm going to run a second Docker host, the second MDMT.  They'll automatically cluster together.  And if we're lucky, you can see now that it starts up.  And in a second, now we see a second container running because MDMT decided, okay, well,  I have two hosts available, I asked for two containers to be running, and I put one on  H1, one on H2, and you can see them running here.  H2 right here and H1 down below.  Do you see that?  And each host is actually creating a text file just so it can visualize what's happening.  This is the text file from the first host that says I am scheduled and H1 and H2 are  the two hosts in the schedule set.  Does that make sense?  Just a cheap little schedule.  You want to start a third one?  Come on.  Do you want to see a third one?  Yeah.  All right.  So, we have one.  Now, I've only asked for two things to be scheduled, right?  So, if you look at the example, so watch what happens.  Nothing happens, right?  Because there's only two.  And we've asked for two, a max of two, and a TTL of 10.  So now, if we were to actually shut down the second one, we're going to shut down nicely.  Watch what happens.  We shut down that.  Oh, two went away.  And automatically, the first one has said now we've got to reschedule to host three.  So H3, and then the third one finally came up when we have H3.  And now you see we've got container H3 and container H1 running.  You see that?  And now, what if we restart up H2?  It's not going to flop, because we don't want it to flop back and forth.  So it should start up.  And actually, it decided in this case to actually switch it.  So, normally, it's fairly static, but you get the idea.  And so, what we've basically done is with a very simple built-in function, we've allowed,  I built in an hour, a custom little tiny play scheduler.  Now, this is just a silly schedule.  You could build something much more complicated that has a fancier scheduling algorithm.  I just implemented around Robin strategy and so on.  But that's the basic idea.  Instead of building this huge monolith that has all these built-in components and built-in  knobs, I decided to build this engine in this language that allows you to put in whatever  resource you want, whether it's container, Docker, N-Spawn, whatever, and then other  pieces that glue together to build what you want.  You might not have to write the code for this.  In the future, someone just might write a scheduling package that does all this stuff  for you and you just say what images you want to run.  But that's the basic idea.  Any quick questions?  Want to see some more stuff?  Yeah?  Okay.  You're like somewhat enthusiastic.  It's very warm in this room.  It's like 300 Celsius.  So that's the scheduling thing.  Can kill this.  I'm just going to actually kill it violently.  I want it to go away.  So we can do another demo.  And you can, all this code is online, so you can play with this at home.  So as I said, you don't need to have N-Spawn.  This could be a vert resource or anything else.  I think the Kubernetes project actually made a mistake by sort of hard coding the idea  of containers as their primary thing.  But their opinion ended that way, and I think that as a result, because they were so specific  about that one solution, and that's the thing forever, it's caused a lot of problems in  the code base.  Very difficult to change the project for the future.  So what is this?  This is a great example because this room is a million Celsius.  What is this?  Yeah, it's a thermostat.  It's from Canada.  It's a photo I took in my parents' house.  You can see it's in the international units of Celsius.  It's very warm.  I found a clear picture in some sort of arbitrary sauna units on the internet.  And these things have actually a very interesting property.  Does anyone know what it is?  Historizes.  Historizes.  You saw my sock before?  No.  Okay, that's great.  You're just a smart gentleman.  So it has, it's hysteresis.  And what hysteresis is, is it's also smart if you saw my talk, but that's okay.  So let's say you get up to 20, you ask, please set your temperature to 20 Celsius and the  temperature goes up and it hits 20, then it immediately clicks off and then heats up again.  It goes back and forth, tick, tick, tick, tick really fast.  This would be a big disaster.  We don't want this to happen.  So hysteresis actually, I see, we have the most enthusiastic sign wavers.  Like 10 minutes left.  I know, don't worry.  Thank you.  Higher this man.  He needs a sign waving job.  So hysteresis stops that flopping back and forth.  So you can actually wait some amount of time or some distance below a threshold.  And I'm going to show you, you want to see a demo of that?  Yeah.  Come on.  Do you want to see a demo?  Yeah.  Well, it says live demoing for you.  Okay, so I'm going to run this hysteresis example.  Instead of containers, I'm going to use virtual machines just because it's a little change.  And I'm going to run, so I'm going to run this watch command over here.  It's going to print out the contents of this text file and run verse list.  And what you can see, I'm actually, it's, MGMT on the left has started up these two  virtual machines, one and two.  And what I've actually done is it's also printing out the system load.  So the kernel regenerates a new system load value every five seconds.  And I've built in a threshold of 1.5.  And the idea is that once we hit 1.5, we're asking MGMT to automatically shut down one  of the VMs, right?  There's too much load, too much traffic, so we're going to move that to some other machine  somewhere and schedule it elsewhere.  And so we're going to do that.  And when the load drops below 1.5, we don't want to reschedule that VM, in this case,  for 10 seconds.  So let's try that.  Does that make sense?  So I'm just going to artificially increase the load just so that imagine the VM is doing  a whole bunch of work.  And I just got to trick Linux into thinking it's busy.  Oops.  So we're going to do that.  My poor little laptop is hot enough as it is.  It's going to be on fire after this.  So CPUs are going, okay, poof.  So you saw this first machine was there and the second machine disappeared?  You saw that?  So we're just watching it.  And it's past 1.5.  Now we're going to watch it go down.  So it's still there.  And below 1.5, it's going to take 10 seconds.  So we're still not there.  1.8.  MGMT is still running.  It's just waiting, watching the threshold, reevaluating the code every time something  changes.  1.6, 1.62.  Any second?  Keep watching for it.  So it's blowing.  Nothing happened.  Right?  We're waiting 10 seconds and then it should hopefully start it up again.  1.3.  So five seconds left.  Three, two, one.  Boom.  And it started right back up.  You like that?  Thank you.  So again, if you wanted to build something like this with existing tools like Kubernetes,  you'd have to find some special knob that some programmer spent tons of time building.  And that's just absurd because there's always going to be some special infrastructure that's  different that you want to do something custom your way without making it too complicated.  And again, you don't have to necessarily build all this stuff.  You can actually have someone else who wrote a module that does the general things that  you use.  So a bunch of companies use the same scheduling container module up to you.  But the point is it's possible and it's very easy.  You want to see some more stuff?  Yeah.  How much time do we have left?  Let's see.  I got to actually shut this down.  10 minutes left.  Awesome.  So I have some time.  I'm just going to actually destroy these VMs.  I get some, my poor laptop gets some love.  So just some interesting properties about this stuff.  So the language in the engine, these two parts to MGMT, there's actually a bit of a separation  between the two.  So in theory, the code actually is safe and it shouldn't crash.  But if there is some disaster that happens, the engine can keep running even if the code  actually hits an error.  It's actually a safe language so there's no nils.  You can't have like a nil pointer exception in these kinds of bugs.  It's an immutable language.  So if you did something like x equals 5, then x equals 6, it would be a compile time error.  And the whole idea is to eliminate classes of errors before you run the code in production.  So at compile time.  So it's pretty cool.  You might remember that earlier example I showed where things were kind of out of order.  And you're like, what is this guy doing date times up here and the variables down there?  And the reason I did this actually kind of as an example because the code actually doesn't  need to be in a certain order.  The code is actually a graph.  It's just streams going from vertex to vertex of data.  So the code can actually be written in out of order.  Now you shouldn't do this in practice.  You would be insane.  But it's just an example to show you that the language engine doesn't actually care.  Just looks at the code and it follows the data paths.  Does that make sense?  Yeah?  It's a bit of a strange concept.  I talked to you about immutable variables.  I've shown you some hysteresis.  So there's different kinds of hysteresis that based on a time limit or distance, things  like that.  So all sorts of fun stuff.  Hysteresis is actually very important because hysteresis is actually a real world thing that  our brains are sort of programmed to understand.  For example, if your toaster is not working, you push the toaster thing in and the toaster  doesn't work.  You might like, you don't try right away again or you try once right away and then you wait,  hmm, you think, why isn't the toaster working?  And then you try again.  So these delays and these backoffs and these logical things that humans do are actually  possible because we think in time.  So this reactive language is a time-based language for this reason.  It models real life.  And the idea is basically for error scenarios in real life, instead of getting a page and  having to wake up and fix something, you just program in advance what the language should  do for every second of the day and every situation possible.  And then you have built an automatic system in, hopefully.  Good idea, right?  Yeah.  Some people are like, yes, automatic.  So this is some stuff we talked about.  This is the important module system I'm going to be talking about at a different conference  and I'll show you about it at the end.  And there's still a lot to do with this project.  It's still a relatively new project.  It's not big and fancy.  There's still some new stuff we need to do in the standard library.  There's a whole bunch of new functions that would be great to add.  New features and stuff are always something we need to do.  There's a bit of code we need to clean up that's a bit messy.  There's a few bugs and annoying things from some legacy stuff that I didn't know better  at the time.  And this is all about you.  How can you help?  Do you want to help?  Yeah.  I'm going to leave.  Do you want to help?  Yeah.  All right.  So what can you do?  You can use this.  Take a minute.  Start it.  Blog it.  Tweet it.  Discuss it.  Just hack on this stuff.  This is like hacking stuff.  MGMT, I worked at a mostly cool tech company and I left a little while back to sort of  work on this full time because I'm trying to do the free software thing.  But the bad thing is I don't have really any money to do this.  I'm just living off my savings.  So if you want to help, send patches or money or both, that's cool.  I have Patreon.  It's just an attempt to see if it's helpful.  And funding a hacker, it's very sexy.  So don't be shy if you want to be sexy.  This is the easy way.  Let's just recap.  I want to recap.  There's no audio.  This is just a bad joke of Arthur Benjamin putting the cat back on his pen.  It's a bad joke.  I reuse all my jokes and I apologize.  So if you want to get involved, we have an IRC channel, MGMT config on FreeNode.  We hang out.  I see at least one MGMT contributor in the room.  So join us.  We have a Twitter account and a mailing list which you can hang out and listen to.  There's a technical blog of James you all know about it now so you can check out RSS  and read all the great blog posts I write.  I'm purple idea at IRC and Twitter and GitHub and so on.  So you can find me there.  More stuff today.  I'm actually somehow, Foste, I've accepted all of my talks except for my main track  talk.  It's just too bad.  So I gave a whole bunch of talks already which is just absurd.  That was yesterday.  And then today I have one more talk in the config management and monitoring room which  is basically just a five minute lightning talk.  So you probably won't see anything new but it never ends.  And these old talks you can go see the recordings of.  I showed some different demos and different stuff and talked faster.  So if you want to...  The best fun thing is that there's this really cool conference called Config Management Camp  which is happening tomorrow and the day after in Ghent.  I'm giving two talks and Felix is also giving another talk who's a great contributor as  well.  And on the sixth right after that conference in Ghent as part of it, there's a hackathon.  So if you want to actually get your hands on MGMT with me and some other people, writing  code, writing your own function, writing your own resource, doing stuff, come check it out.  Totally free, totally cool.  If you like this talk, please go find a Foste organizer in those yellow shirts and say,  hey, I really like James's talk, James Schubin purple idea.  If everyone does that, it'll be like a DDoS and it'll be really funny.  And if you don't want to do that, that's okay.  If you go to the schedule link on the Foste website, if you look, there's a secret link  right here.  I didn't know about this for many years called submit feedback.  So you find my talk, you click submit feedback, say this was dope, it was awesome, it was fun.  If you give me some good advice, it costs you nothing, it would really help.  Deal?  Just do this.  Imagine one guy gets like 200 submissions of awesome.  They're going to eventually have me for a main track talk, right?  All right.  So I have some free stickers.  There's a sticker cartel, so they're incredibly expensive.  But if you promise to use them on your laptops, come at the end out there in the corner, I'll  give you some away if you'd like and show them to your friends.  You're going to have a cool laptop like me.  And yeah, thank you very much.  So before you leave, I have probably a few more minutes left.  For questions?  Maybe someone has questions?  Yes.  Questions.  This is an incredible gentleman up here.  I'm going to start putting stuff away so I'm out of the way.  Yeah.  Hi.  Thanks for your talk.  Here, run for our view.  Oh, there you are.  Hi.  Like I'm a movie, so sorry if that's a completely absurd question that's been answered like  many times before, like in the first page of the FAC or something.  But I imagine that the whole dealing with being responsive when you're dealing with stuff  that can send you like interrupts and tell you, oh, I changed this relatively easy.  But what about like being reactive about stuff that doesn't announce it?  Like do you have fears of like newbies starting to write code and then it's suddenly getting  MGMT getting 100% CPU just because it's like polling stuff that isn't reactive?  Yeah.  So the way we work, every event source, whether the functions that actually look at data or  the resources that actually look at changes, they all are based on events.  So select calls, I notify her files, there's no polling.  So if there's something we cannot do and we must poll, then the solution is to ask the  kernel people, hey, we're missing something, we need an event source for this.  So there's almost nothing, there's a few things left that don't have things and in those cases  we could poll, but almost everything does have an event system or it's a bug.  So really good question.  Any more questions?  Got another question over here.  Hi, great talk.  You talked about monitoring.  Does it only work for the system itself or can we use it for logging or stuff like that  and can we replace everything and just rely on the monitoring system?  I don't, if you're leaving, try and be a little quiet.  I don't understand 100% of what you're talking about, but if you can explain to me better,  I think you're asking about can we log and do other things besides just automation?  The answer is yes.  So this is a general purpose engine for infrastructure automation stuff, whether you use it to get  sources from events that come from monitoring systems and integrate them into your decisions  or not, that's up to you.  For example, you could write a Prometheus function which read from a Prometheus server  that was sending data from something else and then respond to those things if you wanted  to.  Patch is welcome for sure.  We have space for one more question.  One more question.  A gentleman in the front here maybe?  And come get a sticker if you want.  Mine goes with the first one.  What's the overhead?  If I'm looking for, I don't know, 1000 files, 100 virtual machines, how painful it is?  It's incredibly fast.  So it's probably the fastest tool out there if you were to compare it to Puppet and all  these other equivalents.  If you had some absurd requirements, there actually is a feature called auto grouping  which will take multiple resources and group them together so it's more efficient.  But here's one second.  This guy really wants a sticker.  He's like, hey.  Yeah, so we actually can do some efficiency things like that.  But if you did have something that was totally absurd and it's not possible, then chances  are you're probably doing something that might not make sense.  So if you find a use case that doesn't work for, tell us.  But I think it's probably going to be good.  It's in goal lying.  It's very fast.  And yeah.  Thank you so much.  Thank you.  Thank you.
Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_0_2399.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_2400_2483.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_2484_4235.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_4236_5027.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_5028_6611.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_6612_7403.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_7404_7475.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_7476_8087.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_8088_10571.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_10572_12575.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_12576_12683.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_12684_13415.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_13416_13811.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_13812_14711.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_14712_14771.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_14772_14819.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_14820_15047.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_15048_15095.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_15096_15539.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_15540_15671.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_15672_15923.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_15924_16835.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_16836_18743.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_18744_24071.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_24072_24407.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_24408_24527.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_24528_26327.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_26328_33671.png Real-time_container_automation_and_scheduling_Mgmt_Config__Containers_lVsrtfmiJfo_33672_42131.png