Why Embedded Linux Needs a Container Manager Written in C
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Mendoza, Ricardo
Container technology has always been part of the cloud domain, and as such, its roadmap has usually been dictated by the use cases and requirements of that world. In the servers’ domain, resource utilization is nowhere near as relevant as it is in the embedded domain. The different languages and technologies that power the tools and mechanisms through which containers are leveraged in the bare metal server and /cloud worlds just don’t fit into the requirements of embedded. Despite the above, these past couple of years have seen an aggressive push from cloud-centric companies trying to tell the Embedded Linux ecosystem and its development community that we should make do with Golang, NodeJS and similar solutions and tools. Most are unaware of the challenges when you cram cloud tools into a resource constrained embedded system. Even though the architecture of some of these frameworks have the right intention (LXD), most just lack the interest in understanding the specific requirements of embedded. In this talk, we’ll explore how using containers for embedded systems modelling can help facilitate development cycles by enabling modular software architectures. We’ll deep dive into what the real requirements of embedded systems are and how modern container technology can help us meet the actual needs of this world. And lastly, we’ll walk through an example with Pantavisor, an open source container framework implemented for embedded systems.
 Hello everyone, my name is Ricardo Mendoza. I come from a company called Pantacore and today I want to tell you why we think that embedded Linux needs a container manager that is written in pure C. First of course, who am I? My nickname is Rick M.M. I have been involved with Linux since the early 2000s. There was a time, also early 2000s, when I got close to the MIPS architecture. At some point I think because I got obsessed with porting Linux on a bunch of early hand LPCs. This is before smartphones. These devices were quite cool and they used to be MIPS early on, some of them SH before ARM came and took the whole pie. Some of these MIPS and then later ARM work led to a drive for understanding low-level OS concepts. I ended up becoming a Gen2 developer as well for a bunch of years, maintaining a bunch of MIPS hardware. I no longer do that. And then I worked at Canonical for five, six years, something like that, working not on the main distro but actually on the device strategies and device side of things from Canonical. For example, early day testing of Ubuntu and mobiles and tablets, which resulted in Ubuntu Touch or Ubuntu and mobile as it was called back then. Now it's called Ubuntu Touch thanks to the community effort that still maintains this, which I'm also a part. And after that I was part of the team that wrote about Ubuntu Core. That was the end of my relationship with Canonical. After that I left on the most part to find what we are doing now. I am one of the founders at Pantacore since 2016. So been working there for a bunch of years now. And as a side note, I also sit on the board of the Ureports Foundation, which advances Ubuntu Touch with a pure community effort. Now a little disclaimer. This session is not a tech deep dive. We're not going to be running tutorials for you to follow up on your laptops. But rather is a conversation provoking exercise. The idea here is to discuss the topics at hand. And of course, also at the end, there will be a shameless blog. I will tell you a bit about the container management engine that I wrote for embedded. Now the important thing here is not what is written or what did I write. It's actually more why was something written in the way that I did and why do we believe that a container management engine in pure sea is very important to the modernization and the advancement of all the great work that this bunch of people here do. At the end of the day, what we do powers the infrastructure of the world. There is everything that people use, all the gadgets, all the things that makes people's lives easier. It's thanks to embedded Linux. So first and foremost, whales are too big for embedded boards. Why do I say that? Well, it has been a huge push from the cloud, from cloud vendors, cloud people, cloud developers, whatever you want to call it, about bringing the technologies that have made sense on the cloud and how to fit them one to one in the world of embedded. You know, this is kind of weird because there's a bunch of people that have not a lot of experience, let's say, on what true embedded means, telling us how we should be using certain tools that have made sense for cloud. And that's why I have this image here, you know, a managerial decision and an issue that happened at some point meant that these people had to solve a weird problem of putting a square into a hole. Over there it worked and they were able to bring their astronauts back, but that doesn't mean that we should always follow that approach, square peg into a run hole type of thing. So that's not to say that containers are not useful for embedded, on the contrary. Containers have modernized cloud computing and we think it can do the same for embedded. We're seeing it already. We know the usual benefits of containers, efficient resource utilization, modularization of software, portability, security, practices like CI CD pipelines become straightforward when you have this type of modular software architectures. And you know, Docker containers and OCI containers and the like have been perfect for the cloud where resources are near limitless. How does that work with most embedded Linux devices today? Let's see. Well, first the question is how is the cloud different from embedded Linux devices? This might sound like an obvious answer to you guys, right? But to a bunch of these people that are investing tens of millions of dollars in marketing budget to tell us run Docker on your tiny device, I think it's not obvious, right? So let's say it here. A data center has near infinite resources, right? A server is, you know, you won't lose power. It has infinite connectivity. It is resilient. It has a technician next to it. If something happens to it, right? And it's not truly mission critical, right? With the current type of cloud architecture, things are very ephemeral, right? Something dies somewhere. It will just spawn again somewhere else, right? Infinite replication means that, you know, if infrastructure fails somewhere, it's not a huge issue, right? It is an issue. Don't get me wrong, but it's not a huge one. Now tell that to your mother's router, right? Your Wi-Fi router in her home. If it goes down, that's it. No more Netflix, right? That's to call the technician at the ISP. That might involve a call. You might be without connectivity for 24 hours. That's mission critical in a user's home, right? I'm not even going to get started with robots in a factory. I mean, I'm pretty sure you can think about that one yourselves. But, you know, for the common user, it's also mission critical, right? If your thermostat fails, well, that's bad, right? You freeze. If your fridge fails, that's bad. Your food goes bad, right? So embedded is mission critical. Embedded requires extreme resilience, right? And usually the resources are extremely constrained. So, you know, this is basically what we're saying. We can agree that maybe your minimum specifications for an embedded Linux device look something like this in this current day and age, January 2022. This evolves. Maybe last year I would have said 16 megabytes of Nor Flash. Some people would be doing. But for example, we can see a bunch of routers nowadays that are going to 128 megabytes of NAND storage. Some of them even have EMMC. But still, most embedded devices aren't the lower range of this. A lot of them we don't have a flash translation layer even, right? Our usual architectures, we know them, ARM, MIPS, RISC-V. And virtually, of course, I say any Linux distribution, but in reality what it means is whatever goes, right? Whatever you need to actually build your system. You want to build something atop a yachto, open embedded, or whatever, go for it, right? You want to use a Alignar reference build for some board. Go for it, right? Or let's say you want to make a router, most likely you're going to go the way of open WRT, right? So now, going back to the topic, right? Why do we think that with this type of resources or why do we think that containers are important for embedded, right? Well, if we do away with the problem of do this cloud tools fit, right? We remain behind with the benefits. We say, well, modern software development and firmware lifecycle management at speed, right? Modernizing the way that we manage the software on an embedded device is important, right? We have been doing it the same way for the past 20 years, 25 years plus, right? Maybe it's time to change, right? We're starting to see the same problems that the cloud was seeing six, seven years ago, right, when servers and applications were just getting too huge, too complex to manage, right? So, what does that mean, right? I mean, what do containers help us do over here, right? In the embedded world that is starting to see that problem. On the diagram on the left, you can see what I mean by that problem, right? The embedded firm worms becoming this huge monolithic image, right? That just incorporates all those features. The bigger and the more complex that gets, the longer your release cycles are. This prevents valuable and fast innovation, you know? A monolithic release is time consuming to produce, it's error prone, right? And you know, the two above points result in that some devices end up being very stale out there in the field, right? They're not updated. They can be very insecure and vulnerable to attack, right? This creates incredible surfaces of attack. We have seen it many times. You see some ISP in Germany a few couple years ago ended up having over a million routers compromised, created a botnet that actually required, I think, a technician visit or sending the box back or something like that. Imagine the cost of that, right? But you know, that's the sort of catastrophic scenario. But on the normal day-to-day business, just those long release cycles that prevent innovation prevent you from adding new features fast enough, right? So we need to start doing what this diagram on the left is doing. We need to start sort of releasing the pressure of that monolith into a feature-rich pipeline. We need to start releasing so that it becomes modular applications that sit on top of a smaller core, let's call it, right? And to achieve this, well, the easiest way, of course, is with containers, right? Let's not try to reinvent the wheel, right? Let's just make sure that our wheel fits our use case, right? However, that's the main point here. Everybody is telling you use Docker, use OCI runtimes, base solutions. But why? You know, embedded wasn't built, sorry, Docker wasn't built for embedded devices. Most of these OCI-compliant runtimes out there wasn't built with embedded in mind, right? So we propose a different type of architecture that doesn't mean having a huge OS and then on top of that, a container runtime and then on top of that, container applications, but rather a shift in paradigm, right? We're true proponents of something that we like to call the minimal container runtime, right? I think that the legacy, let's say, Docker type of architecture where you have your hardware, or a host OS on top, a bloated host OS, then a Docker engine and then on top of that, your business logic is just too much overhead, right? It's too much for the majority of low spec embedded devices out there, right? And, you know, embedded nowadays, it's about more features, but it still is a well-defined, it still is about a well-defined feature set. The operator, the device manufacturer, they want to extend the features, but they want to be in control of those features, right? So do we even need a main OS, a big host OS sitting below all of everything we just mentioned as we know them in our true embedded use case, right? Is that necessary or do we just need to modularize the actual software functions that are sitting on this device, right? And that's what we propose, right? We propose doing away with the concept of a host OS. Why should there be a host OS? Then we just have a very tiny container runtime that sets up one too many user lands running on top of it via containerization tooling and technologies of the kernel, right? In the diagram on the right, you can see what I mean, right? There's one that says host OS, but in reality, this is just another application. It's just another user land that might start one too many services inside of a container that provides some functionality. For example, networking, whereas another one of these applications might provide power control that's safe for a, I don't know, a fridge or something like that. Now there's a little text here that says hint, in it user namespace, right? I have always said that this is an evolution of, or rather an extension of what the kernel does by default, right? On your left hand model, that host OS is also running in a container for all intents and purposes. I mean, you can extrapolate a little bit from some concepts of the kernel, but your main OS always runs into something called the init user namespace. Oh, surprise, the kernel is actually running containers for everything. So what we're doing here is extending that concept and saying, okay, why don't we use that init user namespace, that main namespace just to provide a very minimal container runtime and then run all of the actual relevant software functions on top of it. This is interesting, right? Because this lets us do what? Let's us manage the entire lifecycle of every software component on the device in a very straightforward manner. It lets us decouple the lifecycle of all the higher level software that is running on that device and the firmware that is running below that high level software, but through a common protocol. Now, if we have a minimal container runtime, well, we have to decide what to write it on, right? And this is where it gets interesting. This is where you start talking about, or at least having the conversation as to, is it what the cloud vendors are telling us or is it something else? We think it's C, right? So a minimal container runtime, YC, right? Well, first of all, most of those cloud IoT proponents out there with their whole 3.7 seconds of experience with embedded have been employing a lot of web developers to solve the challenges of embedded. But these are web developers, right? And they have been telling us how embedded should be done. Okay, well, that's a bit of a brute force attempt, let's say, right? But in reality, we have heard it is not uncommon for these people to tell you, manage your embedded firmware with an OJS or GoLang Ridden Engine on your 32 megabytes of NOR flash, or maybe 128 megabytes of NAND, come on, right? We think that a minimal container runtime should be considered baseline infrastructure. It is much closer to a role of the kernel than that to user land applications. Yes, okay, of course, the details. Let's go back to my conversation about in-user namespace. We think that it should fit anywhere. It should be as portable as possible. So why break, why change what is in broken, right? Let's write it in C after all. Like C is pure C, and this was the true first container runtime, let's call it, NOR engine, right? So what have we done? This is where we plug into our story, right? And in Pandacore, we have built something called Pandavizer, right? Which is an in-built and lightweight container manager, right? We built it with the needs of embedded in mind at the center. That's all we care about here. We're talking about a container engine, a minimum initial RAM disk that, you know, an overhead of no more than one megabyte, let's say, with strict build rules that allow you to configure and change all of those things according to the requirements of your device. It is specially suited to small, NAND or flash type devices, but of course it can scale the way up to huge things like a Raspberry Pi, right? It is highly portable. It is built with C, using a pluggable runtime architecture. And the main thing we use is actually LXC containers to run each one of the individual payloads. Remember, Pandavizer is in charge of the lifecycle management of all of those containers, right? We're not reinventing LXC. We're just providing the right scaffolding around it in this particular case to manage the lifecycle of your embedded devices. And Pandavizer is a single purpose system. It doesn't try to be a full user land or a full OS. It doesn't let you configure your networking or figure out all the drivers for your screens or whatever. No, the only purpose is to orchestrate the lifecycle management of your device's user land, which would be in that case that main OS, which is containerized here as well, as well as other application containers, but also your BSP components, your kernel, your firmware, your modules. By providing a single way of defining your entire system, then you can modularly manage each component of that system. And it is fast, right? This is just containers. As we said, it's a one megabyte C program that runs before anything else. And there is no true performance hit, right? But you get all the benefits of modern lifecycle management. So what does Pandavizer let you do? It lets you convert, turn your monolithic embedded system into a set of portable and reusable microservices, right? Reuse the code across different projects, make it in a way that those dark boxes at the bottom, BSP and hardware, can be interchanged while the things on the top can also be interchanged. Pandavizer is the one that then has the knowledge of who are we running on and how can we bring certain applications to run on top of this BSP, right? This is very powerful because it lets your product teams, it lets developers, it lets everyone actually decouple the lifecycle of those business logic units that are written for these devices, for these embedded devices, and decoupled it from the lifecycle of the firmware components, your BSP, your kernel, your modules, your actual static firmware, right? All of this is achieved with something that we call the Pandavizer system format, which is a way that in a single file, every component that is needed to make, you know, from the kernel to make a device operate is defined and described. In this case, we describe the components that make Hothub board to port package, you know, your kernel, your Pandavizer binary itself, which is your NDRD, right? Configuration for those components, as well as every container that is running on top of that, in the very basic use case, you only have one container, which is running one level up from the inner user namespace and would actually become your main userland or your main OS container. Now in parallel to that, you can add other applications, or it could just be a set of functions, you could have a container, which is your networking function, an application that takes care of screens and an application that takes care of power, right? I understand all of that is relatively high level, right, just to say it with one slide, but it is the type of world that we're trying to define. All of this is, as I said, it's a single project called Pandavizer. It is written in C. It is extremely lean and it has truly been made, well, by embedded engineers, for embedded engineers, right? So I invite you guys to take a look at what we're doing with this project. You can go to pandavizer.io to learn more about it, to try it on different devices. If you have any questions, want to talk to the team, have any suggestions, you want to change anything, or you want to tell us we're completely wrong in our approach, or not, please, you can go to community.pandavizer.io and share your thoughts. You'll find that just like you guys, we're embedded developers by trade and we would like to talk about this with everyone in this community. So thank you guys for taking the time to come to this chat. I hope some of you will stay afterwards for a bit of Q&A or just a friendly chat and see you there. Thank you very much. Let's kind of wrap a couple of them up into one question for you. So we had a question about your target memory, which you kind of addressed in the slides after the question was asked. So can you talk about target memory and why C is the solution versus Rust or TinyGo, which were also suggested in the comments here? Yes, absolutely. Let me actually mute the other time. Okay, there we go. Okay. So first, thanks everyone that actually joined the talk and thank you for your engagement. So I think there's two questions in that one, right? One of them is the size requirements themselves and the target size constraints, let's put it that way, resource constraints. And the other one is YC for that type of target size. So first and foremost, when we started this project, the main driver was to support containerization on Wi-Fi routers, set up boxes and that type of hardware, right? Which five years ago when we started, it was very average to find routers in the 16 to 32 megabytes of Nor Flash range, right? No Flash translation layer, no, nothing. Very static monolithic firmware. A lot of us here are used to that. But those vendors we're working with, they wanted to add horizontal feature extensibility and they kept getting pitched. And this is my second slide. They kept getting pitched by big cloud companies or some of bigger, higher class, let's say IoT vendors, use these high level solutions that just don't fit. And then the conversation was always, how do I get through your skull that 32 megabytes is too small? And the other ones would be like, it's fine, you can figure it out because they were sales people, right? So target original target was 32, 16 megabytes of Nor Flash. In reality right now, we're finding much more that the market is going, at least on the Wi-Fi set of boxes and so on, 64 megabytes of Nan Flash, 120 megabytes of Nan Flash, which is still on the low end. Because in reality, the base firmware, base OS that might be running in there, you know, might take 20, 30, something like that. There's 80, 70 more megabytes to play with to actually deploy dynamic services. So that's the baseline target sizes, let's say, right? And from memory point of view, anything over 64 megabytes of RAM, we'll get this working fine. And the wiser fits in about one megabyte of storage footprint. So that's the overall target. Now YC, and this is a question that Sigmund, I think, said many times. Sigmund, you're right, this could be written in Ros, it could be written in TinyGo. The choice of C was more because of the market dynamics of the deeply embedded ecosystem. And remember, sometimes people sort of put embedded as this sort of big umbrella term. But people that are talking about Raspberry Pi-class devices up, that's not embedded. That's very big, right? At least in my mind, in my world, right? But I'm talking embedded, I'm talking about these things that have no AM and C, no Flash translation, single core, 64 megabytes of NAND Flash, no more, right? The market dynamics of that lower end are a lot more complex, like engineers, developers, architects, or they're just used to working with C. And this is a reality, right? They're used to working with C, you know, all of their systems work with C. So that's the baseline of the statistician. Can there be ROS versions for this? Absolutely, right? So another question we got was from Thelman. Can you give me an idea about boot and startup times for your system as whole reaching app startup? Yeah. So we utilize several different things that help us reduce whatever impact we might have on the boot time. There is a minimal impact in the order of maybe 15 to 20 percent more of what your normal boot time would be. I mean, an exact seconds number depends on the board, depends on the system, depends on what we're trying to bring up, right? But on the baseline case of just having your normal, let's say, user land containerized as that sort of host OS container started through Pantelizer, the time hit might be in the 15 percent time frame, optimizable with certain things that we can do, right? Pretty well asked, how is administration, what interfaces do you use to deploy new services? Right, right. So Pantelizer itself, it exposes a local API, it exposes a local socket based API, Unix domain sockets that any container within the running system can leverage to control the life cycle of all the containers and applications that are running on that system. So in reality, anybody can write an agent that controls those APIs, right? Or they could consume directly the systems that we provide for these type of solutions at commercial grade, right? This basically means Pantelcore Hub, which is the management cloud for this, which we provide, could be used to use that one to one, or you could develop your own. A lot of our customers actually develop their own because they have existing management backplains, right? Maybe you answered this already, I'm not sure. How would this compare to system D and spawn? Right, so system D and spawn does not provide a big scaffolding for life cycle management, right? System D and spawn does a very similar thing in the sense that it starts up a container, you could say kind of, right? It's the primitives for a container. But the life cycle management is where it's really important for this type of unattended mission critical devices, right? So what we have combined is that the minimal container runtime and the life cycle management engine, both of those things come together to provide an entire product solution. That's what we try to bring to the market here. So Ruben asked, is there a method to describe which combinations of container proteins are compatible, i.e. tested? He's concerned about a risk, but go ahead. So this is a story longer than, you know, older than the Bible, let's say, right? How can you figure out a container across dependencies, right? The answer is there isn't a straightforward way, right? We just provide all of the metadata that is needed for you to understand what container is running and what version. What we have seen is that the CI-CD pipelines and processes, NQA processes of our customers, then incorporate that into their decision-making engines. What is deployed and stable release, candidate, canary testing channels and so on, right? As long as enough metadata is available, then you can keep a very comprehensive view of your deployment and the software that is coming from your pipeline, right? Which is what happens in the cloud, kind of. The cloud is not, this container depends on that other one, right? It's more about the DevOps practices behind it. So we have about 30 seconds left. I think we'll continue the conversation in the room if people want to keep asking questions. Thanks for all the great questions. Have you looked at the work done with containers on EZOS and C? So I hadn't looked at EZOS. I just saw it in the question before. I was trying to search for it, do it fair? So I'll look into it right now. Okay, fair enough.
 Hello everyone, my name is Ricardo Mendoza.  I come from a company called Pantacore and today I want to tell you why we think that  embedded Linux needs a container manager that is written in pure C.  First of course, who am I?  My nickname is Rick M.M.  I have been involved with Linux since the early 2000s.  There was a time, also early 2000s, when I got close to the MIPS architecture.  At some point I think because I got obsessed with porting Linux on a bunch of early hand  LPCs.  This is before smartphones.  These devices were quite cool and they used to be MIPS early on, some of them SH before  ARM came and took the whole pie.  Some of these MIPS and then later ARM work led to a drive for understanding low-level  OS concepts.  I ended up becoming a Gen2 developer as well for a bunch of years, maintaining a bunch  of MIPS hardware.  I no longer do that.  And then I worked at Canonical for five, six years, something like that, working not on  the main distro but actually on the device strategies and device side of things from Canonical.  For example, early day testing of Ubuntu and mobiles and tablets, which resulted in Ubuntu  Touch or Ubuntu and mobile as it was called back then.  Now it's called Ubuntu Touch thanks to the community effort that still maintains this,  which I'm also a part.  And after that I was part of the team that wrote about Ubuntu Core.  That was the end of my relationship with Canonical.  After that I left on the most part to find what we are doing now.  I am one of the founders at Pantacore since 2016.  So been working there for a bunch of years now.  And as a side note, I also sit on the board of the Ureports Foundation, which advances  Ubuntu Touch with a pure community effort.  Now a little disclaimer.  This session is not a tech deep dive.  We're not going to be running tutorials for you to follow up on your laptops.  But rather is a conversation provoking exercise.  The idea here is to discuss the topics at hand.  And of course, also at the end, there will be a shameless blog.  I will tell you a bit about the container management engine that I wrote for embedded.  Now the important thing here is not what is written or what did I write.  It's actually more why was something written in the way that I did and why do we believe  that a container management engine in pure sea is very important to the modernization  and the advancement of all the great work that this bunch of people here do.  At the end of the day, what we do powers the infrastructure of the world.  There is everything that people use, all the gadgets, all the things that makes people's  lives easier.  It's thanks to embedded Linux.  So first and foremost, whales are too big for embedded boards.  Why do I say that?  Well, it has been a huge push from the cloud, from cloud vendors, cloud people, cloud developers,  whatever you want to call it, about bringing the technologies that have made sense on the  cloud and how to fit them one to one in the world of embedded.  You know, this is kind of weird because there's a bunch of people that have not a lot of experience,  let's say, on what true embedded means, telling us how we should be using certain tools that  have made sense for cloud.  And that's why I have this image here, you know, a managerial decision and an issue that  happened at some point meant that these people had to solve a weird problem of putting a  square into a hole.  Over there it worked and they were able to bring their astronauts back, but that doesn't  mean that we should always follow that approach, square peg into a run hole type of thing.  So that's not to say that containers are not useful for embedded, on the contrary.  Containers have modernized cloud computing and we think it can do the same for embedded.  We're seeing it already.  We know the usual benefits of containers, efficient resource utilization, modularization  of software, portability, security, practices like CI CD pipelines become straightforward  when you have this type of modular software architectures.  And you know, Docker containers and OCI containers and the like have been perfect for the cloud  where resources are near limitless.  How does that work with most embedded Linux devices today?  Let's see.  Well, first the question is how is the cloud different from embedded Linux devices?  This might sound like an obvious answer to you guys, right?  But to a bunch of these people that are investing tens of millions of dollars in marketing budget  to tell us run Docker on your tiny device, I think it's not obvious, right?  So let's say it here.  A data center has near infinite resources, right?  A server is, you know, you won't lose power.  It has infinite connectivity.  It is resilient.  It has a technician next to it.  If something happens to it, right?  And it's not truly mission critical, right?  With the current type of cloud architecture, things are very ephemeral, right?  Something dies somewhere.  It will just spawn again somewhere else, right?  Infinite replication means that, you know, if infrastructure fails somewhere, it's not  a huge issue, right?  It is an issue.  Don't get me wrong, but it's not a huge one.  Now tell that to your mother's router, right?  Your Wi-Fi router in her home.  If it goes down, that's it.  No more Netflix, right?  That's to call the technician at the ISP.  That might involve a call.  You might be without connectivity for 24 hours.  That's mission critical in a user's home, right?  I'm not even going to get started with robots in a factory.  I mean, I'm pretty sure you can think about that one yourselves.  But, you know, for the common user, it's also mission critical, right?  If your thermostat fails, well, that's bad, right?  You freeze.  If your fridge fails, that's bad.  Your food goes bad, right?  So embedded is mission critical.  Embedded requires extreme resilience, right?  And usually the resources are extremely constrained.  So, you know, this is basically what we're saying.  We can agree that maybe your minimum specifications for an embedded Linux device look something  like this in this current day and age, January 2022.  This evolves.  Maybe last year I would have said 16 megabytes of Nor Flash.  Some people would be doing.  But for example, we can see a bunch of routers nowadays that are going to 128 megabytes of  NAND storage.  Some of them even have EMMC.  But still, most embedded devices aren't the lower range of this.  A lot of them we don't have a flash translation layer even, right?  Our usual architectures, we know them, ARM, MIPS, RISC-V.  And virtually, of course, I say any Linux distribution, but in reality what it means  is whatever goes, right?  Whatever you need to actually build your system.  You want to build something atop a yachto, open embedded, or whatever, go for it, right?  You want to use a Alignar reference build for some board.  Go for it, right?  Or let's say you want to make a router, most likely you're going to go the way of open  WRT, right?  So now, going back to the topic, right?  Why do we think that with this type of resources or why do we think that containers are important  for embedded, right?  Well, if we do away with the problem of do this cloud tools fit, right?  We remain behind with the benefits.  We say, well, modern software development and firmware lifecycle management at speed,  right?  Modernizing the way that we manage the software on an embedded device is important, right?  We have been doing it the same way for the past 20 years, 25 years plus, right?  Maybe it's time to change, right?  We're starting to see the same problems that the cloud was seeing six, seven years ago,  right, when servers and applications were just getting too huge, too complex to manage,  right?  So, what does that mean, right?  I mean, what do containers help us do over here, right?  In the embedded world that is starting to see that problem.  On the diagram on the left, you can see what I mean by that problem, right?  The embedded firm worms becoming this huge monolithic image, right?  That just incorporates all those features.  The bigger and the more complex that gets, the longer your release cycles are.  This prevents valuable and fast innovation, you know?  A monolithic release is time consuming to produce, it's error prone, right?  And you know, the two above points result in that some devices end up being very stale  out there in the field, right?  They're not updated.  They can be very insecure and vulnerable to attack, right?  This creates incredible surfaces of attack.  We have seen it many times.  You see some ISP in Germany a few couple years ago ended up having over a million routers  compromised, created a botnet that actually required, I think, a technician visit or sending  the box back or something like that.  Imagine the cost of that, right?  But you know, that's the sort of catastrophic scenario.  But on the normal day-to-day business, just those long release cycles that prevent innovation  prevent you from adding new features fast enough, right?  So we need to start doing what this diagram on the left is doing.  We need to start sort of releasing the pressure of that monolith into a feature-rich pipeline.  We need to start releasing so that it becomes modular applications that sit on top of a  smaller core, let's call it, right?  And to achieve this, well, the easiest way, of course, is with containers, right?  Let's not try to reinvent the wheel, right?  Let's just make sure that our wheel fits our use case, right?  However, that's the main point here.  Everybody is telling you use Docker, use OCI runtimes, base solutions.  But why?  You know, embedded wasn't built, sorry, Docker wasn't built for embedded devices.  Most of these OCI-compliant runtimes out there wasn't built with embedded in mind, right?  So we propose a different type of architecture that doesn't mean having a huge OS and then  on top of that, a container runtime and then on top of that, container applications, but  rather a shift in paradigm, right?  We're true proponents of something that we like to call the minimal container runtime,  right?  I think that the legacy, let's say, Docker type of architecture where you have your hardware,  or a host OS on top, a bloated host OS, then a Docker engine and then on top of that, your  business logic is just too much overhead, right?  It's too much for the majority of low spec embedded devices out there, right?  And, you know, embedded nowadays, it's about more features, but it still is a well-defined,  it still is about a well-defined feature set.  The operator, the device manufacturer, they want to extend the features, but they want  to be in control of those features, right?  So do we even need a main OS, a big host OS sitting below all of everything we just mentioned  as we know them in our true embedded use case, right?  Is that necessary or do we just need to modularize the actual software functions that are sitting  on this device, right?  And that's what we propose, right?  We propose doing away with the concept of a host OS.  Why should there be a host OS?  Then we just have a very tiny container runtime that sets up one too many user lands running  on top of it via containerization tooling and technologies of the kernel, right?  In the diagram on the right, you can see what I mean, right?  There's one that says host OS, but in reality, this is just another application.  It's just another user land that might start one too many services inside of a container  that provides some functionality.  For example, networking, whereas another one of these applications might provide power  control that's safe for a, I don't know, a fridge or something like that.  Now there's a little text here that says hint, in it user namespace, right?  I have always said that this is an evolution of, or rather an extension of what the kernel  does by default, right?  On your left hand model, that host OS is also running in a container for all intents and  purposes.  I mean, you can extrapolate a little bit from some concepts of the kernel, but your main  OS always runs into something called the init user namespace.  Oh, surprise, the kernel is actually running containers for everything.  So what we're doing here is extending that concept and saying, okay, why don't we use  that init user namespace, that main namespace just to provide a very minimal container runtime  and then run all of the actual relevant software functions on top of it.  This is interesting, right?  Because this lets us do what?  Let's us manage the entire lifecycle of every software component on the device in a very  straightforward manner.  It lets us decouple the lifecycle of all the higher level software that is running on that  device and the firmware that is running below that high level software, but through a common  protocol.  Now, if we have a minimal container runtime, well, we have to decide what to write it on,  right?  And this is where it gets interesting.  This is where you start talking about, or at least having the conversation as to, is  it what the cloud vendors are telling us or is it something else?  We think it's C, right?  So a minimal container runtime, YC, right?  Well, first of all, most of those cloud IoT proponents out there with their whole 3.7  seconds of experience with embedded have been employing a lot of web developers to solve  the challenges of embedded.  But these are web developers, right?  And they have been telling us how embedded should be done.  Okay, well, that's a bit of a brute force attempt, let's say, right?  But in reality, we have heard it is not uncommon for these people to tell you, manage your  embedded firmware with an OJS or GoLang Ridden Engine on your 32 megabytes of NOR flash,  or maybe 128 megabytes of NAND, come on, right?  We think that a minimal container runtime should be considered baseline infrastructure.  It is much closer to a role of the kernel than that to user land applications.  Yes, okay, of course, the details.  Let's go back to my conversation about in-user namespace.  We think that it should fit anywhere.  It should be as portable as possible.  So why break, why change what is in broken, right?  Let's write it in C after all.  Like C is pure C, and this was the true first container runtime, let's call it, NOR engine,  right?  So what have we done?  This is where we plug into our story, right?  And in Pandacore, we have built something called Pandavizer, right?  Which is an in-built and lightweight container manager, right?  We built it with the needs of embedded in mind at the center.  That's all we care about here.  We're talking about a container engine, a minimum initial RAM disk that, you know, an  overhead of no more than one megabyte, let's say, with strict build rules that allow you  to configure and change all of those things according to the requirements of your device.  It is specially suited to small, NAND or flash type devices, but of course it can scale the  way up to huge things like a Raspberry Pi, right?  It is highly portable.  It is built with C, using a pluggable runtime architecture.  And the main thing we use is actually LXC containers to run each one of the individual payloads.  Remember, Pandavizer is in charge of the lifecycle management of all of those containers, right?  We're not reinventing LXC.  We're just providing the right scaffolding around it in this particular case to manage  the lifecycle of your embedded devices.  And Pandavizer is a single purpose system.  It doesn't try to be a full user land or a full OS.  It doesn't let you configure your networking or figure out all the drivers for your screens  or whatever.  No, the only purpose is to orchestrate the lifecycle management of your device's user  land, which would be in that case that main OS, which is containerized here as well, as  well as other application containers, but also your BSP components, your kernel, your  firmware, your modules.  By providing a single way of defining your entire system, then you can modularly manage  each component of that system.  And it is fast, right?  This is just containers.  As we said, it's a one megabyte C program that runs before anything else.  And there is no true performance hit, right?  But you get all the benefits of modern lifecycle management.  So what does Pandavizer let you do?  It lets you convert, turn your monolithic embedded system into a set of portable and  reusable microservices, right?  Reuse the code across different projects, make it in a way that those dark boxes at  the bottom, BSP and hardware, can be interchanged while the things on the top can also be interchanged.  Pandavizer is the one that then has the knowledge of who are we running on and how can we bring  certain applications to run on top of this BSP, right?  This is very powerful because it lets your product teams, it lets developers, it lets  everyone actually decouple the lifecycle of those business logic units that are written  for these devices, for these embedded devices, and decoupled it from the lifecycle of the  firmware components, your BSP, your kernel, your modules, your actual static firmware,  right?  All of this is achieved with something that we call the Pandavizer system format, which  is a way that in a single file, every component that is needed to make, you know, from the  kernel to make a device operate is defined and described.  In this case, we describe the components that make Hothub board to port package, you know,  your kernel, your Pandavizer binary itself, which is your NDRD, right?  Configuration for those components, as well as every container that is running on top  of that, in the very basic use case, you only have one container, which is running one level  up from the inner user namespace and would actually become your main userland or your  main OS container.  Now in parallel to that, you can add other applications, or it could just be a set of  functions, you could have a container, which is your networking function, an application  that takes care of screens and an application that takes care of power, right?  I understand all of that is relatively high level, right, just to say it with one slide,  but it is the type of world that we're trying to define.  All of this is, as I said, it's a single project called Pandavizer.  It is written in C. It is extremely lean and it has truly been made, well, by embedded  engineers, for embedded engineers, right?  So I invite you guys to take a look at what we're doing with this project.  You can go to pandavizer.io to learn more about it, to try it on different devices.  If you have any questions, want to talk to the team, have any suggestions, you want to  change anything, or you want to tell us we're completely wrong in our approach, or not,  please, you can go to community.pandavizer.io and share your thoughts.  You'll find that just like you guys, we're embedded developers by trade and we would  like to talk about this with everyone in this community.  So thank you guys for taking the time to come to this chat.  I hope some of you will stay afterwards for a bit of Q&A or just a friendly chat and see  you there.  Thank you very much.  Let's kind of wrap a couple of them up into one question for you.  So we had a question about your target memory, which you kind of addressed in the slides  after the question was asked.  So can you talk about target memory and why C is the solution versus Rust or TinyGo, which  were also suggested in the comments here?  Yes, absolutely.  Let me actually mute the other time.  Okay, there we go.  Okay.  So first, thanks everyone that actually joined the talk and thank you for your engagement.  So I think there's two questions in that one, right?  One of them is the size requirements themselves and the target size constraints, let's put  it that way, resource constraints.  And the other one is YC for that type of target size.  So first and foremost, when we started this project, the main driver was to support containerization  on Wi-Fi routers, set up boxes and that type of hardware, right?  Which five years ago when we started, it was very average to find routers in the 16 to 32  megabytes of Nor Flash range, right?  No Flash translation layer, no, nothing.  Very static monolithic firmware.  A lot of us here are used to that.  But those vendors we're working with, they wanted to add horizontal feature extensibility  and they kept getting pitched.  And this is my second slide.  They kept getting pitched by big cloud companies or some of bigger, higher class, let's say  IoT vendors, use these high level solutions that just don't fit.  And then the conversation was always, how do I get through your skull that 32 megabytes  is too small?  And the other ones would be like, it's fine, you can figure it out because they were sales  people, right?  So target original target was 32, 16 megabytes of Nor Flash.  In reality right now, we're finding much more that the market is going, at least on the  Wi-Fi set of boxes and so on, 64 megabytes of Nan Flash, 120 megabytes of Nan Flash,  which is still on the low end.  Because in reality, the base firmware, base OS that might be running in there, you know,  might take 20, 30, something like that.  There's 80, 70 more megabytes to play with to actually deploy dynamic services.  So that's the baseline target sizes, let's say, right?  And from memory point of view, anything over 64 megabytes of RAM, we'll get this working  fine.  And the wiser fits in about one megabyte of storage footprint.  So that's the overall target.  Now YC, and this is a question that Sigmund, I think, said many times.  Sigmund, you're right, this could be written in Ros, it could be written in TinyGo.  The choice of C was more because of the market dynamics of the deeply embedded ecosystem.  And remember, sometimes people sort of put embedded as this sort of big umbrella term.  But people that are talking about Raspberry Pi-class devices up, that's not embedded.  That's very big, right?  At least in my mind, in my world, right?  But I'm talking embedded, I'm talking about these things that have no AM and C, no Flash  translation, single core, 64 megabytes of NAND Flash, no more, right?  The market dynamics of that lower end are a lot more complex, like engineers, developers,  architects, or they're just used to working with C. And this is a reality, right?  They're used to working with C, you know, all of their systems work with C. So that's  the baseline of the statistician.  Can there be ROS versions for this?  Absolutely, right?  So another question we got was from Thelman.  Can you give me an idea about boot and startup times for your system as whole reaching app  startup?  Yeah.  So we utilize several different things that help us reduce whatever impact we might have  on the boot time.  There is a minimal impact in the order of maybe 15 to 20 percent more of what your normal  boot time would be.  I mean, an exact seconds number depends on the board, depends on the system, depends  on what we're trying to bring up, right?  But on the baseline case of just having your normal, let's say, user land containerized  as that sort of host OS container started through Pantelizer, the time hit might be  in the 15 percent time frame, optimizable with certain things that we can do, right?  Pretty well asked, how is administration, what interfaces do you use to deploy new services?  Right, right.  So Pantelizer itself, it exposes a local API, it exposes a local socket based API, Unix  domain sockets that any container within the running system can leverage to control the  life cycle of all the containers and applications that are running on that system.  So in reality, anybody can write an agent that controls those APIs, right?  Or they could consume directly the systems that we provide for these type of solutions  at commercial grade, right?  This basically means Pantelcore Hub, which is the management cloud for this, which we  provide, could be used to use that one to one, or you could develop your own.  A lot of our customers actually develop their own because they have existing management  backplains, right?  Maybe you answered this already, I'm not sure.  How would this compare to system D and spawn?  Right, so system D and spawn does not provide a big scaffolding for life cycle management,  right?  System D and spawn does a very similar thing in the sense that it starts up a container,  you could say kind of, right?  It's the primitives for a container.  But the life cycle management is where it's really important for this type of unattended  mission critical devices, right?  So what we have combined is that the minimal container runtime and the life cycle management  engine, both of those things come together to provide an entire product solution.  That's what we try to bring to the market here.  So Ruben asked, is there a method to describe which combinations of container proteins are  compatible, i.e. tested?  He's concerned about a risk, but go ahead.  So this is a story longer than, you know, older than the Bible, let's say, right?  How can you figure out a container across dependencies, right?  The answer is there isn't a straightforward way, right?  We just provide all of the metadata that is needed for you to understand what container  is running and what version.  What we have seen is that the CI-CD pipelines and processes, NQA processes of our customers,  then incorporate that into their decision-making engines.  What is deployed and stable release, candidate, canary testing channels and so on, right?  As long as enough metadata is available, then you can keep a very comprehensive view of your  deployment and the software that is coming from your pipeline, right?  Which is what happens in the cloud, kind of.  The cloud is not, this container depends on that other one, right?  It's more about the DevOps practices behind it.  So we have about 30 seconds left.  I think we'll continue the conversation in the room if people want to keep asking questions.  Thanks for all the great questions.  Have you looked at the work done with containers on EZOS and C?  So I hadn't looked at EZOS.  I just saw it in the question before.  I was trying to search for it, do it fair?  So I'll look into it right now.  Okay, fair enough.
container_manager_in_c_0_143.png container_manager_in_c_144_491.png container_manager_in_c_492_4907.png container_manager_in_c_4908_6719.png container_manager_in_c_6720_7631.png container_manager_in_c_7632_10331.png container_manager_in_c_10332_12107.png container_manager_in_c_12108_12239.png container_manager_in_c_12240_12611.png container_manager_in_c_12612_12983.png container_manager_in_c_12984_13223.png container_manager_in_c_13224_16823.png container_manager_in_c_16824_21875.png container_manager_in_c_21876_23927.png container_manager_in_c_23928_26675.png container_manager_in_c_26676_28043.png container_manager_in_c_28044_30059.png container_manager_in_c_30060_30311.png container_manager_in_c_30312_31139.png container_manager_in_c_31140_31415.png container_manager_in_c_31416_32051.png container_manager_in_c_32052_33923.png container_manager_in_c_33924_35951.png container_manager_in_c_35952_36071.png container_manager_in_c_36072_36119.png container_manager_in_c_36120_36539.png container_manager_in_c_36540_37043.png container_manager_in_c_37044_37295.png container_manager_in_c_37296_38063.png container_manager_in_c_38064_38399.png container_manager_in_c_38400_38639.png container_manager_in_c_38640_39347.png container_manager_in_c_39348_39755.png container_manager_in_c_39756_39911.png container_manager_in_c_39912_40799.png container_manager_in_c_40800_41135.png container_manager_in_c_41136_42191.png container_manager_in_c_42192_42347.png container_manager_in_c_42348_42743.png container_manager_in_c_42744_42947.png container_manager_in_c_42948_42995.png container_manager_in_c_42996_43967.png container_manager_in_c_43968_44081.png