Tango with systemd
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hadjinlian, Maxime
Used by many major distributions, systemd is widely known in the desktop and server world. But it is not so common to find it in embedded product. In this talk, we will show how systemd can be a real benefit for the embedded world; for both your sanity and your time. We will discuss how systemd was integrated into Phantom, a speaker from Devialet, and what was the pro and cons of using it.
 All right, everybody. So I know it's Sunday, it's the end of the conference, we're all tired, but bear with me, this should be a fun talk, at least it was for me. So we are going to talk about Stengo with SystemD and the reason for this title is that as in every dance, such as Tango or any other, first you have to learn the moves to dance and at first you kind of step on each other's toe, you fumble and it's not going to go great, then you learn and then it's great and you kind of look like this. Let's first address the font of this slide, it's coming sense, I know, deal with it. So first we have to go back a little bit, the project I'm going to talk about starts in 2013 and we chose to use SystemD back then which was the version 207. So all of the features that I'm going to talk about are a little bit dated, but as I go along with the slides, we updated our version of SystemD even if we kept doing the same mistakes. At the time, SystemD was not really popular, there was still a lot of troll, a lot of heat in the community, I'm sure many of you remember that, it was not really a really great time. And especially you did not speak about SystemD embedded devices, you used PC box or SystemD, but not SystemD, it was huge, it was a lot of tools, it was unmaintainable, it was horrible. But yet we chose to use it. Was it madness? Maybe at the time, today, not so much. So that's my preface you can see, that's my name, complicated, my mail, much easier to remember. I work at this company which is called De Vialet, it's a French company, we make this product, it's a high-five speaker with a lot of connected services, you name it, so it runs Linux and SystemD. Let's tango. First dates. So as I said, we started with SystemD 207, so some of the features that I'm going to talk about were not implemented or not as useful as they are today. First, JournalD, so even back then, it's the way to store your log, it's the way to keep you all your logs with SystemD, but we did not use it. Worse, we really went against it. For example, in order to make it boot after a mount point, read write was created on our systems, we hacked the service and I do not mean we hacked it using a drop-ins like you should do, we hacked it like a patch in the code of SystemD. And as this system is a very basic one, it basically slowed our whole boot by order of seconds. We did not use the API either, in fact, we didn't even knew there was one. We even designed our own logger because we thought, why not? Time SyncD, a great tool we could have used, but why use something embedded with your own init system if you can use something else, right? So we used OpenNTPD. OpenNTPD is a very good piece of software. I'm not saying that it's not good. That's not what I'm talking about at all. But it has a few drawbacks, especially on an embedded device. When you don't have interfaces available for some reasons, OpenNTPD will take a lot of time before trying to sync again. And when your user is trying to listen to music, online music services, you don't want him to wait for 10 minutes, so OpenNTPD, we sync again. So we patched OpenNTPD. Your product starts, you don't have a DNS. Again OpenNTPD takes a lot of time before it syncs again. Again add a patch, fix the problems. We don't have an RTC in our product. So we created a small script shell, which basically is a daemon that every five seconds will touch a file in the system in some read-write partition. Ring a bell. Again, network D. So network D was introduced in mid-October 2013. At the time it wasn't really useful for us. So we did not use it. We rolled our own. As I said in the beginning of the talk, remember that as time goes by, we keep update our system D. I don't remember what version we were using at this time, but I'm pretty sure network D was more useful than when it was just introduced. But still, we did not use it. We rolled our own again. To try to detect the interfaces at boot, you had to use a shell script which basically was a while loop which was looking for sysclassnet, insert the name of my interface here for all the interfaces that we used. And once all the interfaces were detected by this script shell, our network manager could run. Because it was not dynamic, the interfaces, if one interface was not available, it would crash and then you were kind of screwed. You basically had to restart the whole process. So the name of the interfaces were hard-coded in the program. And so it was a lot of line of code. I think I looked at it before. It was 10K line of code because, well, you have to handle WPS applicants, host APD, if config, root, DHCP, server, clients, MDNS, whatever. So basically, that's us trying to use system D. We tried, but kind of missed the mark. So, what we did is that we took system D and we really beat him up until we shaped him into something that worked as we expected, not as you should work with it. So it was not a great experience at first, but honestly, it was our fault because we didn't really try to understand how it worked. To put this in some context, the whole project, the whole product, and by the product, I mean, the product, the test software, the manufacturing software, and everything that goes with the product, was all done by a team of five people. So we didn't really add the time. So I did not speak of this, but why only one in it when you can have two? Two is better than one, right? What do I mean by two in its systems? Basically we had system D run all our services and at the end of the boot, it would run another program, a demon, which from various inputs from users would start, stop some services or some target, which is exactly what system D do. And because to start services or targets, you have to be root, but you don't want your demon to be root because it's a security issue. And then we created some root launchers. What is a root launcher? Basically it's a small program that only takes a few arguments and only allow you to start specific binary with specific arguments. And it's set with set UID. Not so great. And because of all the things that I talked about, our boot was almost sequential, so we did not take any advantage of the parallel capabilities of system D. There was a lot of contention, so a really slow boot, a lot of code from that has to be maintained by us, tested, debugged. And many, many, many more errors. It would last all day if I would list them all. Here is an idea of the boot chart that we had on our product. As you can see, there is a few really big red lines, but not really great. Then we had some epiphany. Yeah. Much better than the second time around, especially when you have time. So, journal D. Yes, you should use it. There's an API for that. You can store blob. There is compression. It's easy to add context. You can have some fields. It's really easy to insert into your daemon. I mean, just use it. It's easy. So as I said, binary lock format, great. It's great to compress. It's easy to send to a remote server. Nice to use. It's also on all the core dumps, which is kind of nice. We don't really use it because we have some very specific needs regarding the core dump. So we have a small strip shell, but it's only 20 lines. Not so bad. And also, you have to learn to use the command line. I would say that journal CTL command line and many system D tools to me kind of remind me of Git. When you first start, really hard to understand. Once you get the hang of it, it's great. You can do great things with it. Time seeing D. You should use it too. It's nice. It works along with network D, which is really great because when you lose an interface or it disconnects, reconnects, you won't lose track of the time. It will try to re-sync when the new interface arrives, if there's a route to Internet or anything. Nice. Maintains a clock file for you as a fake RTC. Nice. You can get rid of this ugly shell script. It works even if you don't have DNS. There's this bug with ETC resolve conf that if it's empty and your Gmin is starting, and it uses the libc, basically the libc will never reopen ETC resolve conf, even if it changes. So you're kind of stuck. That's also one of the reasons that we use system D resolve D. I'm not talking about this here, but it's also a really great tool even for embedded devices. And yeah, work with DNS because it's hard-cutting the source that you can use the Google DNS. You can change it at compile time if you want. If you don't want to use Google DNS. It's also pretty fast because instead of implementing the whole NTP protocol, it will only implement the SNTP protocol. Network D. So actually we are using version 234. So we are pretty up to date looking forward to use 235. Again, good tool. Dynamically detect interfaces much better than what we had. Using network files or link files, you can set up whatever you want on your interfaces, NTU, the MAC addresses, you name it. It's really good for simple network like you could have on your laptop or even maybe your server. And even not so simple network like you can have on such a product because as I showed you, we create a speaker, but you can have many speakers in a room. So it's a distributed system that runs. So the network configuration can be kind of tricky. One of the drawbacks is that there is no Wi-Fi AP support as far as I know. So we still had to wrote some code to handle WPS and host APD. So in conclusion, I would say that this time around we have seen some real gain from the boot time because finally it was paralyzed. We removed most of our crappy code and so now we have maintained a test code base and something that is documented that we can contribute to, it's much better for everyone. And also, it's not our job to make an in-system. Our job is to make software that creates features for our users so if we can use system D, great. And there are one of the features that we really, really overlooked which are really awesome for us because it enables us to create our system in the manner as of an OEM, if you would. Because if we were to say license our system, it's easy to do. We can just give our kind of distro to a vendor and it would just have to put some files into the directory that it wants, some drop-ins of its own. So to troubleshoot, there is a lot of tool added by system D. There is the system D delta, there is the cat which is pretty useful, analyze, really useful too, and many more. Debuss interface also, use it, it's good. Finally we can have a stateless system. Instead of having a system which would evolve over time, which is not really what you want in an embedded system, each time your users boot your product, you want it in a pristine state so that you know that it is what you designed, not something that's wrought over time or you don't really know what state it's in. Also I've heard a lot of time that system D was really huge and so you cannot really use it on embedded devices. Maybe it's true if you have really, really tiny storage requirements, but nowadays it's getting pretty rare. If you're using EMMC or whatever, they are pretty, pretty large. Also you have to remember that even if system V is maybe small, there's a lot of dependencies on a lot of tools. You can really, really chop system D down with a lot of compiled flags. Just have to look them up. To build our distro we are using build routes and it exposes a lot of flags to disable a lot of features in system D so you can easily get to a system D that I think the whole system D package and so all of the tools are around 5 megabytes. So not so bad. Great support for read only root file system. There's the watchdogs also that are great. TMP files instead of doing some overlay hacks that we did. What we did in the first version of our system is that we would rsync files over the root FS once it was created in order to replace some services or some configuration file. Not good. Do not do this at home. TMP files, great. We also are using dynamic users. Also great feature. Previously we had many, many users that were not really used or they were used only once and then not dropped because they were in the system. With dynamic users you can avoid that. A lot of security capabilities that we now use protect system, protect home. You can define capabilities for your daemon. So you can use C groups. You can define RTPrio. One thing that is really great. You don't even need to modify files that are in slash sleep slash system D. You can put them in run. So in the end system D really was the right choice for us because it allowed us to really focus on the features that our user wanted and not something that is not our job. We gained a lot in boot time. We gained in security because of all the security features that are available in system D and really easy to implement. It's easier for us to manage a single firmware for multiple products because I only show you one product but we have others because you can use some drop-ins or two of the right things. You don't have to do some ugly patches. And there is many, many, many more features available in system D that surely we don't know about yet. Now we need something from system D. Now we really try to look it up in the manual or in some forum IRC or anything because usually it already exists. And also I have kind of an habit to read the news file now. There's a lot of gems that are available in there that are not widely discussed. So you maybe won't know them if you don't look. So yeah, we use system D now for what it's good at which is booting and managing the system and we do what we are good at which is audio speakers. And I don't know why it doesn't work. Yeah. Thanks. If you have any question? Nope. Nobody? Ah, here's one. Do you have the time and diagram for booting with system D on the second tense? I did not tweak it. I can give you an estimated time. First version was around 40 seconds. And right now we are at 18. Pretty cool. And still we haven't really, there is a lot of stuff that we can do to optimize. So I'm fairly confident that we could go way quicker. Anything else? Nope. No one? Really? You're sleeping? Wake up. All right. That's all for me. Thank you. Thank you.
 All right, everybody.  So I know it's Sunday, it's the end of the conference, we're all tired, but bear with  me, this should be a fun talk, at least it was for me.  So we are going to talk about Stengo with SystemD and the reason for this title is that  as in every dance, such as Tango or any other, first you have to learn the moves to dance  and at first you kind of step on each other's toe, you fumble and it's not going to go great,  then you learn and then it's great and you kind of look like this.  Let's first address the font of this slide, it's coming sense, I know, deal with it.  So first we have to go back a little bit, the project I'm going to talk about starts  in 2013 and we chose to use SystemD back then which was the version 207.  So all of the features that I'm going to talk about are a little bit dated, but as I go  along with the slides, we updated our version of SystemD even if we kept doing the same  mistakes.  At the time, SystemD was not really popular, there was still a lot of troll, a lot of heat  in the community, I'm sure many of you remember that, it was not really a really great time.  And especially you did not speak about SystemD embedded devices, you used PC box or System D, but not SystemD, it was huge, it was a lot of tools, it was unmaintainable, it was horrible.  But yet we chose to use it.  Was it madness?  Maybe at the time, today, not so much.  So that's my preface you can see, that's my name, complicated, my mail, much easier  to remember.  I work at this company which is called De Vialet, it's a French company, we make this  product, it's a high-five speaker with a lot of connected services, you name it, so it  runs Linux and SystemD.  Let's tango.  First dates.  So as I said, we started with SystemD 207, so some of the features that I'm going to talk  about were not implemented or not as useful as they are today.  First, JournalD, so even back then, it's the way to store your log, it's the way to keep  you all your logs with SystemD, but we did not use it.  Worse, we really went against it.  For example, in order to make it boot after a mount point, read write was created on our  systems, we hacked the service and I do not mean we hacked it using a drop-ins like you  should do, we hacked it like a patch in the code of SystemD.  And as this system is a very basic one, it basically slowed our whole boot by order of  seconds.  We did not use the API either, in fact, we didn't even knew there was one.  We even designed our own logger because we thought, why not?  Time SyncD, a great tool we could have used, but why use something embedded with your own  init system if you can use something else, right?  So we used OpenNTPD.  OpenNTPD is a very good piece of software.  I'm not saying that it's not good.  That's not what I'm talking about at all.  But it has a few drawbacks, especially on an embedded device.  When you don't have interfaces available for some reasons, OpenNTPD will take a lot of  time before trying to sync again.  And when your user is trying to listen to music, online music services, you don't want  him to wait for 10 minutes, so OpenNTPD, we sync again.  So we patched OpenNTPD.  Your product starts, you don't have a DNS.  Again OpenNTPD takes a lot of time before it syncs again.  Again add a patch, fix the problems.  We don't have an RTC in our product.  So we created a small script shell, which basically is a daemon that every five seconds  will touch a file in the system in some read-write partition.  Ring a bell.  Again, network D. So network D was introduced in mid-October 2013.  At the time it wasn't really useful for us.  So we did not use it.  We rolled our own.  As I said in the beginning of the talk, remember that as time goes by, we keep update our system  D. I don't remember what version we were using at this time, but I'm pretty sure network  D was more useful than when it was just introduced.  But still, we did not use it.  We rolled our own again.  To try to detect the interfaces at boot, you had to use a shell script which basically  was a while loop which was looking for sysclassnet, insert the name of my interface here for all  the interfaces that we used.  And once all the interfaces were detected by this script shell, our network manager could  run.  Because it was not dynamic, the interfaces, if one interface was not available, it would  crash and then you were kind of screwed.  You basically had to restart the whole process.  So the name of the interfaces were hard-coded in the program.  And so it was a lot of line of code.  I think I looked at it before.  It was 10K line of code because, well, you have to handle WPS applicants, host APD, if  config, root, DHCP, server, clients, MDNS, whatever.  So basically, that's us trying to use system D. We tried, but kind of missed the mark.  So, what we did is that we took system D and we really beat him up until we shaped him  into something that worked as we expected, not as you should work with it.  So it was not a great experience at first, but honestly, it was our fault because we  didn't really try to understand how it worked.  To put this in some context, the whole project, the whole product, and by the product, I mean,  the product, the test software, the manufacturing software, and everything that goes with the  product, was all done by a team of five people.  So we didn't really add the time.  So I did not speak of this, but why only one in it when you can have two?  Two is better than one, right?  What do I mean by two in its systems?  Basically we had system D run all our services and at the end of the boot, it would run another  program, a demon, which from various inputs from users would start, stop some services  or some target, which is exactly what system D do.  And because to start services or targets, you have to be root, but you don't want your  demon to be root because it's a security issue.  And then we created some root launchers.  What is a root launcher?  Basically it's a small program that only takes a few arguments and only allow you to  start specific binary with specific arguments.  And it's set with set UID.  Not so great.  And because of all the things that I talked about, our boot was almost sequential, so  we did not take any advantage of the parallel capabilities of system D.  There was a lot of contention, so a really slow boot, a lot of code from that has to  be maintained by us, tested, debugged.  And many, many, many more errors.  It would last all day if I would list them all.  Here is an idea of the boot chart that we had on our product.  As you can see, there is a few really big red lines, but not really great.  Then we had some epiphany.  Yeah.  Much better than the second time around, especially when you have time.  So, journal D. Yes, you should use it.  There's an API for that.  You can store blob.  There is compression.  It's easy to add context.  You can have some fields.  It's really easy to insert into your daemon.  I mean, just use it.  It's easy.  So as I said, binary lock format, great.  It's great to compress.  It's easy to send to a remote server.  Nice to use.  It's also on all the core dumps, which is kind of nice.  We don't really use it because we have some very specific needs regarding the core dump.  So we have a small strip shell, but it's only 20 lines.  Not so bad.  And also, you have to learn to use the command line.  I would say that journal CTL command line and many system D tools to me kind of remind  me of Git.  When you first start, really hard to understand.  Once you get the hang of it, it's great.  You can do great things with it.  Time seeing D. You should use it too.  It's nice.  It works along with network D, which is really great because when you lose an interface or  it disconnects, reconnects, you won't lose track of the time.  It will try to re-sync when the new interface arrives, if there's a route to Internet or  anything.  Nice.  Maintains a clock file for you as a fake RTC.  Nice.  You can get rid of this ugly shell script.  It works even if you don't have DNS.  There's this bug with ETC resolve conf that if it's empty and your Gmin is starting, and  it uses the libc, basically the libc will never reopen ETC resolve conf, even if it changes.  So you're kind of stuck.  That's also one of the reasons that we use system D resolve D. I'm not talking about this  here, but it's also a really great tool even for embedded devices.  And yeah, work with DNS because it's hard-cutting the source that you can use the Google DNS.  You can change it at compile time if you want.  If you don't want to use Google DNS.  It's also pretty fast because instead of implementing the whole NTP protocol, it will only implement  the SNTP protocol.  Network D. So actually we are using version 234.  So we are pretty up to date looking forward to use 235.  Again, good tool.  Dynamically detect interfaces much better than what we had.  Using network files or link files, you can set up whatever you want on your interfaces,  NTU, the MAC addresses, you name it.  It's really good for simple network like you could have on your laptop or even maybe your  server.  And even not so simple network like you can have on such a product because as I showed  you, we create a speaker, but you can have many speakers in a room.  So it's a distributed system that runs.  So the network configuration can be kind of tricky.  One of the drawbacks is that there is no Wi-Fi AP support as far as I know.  So we still had to wrote some code to handle WPS and host APD.  So in conclusion, I would say that this time around we have seen some real gain from the  boot time because finally it was paralyzed.  We removed most of our crappy code and so now we have maintained a test code base and  something that is documented that we can contribute to, it's much better for everyone.  And also, it's not our job to make an in-system.  Our job is to make software that creates features for our users so if we can use system D, great.  And there are one of the features that we really, really overlooked which are really awesome  for us because it enables us to create our system in the manner as of an OEM, if you  would.  Because if we were to say license our system, it's easy to do.  We can just give our kind of distro to a vendor and it would just have to put some files into  the directory that it wants, some drop-ins of its own.  So to troubleshoot, there is a lot of tool added by system D. There is the system D delta,  there is the cat which is pretty useful, analyze, really useful too, and many more.  Debuss interface also, use it, it's good.  Finally we can have a stateless system.  Instead of having a system which would evolve over time, which is not really what you want  in an embedded system, each time your users boot your product, you want it in a pristine  state so that you know that it is what you designed, not something that's wrought over  time or you don't really know what state it's in.  Also I've heard a lot of time that system D was really huge and so you cannot really  use it on embedded devices.  Maybe it's true if you have really, really tiny storage requirements, but nowadays it's  getting pretty rare.  If you're using EMMC or whatever, they are pretty, pretty large.  Also you have to remember that even if system V is maybe small, there's a lot of dependencies  on a lot of tools.  You can really, really chop system D down with a lot of compiled flags.  Just have to look them up.  To build our distro we are using build routes and it exposes a lot of flags to disable a  lot of features in system D so you can easily get to a system D that I think the whole system  D package and so all of the tools are around 5 megabytes.  So not so bad.  Great support for read only root file system.  There's the watchdogs also that are great.  TMP files instead of doing some overlay hacks that we did.  What we did in the first version of our system is that we would rsync files over the root  FS once it was created in order to replace some services or some configuration file.  Not good.  Do not do this at home.  TMP files, great.  We also are using dynamic users.  Also great feature.  Previously we had many, many users that were not really used or they were used only once  and then not dropped because they were in the system.  With dynamic users you can avoid that.  A lot of security capabilities that we now use protect system, protect home.  You can define capabilities for your daemon.  So you can use C groups.  You can define RTPrio.  One thing that is really great.  You don't even need to modify files that are in slash sleep slash system D. You can put  them in run.  So in the end system D really was the right choice for us because it allowed us to really  focus on the features that our user wanted and not something that is not our job.  We gained a lot in boot time.  We gained in security because of all the security features that are available in system D and  really easy to implement.  It's easier for us to manage a single firmware for multiple products because I only show  you one product but we have others because you can use some drop-ins or two of the right  things.  You don't have to do some ugly patches.  And there is many, many, many more features available in system D that surely we don't  know about yet.  Now we need something from system D. Now we really try to look it up in the manual or  in some forum IRC or anything because usually it already exists.  And also I have kind of an habit to read the news file now.  There's a lot of gems that are available in there that are not widely discussed.  So you maybe won't know them if you don't look.  So yeah, we use system D now for what it's good at which is booting and managing the  system and we do what we are good at which is audio speakers.  And I don't know why it doesn't work.  Yeah.  Thanks.  If you have any question?  Nope.  Nobody?  Ah, here's one.  Do you have the time and diagram for booting with system D on the second tense?  I did not tweak it.  I can give you an estimated time.  First version was around 40 seconds.  And right now we are at 18.  Pretty cool.  And still we haven't really, there is a lot of stuff that we can do to optimize.  So I'm fairly confident that we could go way quicker.  Anything else?  Nope.  No one?  Really?  You're sleeping?  Wake up.  All right.  That's all for me.  Thank you.  Thank you.
2017_Tango_with_systemd_-3kOmsp74_0_0_155.png 2017_Tango_with_systemd_-3kOmsp74_0_156_995.png 2017_Tango_with_systemd_-3kOmsp74_0_996_1283.png 2017_Tango_with_systemd_-3kOmsp74_0_1284_2243.png 2017_Tango_with_systemd_-3kOmsp74_0_2244_2783.png 2017_Tango_with_systemd_-3kOmsp74_0_2784_2915.png 2017_Tango_with_systemd_-3kOmsp74_0_2916_3455.png 2017_Tango_with_systemd_-3kOmsp74_0_3456_3503.png 2017_Tango_with_systemd_-3kOmsp74_0_3504_4007.png 2017_Tango_with_systemd_-3kOmsp74_0_4008_14759.png 2017_Tango_with_systemd_-3kOmsp74_0_14760_15143.png 2017_Tango_with_systemd_-3kOmsp74_0_15144_15251.png 2017_Tango_with_systemd_-3kOmsp74_0_15252_15311.png 2017_Tango_with_systemd_-3kOmsp74_0_15312_17303.png 2017_Tango_with_systemd_-3kOmsp74_0_17304_17399.png 2017_Tango_with_systemd_-3kOmsp74_0_17400_21491.png 2017_Tango_with_systemd_-3kOmsp74_0_21492_21575.png 2017_Tango_with_systemd_-3kOmsp74_0_21576_28715.png 2017_Tango_with_systemd_-3kOmsp74_0_28716_30875.png 2017_Tango_with_systemd_-3kOmsp74_0_30876_31103.png 2017_Tango_with_systemd_-3kOmsp74_0_31104_31163.png 2017_Tango_with_systemd_-3kOmsp74_0_31164_31271.png 2017_Tango_with_systemd_-3kOmsp74_0_31272_31319.png 2017_Tango_with_systemd_-3kOmsp74_0_31320_31427.png 2017_Tango_with_systemd_-3kOmsp74_0_31428_31499.png 2017_Tango_with_systemd_-3kOmsp74_0_31500_31607.png 2017_Tango_with_systemd_-3kOmsp74_0_31608_31883.png 2017_Tango_with_systemd_-3kOmsp74_0_31884_33275.png 2017_Tango_with_systemd_-3kOmsp74_0_33276_33323.png 2017_Tango_with_systemd_-3kOmsp74_0_33324_33383.png 2017_Tango_with_systemd_-3kOmsp74_0_33384_33424.png