Incremental package builds
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Maudoux, Guillaume (layus)
The current CI infrastructure around nixpkgs has difficulties to build the whole package set in due time. While there were attempts to build every pull requests, these are not enabled by default because it requires even more resources. We will explore how to optimize even more the build process to meet the size of nixpkgs. We will survey with state-of-the-art build systems like scons, bazel and tup to see how we could improve incremental builds in nix. Based on nix-make, we will describe how nix could cooperate with the build tool of a given package to bring incremental builds at a finer granularity. In particular, we will explore how the intensional store model (as used by bazel for example) could help avoiding some mass rebuilds.
 Next speaker is Guillaume Modoux, your sonorema's Leius on GitHub. I started using Nix at Mozilla a few years ago. Now he's working on a PhD on incremental builds. Yes. Thank you. Give it up to Guillaume. Okay, so let's start with a small background on me. So I do my PhD in Louvain-Leneuve near Brussels. And just for the story, Louvain-Leneuve has a huge biking event, which is also the second biggest drinking event in Europe after the one in Munich. So you know everything about my username. Okay, it's like a pun with a French word. But yeah, it's not very good as an introduction to this presentation. So please mind, there is one letter different, and that's very important. Right? I contribute to Nix packages, of course. And my GitHub picture is a stone like that. This stone is a stone from that castle, which I also contributed to build. So now you already know that I like building stuff, right? So I started my PhD on incremental builds, and I want to feed back on this presentation. So this is very important to me. Do not, is it to interrupt the presentation if something, if you want to ask a question in the meantime? So let's start the presentation. Incremental package builds. What is incremental? I went to the dictionary, which is always a good way to start, except that in this case it doesn't mean anything. Incremental is something that occurs in increments. Okay, come on. So what is increments? It's the amount of debris by which something changes. Okay, we cannot do anything with the definition, so I just came with mine. An incremental build system is something that works with very small steps. We can do each of them separately and work with them. We can use old build products, and we use them so we do not have to do things that are already done. And we also, of course, need the ability to detect what needs to be done. So this is the plan of this presentation. I will take three examples from real life. One with Firefox, one with i3, the window manager, and one with the store. The Firefox example shows why it's important to have small steps. If you ever try to build Firefox, then the build phase takes one hour, and everything else is basically negligible compared to the huge build time. I was playing with that package, and I had to change the fix-up phase. And of course, I made a typo in the bash script, and then, whoop, everything was lost. Okay, so we really need checkpoints. I mean, it's just like when you are doing rock climbing, you are climbing, okay, and then at some point, there is a problem, and then if there is no checkpoint, then, wow, you are back 2.0, right? So we really need that kind of stuff. It's not to lose everything all the time. So how can we do that? A very simple idea is just, okay, we already have parts in every build. We have different phases. So we could split them in different packages so that we, once one phase is finished, I mean, it's done. We can work with the next one. Yeah, but you know, it doesn't always work. For Firefox, it's good, but if the failure occurs in the build phase, then basically, you have still the same problem, that you lose one hour of build phase. It's not good for the store, because you will have every phase in the store. What does it mean? Can that even work? But this idea has already been implemented in some places with wrappers. Firefox, for example, has an external wrapper, which is another package that just writes the wrapper. That way, you can change the wrapper easily without changing Firefox all the time. So incremental builds are about small steps. And in fact, Nix is incremental in some way. It's just that it's incremental at the level of package. And the small step for Nix is a package. At the other hand, we have build systems, they already manage incremental builds of package, like Firefox has its own build system, and it's also incremental, but we are not able to use that now in Nix, because we want things to be very pure, so there is no interaction with the external world. So the idea here is to use existing build systems and to make them interoperate with Nix, so that we can achieve incremental builds within Nix. First build system that everyone knows is make. And well, I will not speak too much about make because it works, but there is nothing you can do that's correct or in the sense that we like it. Our make will never be pure. That's not in the design. We use timestamps and we do not want timestamps in the store, for example. It requires to have the previous builds in the same tree as filing the tree, so you need to add that as an input to the package and everything gets ugly. But there are other tools that are quite interesting, like C-Cache. We know C-Cache. Not that much people. Okay. So C-Cache memorize compiler invocations, so you can wrap a compiler code with that and if the same compiler code has been made before, it will fetch the result from the cache. Even better, Mozilla developed the shared C-Cache, where this cache of previously built stuff is shared on the network, so you can reuse a build that someone else did. And well, this is a very good and robust design because it's the design behind NICS. There is a binary cache that you can use and that we are sharing and this is used to replace building of some packages if they were already built before. So caching is a very good idea that we will reuse later. But I want to show a small project by Elko, which is NICS make. And NICS make basically allows you to, it's a build system. At the moment, it only works with very simple stuff like that. You see that it's 15 commits, okay, so it's not very active, but you can still do things like that. There is one derivation, compiler C produces derivation and derivation just generates a.o file and then there is the link derivation that just takes all the o file and produces the binary. So we can use NICS to do incremental compilation at a final level than just packages, but, okay, you can do very small steps. It's compatible with NICS, of course, but it's even worse than before. Every single.o file ends up in the store and you also need to port every project to NICS make. Okay, so if you're using whatever before like Bazel, then you need to write another layer to port it to NICS make. This is not feasible. Speaking about Bazel, this is an amazing, a tremendous project from Google. They are very proud of their stuff, fast or correct, pick two. And it's a project that, as most of things write, they manage to build every command. They can basically cache every command and put that in a shared cache. They can even ask another machine to compile that step and then get the results. So this is something that we would like to have. It's also, it's really, really correct. There is sandboxing going in there. But you know, it's so good that it does exactly the same stuff as NICS. So we are, when you try to make both of them work together, they're fighting to take the control, be the one that manages sandbox and do not let the others do other stuff. So it's quite difficult to use it. In fact, it's used at Google and at Google, everything uses Bazel. So it's quite easy. But with NICS, NICS packages, we have something that's much more heterogeneous. I mean, there are packages with make, packages with SBT, packages with CMake, and we cannot use just one build system to rule them all. So is it possible to do something intermediate? And my solution is, okay, we want caching. We want to be able to cache smaller steps than just packages. Can we somehow allow the build system to access an external cache while remaining correct? And to do that, we may need the help of NICS build to provide the same interface. But in the first attempt, if you do not care too much about correctness, then it's very easy to implement. Basically, you build not so pure, allow access to an external store, and allow, for example, ccache to access that store. It looks like that. You call make. I mean, NICS builds Starless Sandbox, makes, works in there, calls ccache, for example. And then ccache grabs in the environment a socket to discuss with the external store. As simple as that. If you trust ccache to do things correctly, then you will have correct builds. Of course, you cannot always trust ccache, so we will have to implement some checking on top of it. But this is the basic idea. And if you start with something simple like that, then you can port one package at a time, try to make it work, start maybe with big package like Firefox. And yes, this is my master plan. This is something that I want to do for my PhD. The ability to use incremental builds from the build system within NICS and to be able to go both directions to make them cooperate. So at the end, we should be able to rule over the world, right, with the system. But it still works. It still works in progress. Okay. Any question about that, maybe, before the next part? I mean, you can see us question afterwards. Yes? Correctness part is the interesting one, right? If you have a problem in caching normally, you have a problem in the end problem. Does it, do you have plans to work like tension list or stuff where you check output? Do you have like one or two words how you are approaching problems? Okay. I can say two things. Yes, the intentional store would help. I will speak about that later. And what can I say? Bazel already does something like that. Much more advanced. That was C-Cache does. Except that he does a lot of other stuff that we do not want. So we may need to import that part of the code. And then, you know, if it works for them, it should work for us too, right? If you want more correctness than what they achieve, which is already quite high, then we may still work on it, implement it. The interface here may not be just access to files. It may be, okay, I want to build, to call that command with these files, with these environments. And then we let nix build, run the commands and produce a cache result. So if everything is in the hands of nix build, then it should be clean. Okay. But this is more strict and this needs more, I mean, it needs to change a bit of C-Cache works at the moment. So the more correctness you want and the more work you have to do, of course. Okay, I was just wondering if that is very result instead of putting them in plain. Okay, so the idea about this stuff is that this is not defined. This may be the next store. This may not be. It depends on what you want to do. I mean, this is not the first time that I did something like that. There are a lot of you that want to have hierarchical nix stores. When you have one nix store for the packages, then one nix store for other stuff that are less important, that can be a rich collected morphine, and maybe another nix store for custom projects. I'm not sure that this idea of nix store is the perfect way of doing caching. The idea of nix store is to have, I mean, you need a directory because you need to access the files, but it may be more efficient to cache compressed versions of the files, for example. So you may not want to, you may not want exactly a nix store, but you need some way to cache stuff. Okay, let's continue anyway. You can ask questions later. So another use case I had, I had a problem with itry, which is my window manager. And being the window manager is something that you need all the time. But to reproduce the bug, I had to use it on my main laptop because basically the bug happened when I undocked it and dogged it again. So I mean, this was the only way I found to reproduce that bug. And so to test new versions of I3, I had to propagate them to nix stores configuration. I mean, you all already had something like that, so you want to test something and it's very complex because it needs to go into the nix stores configuration and how do I do that? My typical debug session looks like that. I use nix shell. I try to patch, see if it compiles, et cetera. If I can do that, then I extract a diff and then I generate an overlay to build a new package. And I check that it builds with nix builds. Then I insert that package in nix OS and just rebuild nix OS and then I restart, I test, and if it works, I mean, that's fine. If I can reproduce the bug, it's not that good. And if it crashed everything, then I need to roll back, of course. And then back again because it's a loop. I mean, when you are trying to debug something, you have to insert debug statements, then maybe it starts gdb, stuff like that, so it's quite complex. And it would be so simple if I could just compile it three and say, okay, put it in the store and use that. I know it's not clean, but then it would be easy, right? And there are solutions to do that. I should maybe not say that. Maybe we need to cut the video at some point. But it is possible to mount the nix store with rights. But yeah, okay, you should not do that. More clean but very hacky solution is to insert a sim link into the nix store. Make a fake derivation that's just a sim link to somewhere else, your own directory or your project directory. Then there are some technicalities with, you know, self-loops and stuff like that, but it's possible. And yeah, you can go straight to the solution and say, okay, this service use this package, okay, I can override the service, the nix store service and say, okay, use the package in my home environment. That works too. But then, I mean, it's not what we want. We are used to something that's pure, that's clean, and then we are trying to hack everything. So this is indeed very hacky. So I went to look for tools that allow to do that. And one of them is Git-rebase interactive. When you do Git-rebase interactive, it tries to apply all the commits in turn. And when there is a conflict, you are dropped in an environment that looks like it's a normal Git repo, but it's not really. If you try to push that on a branch, it will be strange, et cetera. But you can commit, behave like you are in normal commit environment. And when you're done, you say just continue, and you just continue this job integrating your modification. That's something that we would like with nix shell. Basically nix shell set up the right environment to build the stuff. You can build it. You can go to the end, but when you are doing make install, it doesn't install because you cannot access the nix store. Why? It would be interesting to be able to write in the nix store at that point and produce a package. And if you are annoyed about purity, then just make nix shell accept the hack option. And the hack option produce a derivation with a random input parameter. So it doesn't conflict with the normal one, the pure one, and this random input says, OK, it represents the fact that a human can just tamper with the build, so the human can add random stuff into the build. But that would be very, very nice because then we could insert basically anything in the store in a way that's quite clean and corresponds to the nix model. To explain that differently, I have defined the nix OSI model, which is not to be taken too seriously, OK. But we have layers, and each of these layers are related to different tools. So if you use nix builds, then you know nothing about nix source, basically. If you use nix source, then you know nothing about different servers like nix-obs do. And if you use make, then you know nothing about everything that's in nix. Nowadays, it's nearly impossible to just run make on a project. You need a nix shell to get the right environments. So basically what you do is you start at the nix shell level, you open the zipper, OK, and then you enter an environment where you can just play with i3 and your project. When you're done, you cannot use that package, but it would be nice if you could just close the zipper and say, OK, I have a package now. I enter it, and I can exit it. Of course, you can go further and say, OK, let's add a hack command to nix-rebuild, to nix-source-rebuild. And then you would start higher in the hierarchy, zip inside it, and then go outside of the store. So it would drop me, for example, in a shell where I can just edit i3, and then it would use that version of i3 only for that build of nix-source. OK, it's not that clean, but in the end, it's correct because it's a version with a random input which corresponds to the fact that they've changed something. And I can easily enter and get back off the tree. OK. Now for the extra step, if you remember caching, then you can still use caching in that. If what I did in the nix shell corresponds to what will be done by the nix-build, then there is no reason nix-build could not reuse that. OK, if I was using correctly the API to the cache and that nix-build cached my builds, then it could be reused. So this is very efficient because we never compile the same thing twice. If we compile it twice, then what? What happened? Oh, no, crap. If we compile the same thing twice, then there is a bug. This should never happen. OK. So this is the part about having a nicer interface. It is good, but sometimes too strict is just, I mean, it gets in the way. So I think we can go a bit further and improve the interface of nix to play nicely with users. Yes. And this is the second part of my presentation. So if you have questions on that part precisely, I would be happy to take them. Then I have a question for you. Do you use a tool like that? And I have another question. Do you think it's easy to implement? Elko, any idea? OK. I just wanted to point out that some way there are already some hacky ways how you can roughly what you already mentioned, so for example, some larger Haskell builds, problem that building them from scratch every time with a couple hundred packages, like each one, like a new bug, so one way to work around it, for example, is to usually you're supposed to add built inputs like SRC, right? You're actually add on your source. But you could claim a standard way to get this kind of fashion content as you have shown is that instead of just adding the sources, you also just add build directly as well. I think if you have a correct build system for your language, for example, PHD Haskell, and have build directly present, then it will resume from where it left off last time. You're basically hacking your way in by saying your sources are not just the sources, but actually the source plus the build directly. That already works, but of course, it's pretty hacking. The build directly can be large, large. Every time you do this kind of build, you create lots of stuff in the next store, so it would be really nice to have an official way to do that as you suggest, which that you have the same benefits but without the drawbacks. That kind of idea that this concept of extra data that is like source input that also has been fuming that the build is the effect that adds to the solution. Yeah, I didn't speak about that, but if you want to build a local project, then more often than not, every time you call a next build, then it will fetch the sources and put them in the store. Depending on your disk, this may be a very quite long operation already, and then you still need to build it from scratch, and this is stupid. One solution is to add the extra input, like you say, but then you do it. This build will never acknowledge that it was the same thing that you built before. There is no way to make this work efficiently for building, and you will never be able to share what you did. Okay, I think you're doing this. Yes, yes, exactly. I will add that to my do not do that slide. Come on. Yeah, there is a question. So in that environment, would you be allowed to access the network or not? And I guess we could have different versions of your and correct stuff. Okay, so next user story, and this may be a bit more technical. I have a problem with the next story. It's not a big problem, but it annoys me most of the time. Is that in my next story, there are so many packages that are just basically the same. Okay, if you take, for example, two popular data and you div them, the only difference is in the package config file, okay, and package config file, and there is only the popular data here and C flags. And why are they different because they reference the derivation itself? So this is the out hash of the derivation. It's not something, no, it's not an input that changed. It's only the out hash that has changed, and that was written in the derivation. So what does that mean? It means that because we gave two different out hash to this derivation, they end up being different. But if they had the same out hash, they would be the same. These are exactly the same derivation, except that this one is stored in a different location than this one, and this must be written into the files there. Okay, so this is what it looks like when there is no optimization, right? Oh, yeah, sorry, it's I3, but it could be any package. You have this derivation file, and then there is a very small difference, maybe an input change, but it's not important to the derivation. It doesn't change what gets built in the end. And so you end up with two different packages in the binary package. You will need to download this package two times if you use both, and then you end up with two packages in your store, and they are nearly identical. So the first optimization, and sorry, who is using an optimized store? Okay, quite a lot of people. There is this nifty tool that's called disk usage, and for a long time I did not understand why it's very weird result like that, but in fact the tool is very smart, so it only counts the space that it has not counted yet. Okay. So what happens here? I ask for the space of three popular data derivation, and the first one took 12 megabytes, and the others are only 60K. All right. This is because the optimization of the store hard links every file, the files that are the same together. So all the files that were already counted for this derivation are not counted anymore for these ones. Okay. This means that basically there is nothing in here. It corresponds to our intuition that there is only one file that changes, and in these derivations, these three derivations, you have folders, which cannot be hard linked, and hard links to the same files. Okay. So these are only counted in the first one. What does it look like on my picture? It looks like this. All right. You are able to optimize, use only the store usage, but you only know that in the end when you have downloaded twice, build it twice, so it's not very efficient except for disk space on your machine. And yes, network usage is important. Every week or so, we merge the unstable branch into master, and unstable branch is the one where you have mass rebuild stuff. Mass rebuild means that mostly, at least half of the packages are impacted. There is a huge impact. If you have three changes like that, you can be sure that you can download everything. So if you are like me and that you update once a month, then every time you update, you need to download a full new distribution. You need to download all the packages. And I think we can do better than that. I mean, we do not need to download everything. Probably there is not a big change, like a small change like we saw before. To do that, it's an idea that I would not implement, but apparently it existed at some point. We could invent this idea of binary diff. All right. Say, okay, this derivation, it looks like the previous one. Okay, it's a bit different, but let's just only ship the diff and keep the diff into the binary cache. Hydra could do that. It's just a different substitute that will be provided. It needs to be understood by Nix, but it looks quite easy to do. One question is, okay, what do you diff with, which is the old package? But this would help a lot to save bandwidth and space on Hydra. But there is another option that I prefer, and that I will call content address storage. Okay, because we all call it intentional store, and it's exactly not intentional. It's exactly the opposite of that. So let's keep its content address storage. Okay. What merchants made the RFC for that? And he's not here, I think. I'm a bit sad because I wanted to work on that with him, but anyway. The idea is that if you can detect that you build exactly the same package up to the self-links, then you can just use the same package and you're done. It's very nice because when you query Hydra about, okay, what's the substitute for that? Hydra will say, it is that one, and then you will check in your store and say, okay, I already have it. That's done. There is nearly no network, no bandwidth, and there is a lot of space that is not there. That is saved on Hydra. So yes, what was the situation before? If you have this standard environment, and then, okay, from that you can build X packages and then you build I3. Okay, you end up with three different packages in the end, and then there is a very small change to the standard environment. Maybe it's not significant, or I added an S in a comment in a bash script. I nobody does that because we know that we will break a lot of stuff, but it's not that important. So I do a small change and I end up with a different set of packages. With the content address storage, I could do something like that. Okay, I realize that, okay, STDN is exactly the same, so I can deduce that all the package will be the same because they will be built from the same package. There is no way they can change. The actual input to this package is the same. You build it and you look at the bytes and you say, okay, these are the same bytes. It's the same package. Yeah. It needs to build it. So there is no way around that. You need to build the package, but once it builds... Yeah, I'm not changing the derivation. You know, basically changing the build command or something like that. But you may be surprised there are a lot of cases where this happens. For example, if you change GCC, for example, then GCC will rebuild your project, but it may happen that it's the exact same project that is built. Okay, it's a new version of GCC, but there is not much change in the byte code that's produced. Right? So you may have that kind of behavior more often than you think. But I must admit that I do not know now what's the actual impact of that. So it would be nice to do some kind of impact analysis. Yes? Mm-hmm. Mm-hmm. Mm-hmm. Mm-hmm. Hi. Hi. I'm happy that you asked the question because I went too fast for that. The idea is that for the output, you need to strip that from the produced binary before hashing them. And this is the hash in which you will store your derivation. Okay, then you can reinsert the outlink. Okay. Yeah. Yes, of course. You need to do that. And it's more complex than it looks like because often the output ends up in very strange locations, for example, in a man page, and that man page might be zipped so it may not be discovered by NICs build when it strips stuff and reinserts them. So there is technicalities there that we need to think about. Yes, it's linked to that, of course. So I mean, it's interesting to see that, okay, let me change the derivation in the way that I'm sure cannot change the build. So for example, I could add a random variable to the environment that nobody should use, right? And if I'm not able to reproduce the exact same build, there is a problem either with content storage detection or with reproducibility. So it will maybe help to highlight, yeah, not stable package. Okay, so the main advantage is that you do not propagate, change to other package. You are able to better detect when the package is really the same as before. So it means that you can do less compiling. You do not need to compile all the dependent packages. You have faster updates. You do not need to download from IDRA. It is not too difficult to implement. Yeah, not too difficult. But I mean, everything is in there. We could do that with existing tools. The real difficulty is with forward and backward compatibility. If we do that, then, yes, of course, we'll change nicks in a major way. So we need to investigate how we can do that without breaking everything. If we break everything, maybe we need to say, okay, this is very important. It has huge advantages. So, okay, let's break nicks and start again over. But I don't know yet. So this is really the problem now with this ID. It's quite difficult to know what would be the real gain. I would like to set up my own IDRA and try to do tests with that. I mean, it has a huge computing power, which I do not have yet. Maybe in two years for next nicks. Yes, the reason why I was really interested in that is because I have this pull request that is now completely bit rotten, but never mind. I wanted to make a standard environment that just trips everywhere, everything, everything that looks like an ELF strip it. Okay? Because we had issues like, for example, PHP was storing libraries in very strange locations. We did not see that. And so PHP was basically depending on GCC. And I mean, this was blowing the closure size. For the story, this was, I mean, this was there for so long that at some point, Elko fixed the PHP stuff before I even merged this one. So it was easier to fix by hand than to fix it globally. The interesting part about this pull request is that for most packages, it changes nothing. Okay? And if the package changes, then there was something that was not stripped before and that we stripped after. So it's interesting to look into that package. And even the ability to detect when packages are exactly the same would help with refactorings. Okay? You refactor something, you build everything, and you see what has changed. This gives you some information about did I do something wrong? Is this something that I really intended in the refactoring? Even that the refactoring should just be as this. Okay. So now if you want to reach the gap between all the IDs, if we do this content address storage and caching inside the build system, then it becomes quite complex. If you cache build, then it becomes much less, it may become less stable or it may become more stable. So you have a lot of interactions in there that needs to be investigated. And I think it would be fun, but maybe a bit complex. So let's do one step at a time. And it would help, like we said before, to catch unstable builds. Right? If the content hash changes too often, then there is something that's not stable in the build. Okay. So I presented these three IDs of improvement to Nix. When I started, I was feeling that you could not change Nix, but it's not true. There are a lot of things that we can improve and we can make it more user friendly. About caching builds, that's something that I'm working on from my PhD. The content address storage is something for which there is an RFC. So if you want to work on that, you're welcome to contact WordMertens and we can do it together. The Nix shell hack with the zipper is something that's not being implemented at all. And I would be happy to start that with someone. And that's it. The behind pillar, over there, over there. The funny thing is that I have been compiling build system to other build systems for two months now, but not to Nix. I guess what I want to do is basically the same, right? Because I asked the build system to just say what it wants to build and pass it to Nix build. So, he is basically exposing all he wants to do. So, I would like the build system to not compile to Nix, but to compile to a standard protocol that could be used for interoperation with any other tools, right? I would like something that is more global than just, okay, these two Nix, these two dots, et cetera. I want just one common protocol to build stuff. Okay, thank you. Thank you.
 Next speaker is Guillaume Modoux, your sonorema's Leius on GitHub.  I started using Nix at Mozilla a few years ago.  Now he's working on a PhD on incremental builds.  Yes.  Thank you.  Give it up to Guillaume.  Okay, so let's start with a small background on me.  So I do my PhD in Louvain-Leneuve near Brussels.  And just for the story, Louvain-Leneuve has a huge biking event, which is also the second  biggest drinking event in Europe after the one in Munich.  So you know everything about my username.  Okay, it's like a pun with a French word.  But yeah, it's not very good as an introduction to this presentation.  So please mind, there is one letter different, and that's very important.  Right?  I contribute to Nix packages, of course.  And my GitHub picture is a stone like that.  This stone is a stone from that castle, which I also contributed to build.  So now you already know that I like building stuff, right?  So I started my PhD on incremental builds, and I want to feed back on this presentation.  So this is very important to me.  Do not, is it to interrupt the presentation if something, if you want to ask a question  in the meantime?  So let's start the presentation.  Incremental package builds.  What is incremental?  I went to the dictionary, which is always a good way to start, except that in this case  it doesn't mean anything.  Incremental is something that occurs in increments.  Okay, come on.  So what is increments?  It's the amount of debris by which something changes.  Okay, we cannot do anything with the definition, so I just came with mine.  An incremental build system is something that works with very small steps.  We can do each of them separately and work with them.  We can use old build products, and we use them so we do not have to do things that are  already done.  And we also, of course, need the ability to detect what needs to be done.  So this is the plan of this presentation.  I will take three examples from real life.  One with Firefox, one with i3, the window manager, and one with the store.  The Firefox example shows why it's important to have small steps.  If you ever try to build Firefox, then the build phase takes one hour, and everything  else is basically negligible compared to the huge build time.  I was playing with that package, and I had to change the fix-up phase.  And of course, I made a typo in the bash script, and then, whoop, everything was lost.  Okay, so we really need checkpoints.  I mean, it's just like when you are doing rock climbing, you are climbing, okay, and  then at some point, there is a problem, and then if there is no checkpoint, then, wow,  you are back 2.0, right?  So we really need that kind of stuff.  It's not to lose everything all the time.  So how can we do that?  A very simple idea is just, okay, we already have parts in every build.  We have different phases.  So we could split them in different packages so that we, once one phase is finished, I  mean, it's done.  We can work with the next one.  Yeah, but you know, it doesn't always work.  For Firefox, it's good, but if the failure occurs in the build phase, then basically,  you have still the same problem, that you lose one hour of build phase.  It's not good for the store, because you will have every phase in the store.  What does it mean?  Can that even work?  But this idea has already been implemented in some places with wrappers.  Firefox, for example, has an external wrapper, which is another package that just writes  the wrapper.  That way, you can change the wrapper easily without changing Firefox all the time.  So incremental builds are about small steps.  And in fact, Nix is incremental in some way.  It's just that it's incremental at the level of package.  And the small step for Nix is a package.  At the other hand, we have build systems, they already manage incremental builds of  package, like Firefox has its own build system, and it's also incremental, but we are not  able to use that now in Nix, because we want things to be very pure, so there is no interaction  with the external world.  So the idea here is to use existing build systems and to make them interoperate with  Nix, so that we can achieve incremental builds within Nix.  First build system that everyone knows is make.  And well, I will not speak too much about make because it works, but there is nothing  you can do that's correct or in the sense that we like it.  Our make will never be pure.  That's not in the design.  We use timestamps and we do not want timestamps in the store, for example.  It requires to have the previous builds in the same tree as filing the tree, so you  need to add that as an input to the package and everything gets ugly.  But there are other tools that are quite interesting, like C-Cache.  We know C-Cache.  Not that much people.  Okay.  So C-Cache memorize compiler invocations, so you can wrap a compiler code with that and  if the same compiler code has been made before, it will fetch the result from the cache.  Even better, Mozilla developed the shared C-Cache, where this cache of previously built  stuff is shared on the network, so you can reuse a build that someone else did.  And well, this is a very good and robust design because it's the design behind NICS.  There is a binary cache that you can use and that we are sharing and this is used to replace  building of some packages if they were already built before.  So caching is a very good idea that we will reuse later.  But I want to show a small project by Elko, which is NICS make.  And NICS make basically allows you to, it's a build system.  At the moment, it only works with very simple stuff like that.  You see that it's 15 commits, okay, so it's not very active, but you can still do things  like that.  There is one derivation, compiler C produces derivation and derivation just generates a .o file and then there is the link derivation that just takes all the o file and produces  the binary.  So we can use NICS to do incremental compilation at a final level than just packages, but,  okay, you can do very small steps.  It's compatible with NICS, of course, but it's even worse than before.  Every single.o file ends up in the store and you also need to port every project to  NICS make.  Okay, so if you're using whatever before like Bazel, then you need to write another layer  to port it to NICS make.  This is not feasible.  Speaking about Bazel, this is an amazing, a tremendous project from Google.  They are very proud of their stuff, fast or correct, pick two.  And it's a project that, as most of things write, they manage to build every command.  They can basically cache every command and put that in a shared cache.  They can even ask another machine to compile that step and then get the results.  So this is something that we would like to have.  It's also, it's really, really correct.  There is sandboxing going in there.  But you know, it's so good that it does exactly the same stuff as NICS.  So we are, when you try to make both of them work together, they're fighting to take the  control, be the one that manages sandbox and do not let the others do other stuff.  So it's quite difficult to use it.  In fact, it's used at Google and at Google, everything uses Bazel.  So it's quite easy.  But with NICS, NICS packages, we have something that's much more heterogeneous.  I mean, there are packages with make, packages with SBT, packages with CMake, and we cannot  use just one build system to rule them all.  So is it possible to do something intermediate?  And my solution is, okay, we want caching.  We want to be able to cache smaller steps than just packages.  Can we somehow allow the build system to access an external cache while remaining correct?  And to do that, we may need the help of NICS build to provide the same interface.  But in the first attempt, if you do not care too much about correctness, then it's very  easy to implement.  Basically, you build not so pure, allow access to an external store, and allow, for example,  ccache to access that store.  It looks like that.  You call make.  I mean, NICS builds Starless Sandbox, makes, works in there, calls ccache, for example.  And then ccache grabs in the environment a socket to discuss with the external store.  As simple as that.  If you trust ccache to do things correctly, then you will have correct builds.  Of course, you cannot always trust ccache, so we will have to implement some checking  on top of it.  But this is the basic idea.  And if you start with something simple like that, then you can port one package at a time,  try to make it work, start maybe with big package like Firefox.  And yes, this is my master plan.  This is something that I want to do for my PhD.  The ability to use incremental builds from the build system within NICS and to be able  to go both directions to make them cooperate.  So at the end, we should be able to rule over the world, right, with the system.  But it still works.  It still works in progress.  Okay.  Any question about that, maybe, before the next part?  I mean, you can see us question afterwards.  Yes?  Correctness part is the interesting one, right?  If you have a problem in caching normally, you have a problem in the end problem.  Does it, do you have plans to work like tension list or stuff where you check output?  Do you have like one or two words how you are approaching problems?  Okay.  I can say two things.  Yes, the intentional store would help.  I will speak about that later.  And what can I say?  Bazel already does something like that.  Much more advanced.  That was C-Cache does.  Except that he does a lot of other stuff that we do not want.  So we may need to import that part of the code.  And then, you know, if it works for them, it should work for us too, right?  If you want more correctness than what they achieve, which is already quite high, then  we may still work on it, implement it.  The interface here may not be just access to files.  It may be, okay, I want to build, to call that command with these files, with these  environments.  And then we let nix build, run the commands and produce a cache result.  So if everything is in the hands of nix build, then it should be clean.  Okay.  But this is more strict and this needs more, I mean, it needs to change a bit of C-Cache  works at the moment.  So the more correctness you want and the more work you have to do, of course.  Okay, I was just wondering if that is very result instead of putting them in plain.  Okay, so the idea about this stuff is that this is not defined.  This may be the next store.  This may not be.  It depends on what you want to do.  I mean, this is not the first time that I did something like that.  There are a lot of you that want to have hierarchical nix stores.  When you have one nix store for the packages, then one nix store for other stuff that are  less important, that can be a rich collected morphine, and maybe another nix store for  custom projects.  I'm not sure that this idea of nix store is the perfect way of doing caching.  The idea of nix store is to have, I mean, you need a directory because you need to access  the files, but it may be more efficient to cache compressed versions of the files, for  example.  So you may not want to, you may not want exactly a nix store, but you need some way to cache  stuff.  Okay, let's continue anyway.  You can ask questions later.  So another use case I had, I had a problem with itry, which is my window manager.  And being the window manager is something that you need all the time.  But to reproduce the bug, I had to use it on my main laptop because basically the bug  happened when I undocked it and dogged it again.  So I mean, this was the only way I found to reproduce that bug.  And so to test new versions of I3, I had to propagate them to nix stores configuration.  I mean, you all already had something like that, so you want to test something and it's  very complex because it needs to go into the nix stores configuration and how do I do that?  My typical debug session looks like that.  I use nix shell.  I try to patch, see if it compiles, et cetera.  If I can do that, then I extract a diff and then I generate an overlay to build a new  package.  And I check that it builds with nix builds.  Then I insert that package in nix OS and just rebuild nix OS and then I restart, I test,  and if it works, I mean, that's fine.  If I can reproduce the bug, it's not that good.  And if it crashed everything, then I need to roll back, of course.  And then back again because it's a loop.  I mean, when you are trying to debug something, you have to insert debug statements, then  maybe it starts gdb, stuff like that, so it's quite complex.  And it would be so simple if I could just compile it three and say, okay, put it in  the store and use that.  I know it's not clean, but then it would be easy, right?  And there are solutions to do that.  I should maybe not say that.  Maybe we need to cut the video at some point.  But it is possible to mount the nix store with rights.  But yeah, okay, you should not do that.  More clean but very hacky solution is to insert a sim link into the nix store.  Make a fake derivation that's just a sim link to somewhere else, your own directory or  your project directory.  Then there are some technicalities with, you know, self-loops and stuff like that, but  it's possible.  And yeah, you can go straight to the solution and say, okay, this service use this package,  okay, I can override the service, the nix store service and say, okay, use the package  in my home environment.  That works too.  But then, I mean, it's not what we want.  We are used to something that's pure, that's clean, and then we are trying to hack everything.  So this is indeed very hacky.  So I went to look for tools that allow to do that.  And one of them is Git-rebase interactive.  When you do Git-rebase interactive, it tries to apply all the commits in turn.  And when there is a conflict, you are dropped in an environment that looks like it's a normal  Git repo, but it's not really.  If you try to push that on a branch, it will be strange, et cetera.  But you can commit, behave like you are in normal commit environment.  And when you're done, you say just continue, and you just continue this job integrating  your modification.  That's something that we would like with nix shell.  Basically nix shell set up the right environment to build the stuff.  You can build it.  You can go to the end, but when you are doing make install, it doesn't install because you  cannot access the nix store.  Why?  It would be interesting to be able to write in the nix store at that point and produce  a package.  And if you are annoyed about purity, then just make nix shell accept the hack option.  And the hack option produce a derivation with a random input parameter.  So it doesn't conflict with the normal one, the pure one, and this random input says,  OK, it represents the fact that a human can just tamper with the build, so the human can  add random stuff into the build.  But that would be very, very nice because then we could insert basically anything in  the store in a way that's quite clean and corresponds to the nix model.  To explain that differently, I have defined the nix OSI model, which is not to be taken  too seriously, OK.  But we have layers, and each of these layers are related to different tools.  So if you use nix builds, then you know nothing about nix source, basically.  If you use nix source, then you know nothing about different servers like nix-obs do.  And if you use make, then you know nothing about everything that's in nix.  Nowadays, it's nearly impossible to just run make on a project.  You need a nix shell to get the right environments.  So basically what you do is you start at the nix shell level, you open the zipper, OK,  and then you enter an environment where you can just play with i3 and your project.  When you're done, you cannot use that package, but it would be nice if you could just close  the zipper and say, OK, I have a package now.  I enter it, and I can exit it.  Of course, you can go further and say, OK, let's add a hack command to nix-rebuild, to  nix-source-rebuild.  And then you would start higher in the hierarchy, zip inside it, and then go outside of the  store.  So it would drop me, for example, in a shell where I can just edit i3, and then it would  use that version of i3 only for that build of nix-source.  OK, it's not that clean, but in the end, it's correct because it's a version with a random  input which corresponds to the fact that they've changed something.  And I can easily enter and get back off the tree.  OK.  Now for the extra step, if you remember caching, then you can still use caching in that.  If what I did in the nix shell corresponds to what will be done by the nix-build, then  there is no reason nix-build could not reuse that.  OK, if I was using correctly the API to the cache and that nix-build cached my builds,  then it could be reused.  So this is very efficient because we never compile the same thing twice.  If we compile it twice, then what?  What happened?  Oh, no, crap.  If we compile the same thing twice, then there is a bug.  This should never happen.  OK.  So this is the part about having a nicer interface.  It is good, but sometimes too strict is just, I mean, it gets in the way.  So I think we can go a bit further and improve the interface of nix to play nicely with users.  Yes.  And this is the second part of my presentation.  So if you have questions on that part precisely, I would be happy to take them.  Then I have a question for you.  Do you use a tool like that?  And I have another question.  Do you think it's easy to implement?  Elko, any idea?  OK.  I just wanted to point out that some way there are already some hacky ways how you can roughly  what you already mentioned, so for example, some larger Haskell builds, problem that building  them from scratch every time with a couple hundred packages, like each one, like a new  bug, so one way to work around it, for example, is to usually you're supposed to add built  inputs like SRC, right?  You're actually add on your source.  But you could claim a standard way to get this kind of fashion content as you have shown  is that instead of just adding the sources, you also just add build directly as well.  I think if you have a correct build system for your language, for example, PHD Haskell,  and have build directly present, then it will resume from where it left off last time.  You're basically hacking your way in by saying your sources are not just the sources, but  actually the source plus the build directly.  That already works, but of course, it's pretty hacking.  The build directly can be large, large.  Every time you do this kind of build, you create lots of stuff in the next store, so  it would be really nice to have an official way to do that as you suggest, which that  you have the same benefits but without the drawbacks.  That kind of idea that this concept of extra data that is like source input that also has  been fuming that the build is the effect that adds to the solution.  Yeah, I didn't speak about that, but if you want to build a local project, then more often  than not, every time you call a next build, then it will fetch the sources and put them  in the store.  Depending on your disk, this may be a very quite long operation already, and then you  still need to build it from scratch, and this is stupid.  One solution is to add the extra input, like you say, but then you do it.  This build will never acknowledge that it was the same thing that you built before.  There is no way to make this work efficiently for building, and you will never be able to  share what you did.  Okay, I think you're doing this.  Yes, yes, exactly.  I will add that to my do not do that slide.  Come on.  Yeah, there is a question.  So in that environment, would you be allowed to access the network or not?  And I guess we could have different versions of your and correct stuff.  Okay, so next user story, and this may be a bit more technical.  I have a problem with the next story.  It's not a big problem, but it annoys me most of the time.  Is that in my next story, there are so many packages that are just basically the same.  Okay, if you take, for example, two popular data and you div them, the only difference  is in the package config file, okay, and package config file, and there is only the popular  data here and C flags.  And why are they different because they reference the derivation itself?  So this is the out hash of the derivation.  It's not something, no, it's not an input that changed.  It's only the out hash that has changed, and that was written in the derivation.  So what does that mean?  It means that because we gave two different out hash to this derivation, they end up being  different.  But if they had the same out hash, they would be the same.  These are exactly the same derivation, except that this one is stored in a different location  than this one, and this must be written into the files there.  Okay, so this is what it looks like when there is no optimization, right?  Oh, yeah, sorry, it's I3, but it could be any package.  You have this derivation file, and then there is a very small difference, maybe an input  change, but it's not important to the derivation.  It doesn't change what gets built in the end.  And so you end up with two different packages in the binary package.  You will need to download this package two times if you use both, and then you end up  with two packages in your store, and they are nearly identical.  So the first optimization, and sorry, who is using an optimized store?  Okay, quite a lot of people.  There is this nifty tool that's called disk usage, and for a long time I did not understand  why it's very weird result like that, but in fact the tool is very smart, so it only  counts the space that it has not counted yet.  Okay.  So what happens here? I ask for the space of three popular data derivation, and the  first one took 12 megabytes, and the others are only 60K.  All right.  This is because the optimization of the store hard links every file, the files that are the  same together.  So all the files that were already counted for this derivation are not counted anymore  for these ones.  Okay.  This means that basically there is nothing in here. It corresponds to our intuition that  there is only one file that changes, and in these derivations, these three derivations,  you have folders, which cannot be hard linked, and hard links to the same files.  Okay.  So these are only counted in the first one.  What does it look like on my picture? It looks like this.  All right. You are able to optimize, use only the store usage, but you only know that in  the end when you have downloaded twice, build it twice, so it's not very efficient except  for disk space on your machine.  And yes, network usage is important.  Every week or so, we merge the unstable branch into master, and unstable branch is the one  where you have mass rebuild stuff.  Mass rebuild means that mostly, at least half of the packages are impacted. There is a huge  impact.  If you have three changes like that, you can be sure that you can download everything.  So if you are like me and that you update once a month, then every time you update, you  need to download a full new distribution.  You need to download all the packages.  And I think we can do better than that.  I mean, we do not need to download everything.  Probably there is not a big change, like a small change like we saw before.  To do that, it's an idea that I would not implement, but apparently it existed at some  point.  We could invent this idea of binary diff.  All right.  Say, okay, this derivation, it looks like the previous one.  Okay, it's a bit different, but let's just only ship the diff and keep the diff into  the binary cache.  Hydra could do that. It's just a different substitute that will be provided.  It needs to be understood by Nix, but it looks quite easy to do.  One question is, okay, what do you diff with, which is the old package?  But this would help a lot to save bandwidth and space on Hydra.  But there is another option that I prefer, and that I will call content address storage.  Okay, because we all call it intentional store, and it's exactly not intentional.  It's exactly the opposite of that.  So let's keep its content address storage.  Okay.  What merchants made the RFC for that?  And he's not here, I think.  I'm a bit sad because I wanted to work on that with him, but anyway.  The idea is that if you can detect that you build exactly the same package up to the self-links,  then you can just use the same package and you're done.  It's very nice because when you query Hydra about, okay, what's the substitute for that?  Hydra will say, it is that one, and then you will check in your store and say, okay, I  already have it.  That's done.  There is nearly no network, no bandwidth, and there is a lot of space that is not there.  That is saved on Hydra.  So yes, what was the situation before?  If you have this standard environment, and then, okay, from that you can build X packages  and then you build I3.  Okay, you end up with three different packages in the end, and then there is a very small  change to the standard environment.  Maybe it's not significant, or I added an S in a comment in a bash script.  I nobody does that because we know that we will break a lot of stuff, but it's not that  important.  So I do a small change and I end up with a different set of packages.  With the content address storage, I could do something like that.  Okay, I realize that, okay, STDN is exactly the same, so I can deduce that all the package  will be the same because they will be built from the same package.  There is no way they can change.  The actual input to this package is the same.  You build it and you look at the bytes and you say, okay, these are the same bytes.  It's the same package.  Yeah.  It needs to build it.  So there is no way around that.  You need to build the package, but once it builds...  Yeah, I'm not changing the derivation.  You know, basically changing the build command or something like that.  But you may be surprised there are a lot of cases where this happens.  For example, if you change GCC, for example, then GCC will rebuild your project, but it  may happen that it's the exact same project that is built.  Okay, it's a new version of GCC, but there is not much change in the byte code that's  produced.  Right?  So you may have that kind of behavior more often than you think.  But I must admit that I do not know now what's the actual impact of that.  So it would be nice to do some kind of impact analysis.  Yes?  Mm-hmm.  Mm-hmm.  Mm-hmm.  Mm-hmm.  Hi.  Hi.  I'm happy that you asked the question because I went too fast for that.  The idea is that for the output, you need to strip that from the produced binary before  hashing them.  And this is the hash in which you will store your derivation.  Okay, then you can reinsert the outlink.  Okay.  Yeah.  Yes, of course.  You need to do that.  And it's more complex than it looks like because often the output ends up in very strange  locations, for example, in a man page, and that man page might be zipped so it may not  be discovered by NICs build when it strips stuff and reinserts them.  So there is technicalities there that we need to think about.  Yes, it's linked to that, of course.  So I mean, it's interesting to see that, okay, let me change the derivation in the  way that I'm sure cannot change the build.  So for example, I could add a random variable to the environment that nobody should use,  right?  And if I'm not able to reproduce the exact same build, there is a problem either with  content storage detection or with reproducibility.  So it will maybe help to highlight, yeah, not stable package.  Okay, so the main advantage is that you do not propagate, change to other package.  You are able to better detect when the package is really the same as before.  So it means that you can do less compiling.  You do not need to compile all the dependent packages.  You have faster updates.  You do not need to download from IDRA.  It is not too difficult to implement.  Yeah, not too difficult.  But I mean, everything is in there.  We could do that with existing tools.  The real difficulty is with forward and backward compatibility.  If we do that, then, yes, of course, we'll change nicks in a major way.  So we need to investigate how we can do that without breaking everything.  If we break everything, maybe we need to say, okay, this is very important.  It has huge advantages.  So, okay, let's break nicks and start again over.  But I don't know yet.  So this is really the problem now with this ID.  It's quite difficult to know what would be the real gain.  I would like to set up my own IDRA and try to do tests with that.  I mean, it has a huge computing power, which I do not have yet.  Maybe in two years for next nicks.  Yes, the reason why I was really interested in that is because I have this pull request  that is now completely bit rotten, but never mind.  I wanted to make a standard environment that just trips everywhere, everything, everything  that looks like an ELF strip it.  Okay?  Because we had issues like, for example, PHP was storing libraries in very strange locations.  We did not see that.  And so PHP was basically depending on GCC.  And I mean, this was blowing the closure size.  For the story, this was, I mean, this was there for so long that at some point, Elko  fixed the PHP stuff before I even merged this one.  So it was easier to fix by hand than to fix it globally.  The interesting part about this pull request is that for most packages, it changes nothing.  Okay?  And if the package changes, then there was something that was not stripped before and  that we stripped after.  So it's interesting to look into that package.  And even the ability to detect when packages are exactly the same would help with refactorings.  Okay?  You refactor something, you build everything, and you see what has changed.  This gives you some information about did I do something wrong?  Is this something that I really intended in the refactoring?  Even that the refactoring should just be as this.  Okay.  So now if you want to reach the gap between all the IDs, if we do this content address  storage and caching inside the build system, then it becomes quite complex.  If you cache build, then it becomes much less, it may become less stable or it may become  more stable.  So you have a lot of interactions in there that needs to be investigated.  And I think it would be fun, but maybe a bit complex.  So let's do one step at a time.  And it would help, like we said before, to catch unstable builds.  Right?  If the content hash changes too often, then there is something that's not stable in the  build.  Okay.  So I presented these three IDs of improvement to Nix.  When I started, I was feeling that you could not change Nix, but it's not true.  There are a lot of things that we can improve and we can make it more user friendly.  About caching builds, that's something that I'm working on from my PhD.  The content address storage is something for which there is an RFC.  So if you want to work on that, you're welcome to contact WordMertens and we can do it together.  The Nix shell hack with the zipper is something that's not being implemented at all.  And I would be happy to start that with someone.  And that's it.  The behind pillar, over there, over there.  The funny thing is that I have been compiling build system to other build systems for two  months now, but not to Nix.  I guess what I want to do is basically the same, right?  Because I asked the build system to just say what it wants to build and pass it to Nix  build.  So, he is basically exposing all he wants to do.  So, I would like the build system to not compile to Nix, but to compile to a standard protocol  that could be used for interoperation with any other tools, right?  I would like something that is more global than just, okay, these two Nix, these two  dots, et cetera.  I want just one common protocol to build stuff.  Okay, thank you.  Thank you.
Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_0_743.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_744_851.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_852_923.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_924_1571.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_1572_2399.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_2400_2675.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_2676_2915.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_2916_3455.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_3456_4031.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_4032_4079.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_4080_4151.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_4152_5735.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_5736_6851.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_6852_8579.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_8580_13427.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_13428_13547.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_13548_17327.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_17328_17435.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_17436_17483.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_17484_18539.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_18540_19727.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_19728_19931.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_19932_22871.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_22872_23975.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_23976_24923.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_24924_25283.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_25284_26075.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_26076_29675.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_29676_29807.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_29808_29915.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_29916_31787.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_31788_40247.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_40248_40859.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_40860_42191.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_42192_42419.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_42420_45611.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_45612_45851.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_45852_46787.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_46788_47171.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_47172_49631.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_49632_49835.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_49836_59051.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_59052_59135.png Incremental_package_builds_by_Guillaume_Maudoux__layus__AED-dgCa9GU_59136_61734.png