Packaging Rust crates in GNU Guix
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Flashner, Efraim
Rust is a language with a healthy ecosystem and a strong developer base. With built-in dependency management it's easy to build and install new programs even for those who have never used the language. But how is its adoption among Linux distros? Come with me as we figure out how best to package rust libraries and binaries in Linux distributions which demand total control over dependency management.
 Okay, so the fun part is I ran out of laptop, so I'm down to my G4 with the micro VGA port, so yeah, so okay, so that'll be fun later also. So packaging crusts. Packaging crates, how hard could it possibly be? So I've been using Geeks for a couple years now. I don't really do a lot of programming myself, I like watching everything build, I like having all the packages actually work. So I guess my first interest in Rust really was when we started needing it for Firefox or for the GNU fork of it for Icecat. So a quick history of I guess how the two run together. Geeks is a source-based distribution, so we did have over here in, since 2016 we went and just grabbed the binaries from Rust and installed those and then used those to build up to all the later versions. And then later we went and removed those and switched to using MRust C for bootstrapping Rust. And so for Icecat we just go ahead and use all of the crates that are already rendered in with Mozilla. The other ones we've gone and started adding them in, it's looking at the timeline it's been more recent, we've less than a year that we've actually had a number of crates in. So in Geeks when we go and we build packages, everything gets installed to its own unique space in the store. So this makes it so that you can have multiple versions of every package installed. This one would add not, I mean it's on the disk, it's not actually installed installed. It gets referenced by whatever uses the specific version of SQLite. So you have the binary, you have the headers, the libraries, the man page has all of the normal stuff that goes with it. And so the problem we were running into with Rust was go ahead, we build, we install and we get the license file. So it's saying okay, so what do we do with this? So yes, I remembered why I put that there, I'm sure that would have been helpful. So with crates, with Rust, it's very friendly for hacking on, go you say Rust build, say cargo build, it builds in all the dependencies, it builds everything that it needs in it, you get your final binary, you don't actually get a whole bunch of libraries installed with it. So we've been loving the API for crates.io, we have all the information, all of the, looks like this part here doesn't, yeah, have all of the dependencies listed on it for the different versions, whether it's needed for building it, for dev, mostly for testing or just in general. So we, with this we've created a recursive importer so it'll import the crate, it'll fill out the template basically of what version of it, where do you download it from, what are the different versions of everything that you need. So big difference that we have with the crates, with the cargo build system is here under arguments everything is a cargo input, whereas for, say for a news boat, we have native inputs and we have inputs and so here it's, you know, compile all of the different libraries and then they get linked in with the cargo dependencies, we're pulling in the source itself and compiling everything together. So I guess, yeah, there's our friendly cat, what's the difference? So, yes, like we saw before, it's, you know, we're not, you know, not vending code, we're not, you know, when we do have vending code we go ahead and pull it out and link to other system libraries or in case with the crates which I guess don't actually vendor the code but it's, you know, grab the source from here, grab the source from there, put everything in and build it, we go ahead and basically mirror the same thing, we download all of the sources ahead of time, we override the, forget exactly what's called the cache, yes, where all of the crates are cached with our own cache directory inside the build tree and say, you know, look over there, that's where all of the crates are for downloading it. So as a typical distro we prefer the shared library option where we go ahead and compile it, give it all of the options we can, link to it everywhere, the shared source, you know, we're working with it, it's not our favorite one. I am definitely liking the size of the binaries that come out from the Rust crates and, again, not vended code. So one of the things that, it was the past couple days we had a small geeks hacking session and a number of us trying to work on how to actually work on the Rust build system and to try to, I guess, actually test the crates that we're building better, things like that and just had a screenshot from the DEPPS file in the build directory. And, yeah, so, you know, we see SerdeyDirive has an SO file and, you know, have over here, this one was for Serdey itself, so, I mean, this one looks like it would be the binary and then a lot of our lib, our metadata ones, the.d, I'm pretty sure were text files that just said how to build it. But you know, basic thing that we got was none of us really programmed in Rust, none of us really knew. It was kind of a, well, instructions are cargo build and it builds it and it downloads everything and so what do we do with it? So we were trying to figure out, you know, can you reuse parts of this? Can you reuse the R lib? I actually went and tried to model the actual build process and say, okay, well, you know, I've gone and installed the different R lib into the lib directory and say, you know, when you build, go and look for it over there and pull it in and didn't actually find that, didn't actually work for us. I mean, this one I tried a while ago, it just ended up not working so well in the end. And so, yeah, so where, you know, at this point we have a couple hundred crates, most of them we, you know, as part of the build process, we actually go and we patch all of the source shibings anywhere in the source. We regenerate the cargo.lock file and we build everything, you know, of the dependency trees. You know, we look at this and say, you know, none of it looks really like something that we would go ahead and install. So for binary, binary, those ones are easier. We actually do install those. So yeah, so we were, you know, I guess, you're wondering if, you know, I also wanted to know if anyone else, you know, is our other people trying to figure out how to reuse the, you know, see the build artifacts or the generated libraries. One of the things that talking with looking around online was that there's no stable ABI from version to version. We want to make sure that we actually rebuild the crates every time there's a rust version change. So Geeks is declarative, so we know exactly which versions of the crates are going in. We know exactly which compiler is going in. Anytime something gets updated, you know, all the packages get rebuilt. So we're sure that we have the same compiler each time. You know, we're really aiming to go for a, you know, for the same shared library type of thing that we had with other ones. Yeah. Yeah, thanks. So, yeah, so I guess you're not just looking for questions also if anyone has, I guess, ideas or suggestions on, you know, other ways to go and, okay, if anyone has other ideas on how to go and I guess reuse, I guess, either shared libraries or build artifacts, even if it's just, you know, taking the build artifacts, storing them somewhere and pulling them back in again to the dependency folder. It's time to build the next crate. You know, if anyone's, I guess, worked with, you know, I guess trying to create a cache of the different files. Are there any other distributions looking into this, like a few of them that maybe the TVMP or something? Yeah. So the question was, are there other people looking into it? See the Debbie and Guy nodding yes. I assume Nix is also looking into it. They have a pretty similar model of everything is, it depends on exact versions going all the way down to the bottom of the bootstrap tree. So some reason I haven't actually looked to see what they've done yet. We should definitely have a talk about this. Yeah. I guess my quick look at what Debian does is have here with the, much more responsive than my computer, with the different features provided. I saw that Debian actually produces crates for, I think they're, I don't know if they're crates or source crates for the different, for the different features that the different crates provide. Right now we just provide, as part of the build and test, we just pull in all of the crates listed and build and then not install the libraries. I'm a Gen2 developer. I'm also a Rust developer but not at the same time. But I do know that Gen2 is looking into this. With Gen2 packages there can be a certain amount of overhead for maintaining each version of the package. You know, key wording of the various architectures. And we also have the concept of a stable and stable. Do you find with the amount of churn in the Rust ecosystem, is it scalable to maintain this many packages and this many versions? Yes. So the question was, I guess there's a lot of versions, a lot of churn in the packages. And I guess, well, this Gen2 has the use flags and the stable and stable we don't have that. We just have, I guess it's either included or it's not included. And does it become too many packages? So for us, for say Rust-Ran 0.6, we just have a package Rust-Ran 0.6. For the Rust packages, for the other ones, say for NewsBoat, it's just NewsBoat. The new version, it gets bumped for Rust-Ran. If there's a 0.6.6, then 0.6 becomes 0.66 in the version field. We have 0.7, which is, in there is 0.7, we have 5, 4, and 3 also. So we just have tons and tons of packages. We, I guess it's not as bad as our Python system yet, but that's really because we only have, I think, four or five actual Rust programs. So for us, the problem is more of, yes, the way the way Geeks is architected, actually compile all of the, not all of the package, but all of the package definitions so that we don't have to interpret it every time. So the file with the Rust packages is five, 600 packages long. It starts to get big. It's about time for us to start splitting it into multiple files already. So in terms of having multiple versions, 0.5 would basically, we just add here between package and name a field that says inherit from the previous version of Rust. And then we just change the input. So it's in the description, doesn't change. The license generally doesn't change. We just say, well, this version has different dependencies. And when I say go ahead and build it, it just figures it out itself. Do you know how Haskell does it in either Davion or Nix? Because I believe they have some problems, but I don't know Rust good enough. In Haskell, so first of all, there is an upstream stackage repository, which is a universe of libraries that gets published every quarter, and then all those libraries are tested to be compatible, and Debian just uses this snapshot, and then it upgrades all the Haskell libraries in Davion. Are they updated in lock stack? So the question was, how does this compare with Haskell with having this with stable release? Right. Let's see. In Geeks, we also just go and basically pull in version from, I haven't spent a lot of time with the Haskell ones. We stick with the stable release more or less to try to just make sure everything actually compiles together. I know Nix has a lot more Haskell developers, and so they're tooling around Haskell is a lot better than ours. I guess in some ways, especially when it comes to the bootstrap binaries, Rust in Geeks is doing better than Haskell, and that we're actually bootstrapping from C, and we haven't figured out how to do that yet with the Haskell. That's just a matter of time until people will do it in Rust. My complaint about the long compile times to compile all the versions of Rust to make the Haskell compiler, but if it's doable, we'd be happy to have it. Yeah. First, thanks for the presentation. Sorry. Sorry. It's, yeah. Yeah, some of it is just, yeah, the green doesn't show up as well with the light one, and yeah, it's, yeah, definitely something I should make sure the colors work better next time. Yeah? The question I have is, what are you trying to, or what's the benefit you expect from caching the artifacts? The R-lib? Yeah, so the question is, what's the benefit of caching the R-libs? I guess, so right now we go and pull in all the sources and just build everything all at once. What we'd love to be able to do is to build each, each create one by one, and actually create the dependency tree and say, you know, these, these packages depend on those packages and to actually be able to track, you know, when I upgrade Rust-lib C, you know, which creates need to be updated, which, you know, which packages do I need to test to see that they actually work? And right now, when we go and we, I mean, as part of the, I guess, in the Geeks tooling, we go and say, you know, say, what, what packages depend on this other package? If I type in what depends on Rust-lib C, nothing depends on it. You know, if we, if we go ahead and compile and install it, we go and we get our nice license files. Nothing depends on the license files. But clearly, you know, Rust-Ran 06 depends on it, Firefox, I assume, depends on it just from, you know, everything pulling in libc. So some of it also is to, to bring down the compile time of the, of each, of each individual crate so that it's not, you see, on my desktop at home, compiling C bind gen for building Firefox takes about 10 minutes. So thinking if I went and actually grabbed all the build artifacts and just put them in place, actually building C bind gen should be about 30, 40 seconds. Okay. If you guys want to continue with more questions, we can take it outside. But yeah, let's thank a friend.
 Okay, so the fun part is I ran out of laptop, so I'm down to my G4 with the micro VGA port,  so yeah, so okay, so that'll be fun later also.  So packaging crusts.  Packaging crates, how hard could it possibly be?  So I've been using Geeks for a couple years now.  I don't really do a lot of programming myself, I like watching everything build, I like having  all the packages actually work.  So I guess my first interest in Rust really was when we started needing it for Firefox  or for the GNU fork of it for Icecat.  So a quick history of I guess how the two run together.  Geeks is a source-based distribution, so we did have over here in, since 2016 we went  and just grabbed the binaries from Rust and installed those and then used those to build  up to all the later versions.  And then later we went and removed those and switched to using MRust C for bootstrapping  Rust.  And so for Icecat we just go ahead and use all of the crates that are already rendered  in with Mozilla.  The other ones we've gone and started adding them in, it's looking at the timeline it's  been more recent, we've less than a year that we've actually had a number of crates in.  So in Geeks when we go and we build packages, everything gets installed to its own unique  space in the store.  So this makes it so that you can have multiple versions of every package installed.  This one would add not, I mean it's on the disk, it's not actually installed installed.  It gets referenced by whatever uses the specific version of SQLite.  So you have the binary, you have the headers, the libraries, the man page has all of the  normal stuff that goes with it.  And so the problem we were running into with Rust was go ahead, we build, we install and  we get the license file.  So it's saying okay, so what do we do with this?  So yes, I remembered why I put that there, I'm sure that would have been helpful.  So with crates, with Rust, it's very friendly for hacking on, go you say Rust build, say  cargo build, it builds in all the dependencies, it builds everything that it needs in it,  you get your final binary, you don't actually get a whole bunch of libraries installed with  it.  So we've been loving the API for crates.io, we have all the information, all of the, looks  like this part here doesn't, yeah, have all of the dependencies listed on it for the different  versions, whether it's needed for building it, for dev, mostly for testing or just in  general.  So we, with this we've created a recursive importer so it'll import the crate, it'll  fill out the template basically of what version of it, where do you download it from, what  are the different versions of everything that you need.  So big difference that we have with the crates, with the cargo build system is here under  arguments everything is a cargo input, whereas for, say for a news boat, we have native inputs  and we have inputs and so here it's, you know, compile all of the different libraries and  then they get linked in with the cargo dependencies, we're pulling in the source itself and compiling  everything together.  So I guess, yeah, there's our friendly cat, what's the difference?  So, yes, like we saw before, it's, you know, we're not, you know, not vending code, we're  not, you know, when we do have vending code we go ahead and pull it out and link to other  system libraries or in case with the crates which I guess don't actually vendor the code  but it's, you know, grab the source from here, grab the source from there, put everything  in and build it, we go ahead and basically mirror the same thing, we download all of  the sources ahead of time, we override the, forget exactly what's called the cache, yes,  where all of the crates are cached with our own cache directory inside the build tree  and say, you know, look over there, that's where all of the crates are for downloading  it.  So as a typical distro we prefer the shared library option where we go ahead and compile  it, give it all of the options we can, link to it everywhere, the shared source, you know,  we're working with it, it's not our favorite one.  I am definitely liking the size of the binaries that come out from the Rust crates and, again,  not vended code.  So one of the things that, it was the past couple days we had a small geeks hacking session  and a number of us trying to work on how to actually work on the Rust build system and  to try to, I guess, actually test the crates that we're building better, things like that  and just had a screenshot from the DEPPS file in the build directory.  And, yeah, so, you know, we see SerdeyDirive has an SO file and, you know, have over here,  this one was for Serdey itself, so, I mean, this one looks like it would be the binary  and then a lot of our lib, our metadata ones, the.d, I'm pretty sure were text files that  just said how to build it.  But you know, basic thing that we got was none of us really programmed in Rust, none  of us really knew.  It was kind of a, well, instructions are cargo build and it builds it and it downloads everything  and so what do we do with it?  So we were trying to figure out, you know, can you reuse parts of this?  Can you reuse the R lib?  I actually went and tried to model the actual build process and say, okay, well, you know,  I've gone and installed the different R lib into the lib directory and say, you know, when  you build, go and look for it over there and pull it in and didn't actually find that,  didn't actually work for us.  I mean, this one I tried a while ago, it just ended up not working so well in the end.  And so, yeah, so where, you know, at this point we have a couple hundred crates, most  of them we, you know, as part of the build process, we actually go and we patch all of  the source shibings anywhere in the source.  We regenerate the cargo.lock file and we build everything, you know, of the dependency trees.  You know, we look at this and say, you know, none of it looks really like something that  we would go ahead and install.  So for binary, binary, those ones are easier.  We actually do install those.  So yeah, so we were, you know, I guess, you're wondering if, you know, I also wanted to know  if anyone else, you know, is our other people trying to figure out how to reuse the, you  know, see the build artifacts or the generated libraries.  One of the things that talking with looking around online was that there's no stable  ABI from version to version.  We want to make sure that we actually rebuild the crates every time there's a rust version  change.  So Geeks is declarative, so we know exactly which versions of the crates are going in.  We know exactly which compiler is going in.  Anytime something gets updated, you know, all the packages get rebuilt.  So we're sure that we have the same compiler each time.  You know, we're really aiming to go for a, you know, for the same shared library type  of thing that we had with other ones.  Yeah.  Yeah, thanks.  So, yeah, so I guess you're not just looking for questions also if anyone has, I guess,  ideas or suggestions on, you know, other ways to go and, okay, if anyone has other ideas  on how to go and I guess reuse, I guess, either shared libraries or build artifacts,  even if it's just, you know, taking the build artifacts, storing them somewhere and pulling  them back in again to the dependency folder.  It's time to build the next crate.  You know, if anyone's, I guess, worked with, you know, I guess trying to create a cache  of the different files.  Are there any other distributions looking into this, like a few of them that maybe the  TVMP or something?  Yeah.  So the question was, are there other people looking into it?  See the Debbie and Guy nodding yes.  I assume Nix is also looking into it.  They have a pretty similar model of everything is, it depends on exact versions going all  the way down to the bottom of the bootstrap tree.  So some reason I haven't actually looked to see what they've done yet.  We should definitely have a talk about this.  Yeah.  I guess my quick look at what Debian does is have here with the, much more responsive  than my computer, with the different features provided.  I saw that Debian actually produces crates for, I think they're, I don't know if they're  crates or source crates for the different, for the different features that the different  crates provide.  Right now we just provide, as part of the build and test, we just pull in all of the crates  listed and build and then not install the libraries.  I'm a Gen2 developer.  I'm also a Rust developer but not at the same time.  But I do know that Gen2 is looking into this.  With Gen2 packages there can be a certain amount of overhead for maintaining each version  of the package.  You know, key wording of the various architectures.  And we also have the concept of a stable and stable.  Do you find with the amount of churn in the Rust ecosystem, is it scalable to maintain  this many packages and this many versions?  Yes.  So the question was, I guess there's a lot of versions, a lot of churn in the packages.  And I guess, well, this Gen2 has the use flags and the stable and stable we don't have that.  We just have, I guess it's either included or it's not included.  And does it become too many packages?  So for us, for say Rust-Ran 0.6, we just have a package Rust-Ran 0.6.  For the Rust packages, for the other ones, say for NewsBoat, it's just NewsBoat.  The new version, it gets bumped for Rust-Ran.  If there's a 0.6.6, then 0.6 becomes 0.66 in the version field.  We have 0.7, which is, in there is 0.7, we have 5, 4, and 3 also.  So we just have tons and tons of packages.  We, I guess it's not as bad as our Python system yet, but that's really because we only have,  I think, four or five actual Rust programs.  So for us, the problem is more of, yes, the way the way Geeks is architected, actually  compile all of the, not all of the package, but all of the package definitions so that  we don't have to interpret it every time.  So the file with the Rust packages is five, 600 packages long.  It starts to get big.  It's about time for us to start splitting it into multiple files already.  So in terms of having multiple versions, 0.5 would basically, we just add here between  package and name a field that says inherit from the previous version of Rust.  And then we just change the input.  So it's in the description, doesn't change.  The license generally doesn't change.  We just say, well, this version has different dependencies.  And when I say go ahead and build it, it just figures it out itself.  Do you know how Haskell does it in either Davion or Nix?  Because I believe they have some problems, but I don't know Rust good enough.  In Haskell, so first of all, there is an upstream stackage repository, which is a universe  of libraries that gets published every quarter, and then all those libraries are tested to  be compatible, and Debian just uses this snapshot, and then it upgrades all the Haskell libraries  in Davion.  Are they updated in lock stack?  So the question was, how does this compare with Haskell with having this with stable  release?  Right.  Let's see.  In Geeks, we also just go and basically pull in version from, I haven't spent a lot of  time with the Haskell ones.  We stick with the stable release more or less to try to just make sure everything actually  compiles together.  I know Nix has a lot more Haskell developers, and so they're tooling around Haskell is a  lot better than ours.  I guess in some ways, especially when it comes to the bootstrap binaries, Rust in Geeks is  doing better than Haskell, and that we're actually bootstrapping from C, and we haven't  figured out how to do that yet with the Haskell.  That's just a matter of time until people will do it in Rust.  My complaint about the long compile times to compile all the versions of Rust to make  the Haskell compiler, but if it's doable, we'd be happy to have it.  Yeah.  First, thanks for the presentation.  Sorry.  Sorry.  It's, yeah.  Yeah, some of it is just, yeah, the green doesn't show up as well with the light one,  and yeah, it's, yeah, definitely something I should make sure the colors work better  next time.  Yeah?  The question I have is, what are you trying to, or what's the benefit you expect from  caching the artifacts?  The R-lib?  Yeah, so the question is, what's the benefit of caching the R-libs?  I guess, so right now we go and pull in all the sources and just build everything all  at once.  What we'd love to be able to do is to build each, each create one by one, and actually  create the dependency tree and say, you know, these, these packages depend on those packages  and to actually be able to track, you know, when I upgrade Rust-lib C, you know, which  creates need to be updated, which, you know, which packages do I need to test to see that  they actually work?  And right now, when we go and we, I mean, as part of the, I guess, in the Geeks tooling,  we go and say, you know, say, what, what packages depend on this other package?  If I type in what depends on Rust-lib C, nothing depends on it.  You know, if we, if we go ahead and compile and install it, we go and we get our nice  license files.  Nothing depends on the license files.  But clearly, you know, Rust-Ran 06 depends on it, Firefox, I assume, depends on it just  from, you know, everything pulling in libc.  So some of it also is to, to bring down the compile time of the, of each, of each individual  crate so that it's not, you see, on my desktop at home, compiling C bind gen for building  Firefox takes about 10 minutes.  So thinking if I went and actually grabbed all the build artifacts and just put them  in place, actually building C bind gen should be about 30, 40 seconds.  Okay.  If you guys want to continue with more questions, we can take it outside.  But yeah, let's thank a friend.
rust_packaging_gnu_guix_0_191.png rust_packaging_gnu_guix_192_623.png rust_packaging_gnu_guix_624_2627.png rust_packaging_gnu_guix_2628_3563.png rust_packaging_gnu_guix_3564_4007.png rust_packaging_gnu_guix_4008_4115.png rust_packaging_gnu_guix_4116_5231.png rust_packaging_gnu_guix_5232_5675.png rust_packaging_gnu_guix_5676_6899.png rust_packaging_gnu_guix_6900_7127.png rust_packaging_gnu_guix_7128_10943.png rust_packaging_gnu_guix_10944_12983.png rust_packaging_gnu_guix_12984_14639.png rust_packaging_gnu_guix_14640_17075.png rust_packaging_gnu_guix_17076_17123.png rust_packaging_gnu_guix_17124_17183.png rust_packaging_gnu_guix_17184_19091.png rust_packaging_gnu_guix_19092_25259.png rust_packaging_gnu_guix_25260_25391.png rust_packaging_gnu_guix_25392_25439.png rust_packaging_gnu_guix_25440_25511.png rust_packaging_gnu_guix_25512_25559.png rust_packaging_gnu_guix_25560_27683.png rust_packaging_gnu_guix_27684_27731.png rust_packaging_gnu_guix_27732_27887.png rust_packaging_gnu_guix_27888_29507.png rust_packaging_gnu_guix_29508_29623.png