Automating deployment with Hercules CI and NixOps
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hensing, Robert
Introduction to Hercules CI, NixOps and a step by step demonstration of setting up a continuous delivery pipeline with these tools. Hercules CI is a continuous integration service that coordinates build agents on your own infrastructure. It is built around Nix for the best developer user experience. NixOps is the Nix ecosystem's primary deployment tool, supporting multiple clouds through its Nix-based input format. By integrating the two, we achieve a continuous delivery pipeline that is configured entirely with Nix.
 Hello, we are back with automating deployments with Hercules CI and NixOps. And you know, it's actually funny because a while back I did an Office R with Robert on the same topic. So this talk is about, is an introduction to Hercules CI, NixOps in a step-by-step demonstration of setting up a continuous delivery pipeline with these tools. Really cool. So about Robert. He started using Nix in 2015 and as you've heard, he is the founder of Hercules CI, the CI CD service for Nix-based projects. He also wrote Arian, a Docker compose wrapper for Nixify local development. He also wrote KTignore.Nix and he co-maintains the NixOS module system, pre-commit hooks and Docker tools. Okay, with that out of the way, you can go ahead, Robert. All right, thank you, Will. Sorry, one second. I should also mention. Robert is having a bit of issue with background noise, so he may be a little bit quiet. He's going to do his best to be as loud as possible so we can hear him. All right, thank you, Will. Yeah, so first of all, thank you so much for organizing this event. It's been excellent so far. Yeah, so let's get started. So we'll be looking at NixOps first. We have a very short introduction. Now I'll explain a bit about Hercules CI and then we'll have a look at using Hercules CI effects to achieve continuous deployment. Yeah, so it won't be exactly the step by step demo that I promised, I guess. I made a bit of a last-minute decision to, well, not do a live demo. But instead I'll give a tour of how effects can be used and specifically the run NixOps function. We'll see some of the internals and how to use it. And I'll end with a short section about the future of Hercules CI. So NixOps is the deployment tool in the Nix ecosystem. There's other deployment tools, but this is the regular one. You can declare your deployment by specifying the nodes. You can specify cloud resources as well. And the attributes of those cloud resources like IP addresses of external services and whatnot can be inserted into your NixOS configurations. So it gives you a lot of flexibility with writing your deployments. So this is a host name and this is just a NixOS configuration. And some more of that resources will go into the NixOps file. And then there's the NixOps command, which you can use to actually deploy it and perform operations tasks like SSH to machines or copying files, that kind of thing. And it's been around for a while. And I think it kind of shows. But there's also some progress being made with NixOps 2.0 so that there's a bunch of commands that work with the local state of NixOps. You can have multiple deployments, which you can list, create more of them. And those deployments in the local state, they know where the Nix files are. So you may have to modify those deployments in various ways. And well, it's obviously from before mixed lakes, so it also relies on NixOps. So these are going to be kind of challenges, I guess, when writing an effect for them. So Hercule-CI is the Nix-based continuous integration service. Of course, there's other solutions as well, like Hydra. But the benefit of Hercule-CI is that you don't have to host the server, which is arguably the most maintenance-intensive part of the CI system. Instead, it works with just build agents, build agents contact the service in cloud, and pick up work from there. So it's a nice low maintenance setup. It looks like this. In your repo, you write a Nix file, CI.Nix, any packages that you put in there, they will be built, and automatically upload it to Binarycache if you have one. And the new feature is effects for stuff like continuous delivery. We have live logs nowadays. So effects are written using derivations, but they're not quite derivations. It's actually similar to how Nix Shell isn't supposed to be built. It's basically the same trick that we're using here to declare how some operations should be executed. It's executed in its own sandbox, which has network access, completes access to the root file system. It can read secrets, which will be configured on the agent machines. So the CI service doesn't need to know about them. It's great for security. And they also have access to a state API, which is useful for, for example, the Nix update that I mentioned earlier. It's going to be a tour. We're going to see quite a bit of code if you don't have to understand everything. So I thought it was a good idea to basically show that it's really not that complicated. That's why I point here that we're just using a function that's been written already. So basically, too, to have an effect. You create an attribute and you assign the result of the runNixHot function to it. That means it will be picked up by the agent. So the build succeeds, the effects will run. Yeah, so you can see the basic arguments here. You specify the source and which files to actually load and secrets and, yeah, a little bit of setup, and that's about it. So like I mentioned, they're basically derivations, but we're using a different set of phases. And there's a special attribute to mark the effects as such. So these are the essential ones. You can have a source attribute. That's great if you need to do evaluation, because you can do evaluation inside the effect sandbox. So there's the standard unpack and fetch phase. And the effect phase, which is where most of the magic happens. And there's some additional phases for stuff like getting state files and writing them out. This also takes care of when there's an error. You probably don't want to put the state back. Well, it depends on your deployment strategy. So let's start with the state files. Practiciary effects, when you're using the library for it, you also get a bunch of bash functions to get the state file from the server and put it back in your own. You can use the hook system from standard end to add some extra behavior to it. So in this case, with KnicksOps, we want to import that state into the KnicksOps state file. So this is an export. Probably should have a problem with export. We're treating it as the only state of the deployment. So we import it, we export it. Actually a bit more complicated, because when it's not there yet, we want to create it. Well, we have to deal with the modifying the setRx. But that's all taken care of. Yeah, so the state files, they're uploaded to the server. Actually they are not encrypted on the agent. Yes, that's something I'm going to build. So for now, it's usable, but it will be even more secure. So they're visible here. You can change them. You can download the state file. There's also a history of each state file. So for example, if your state is corrupted for whatever reason, you'll be able to refer to a known valid state. All right. So KnicksOps secrets. We always have to deal with cloud providers and authentication for them. It's unavoidable to have those. They are configured on the agent in the file. It's a JSON file. And it looks like this. It's a tiny bit boilerplating, because basically here there's going to be a way to configure policies. So not every repo can access every, well, not every branch can access every secret. So the secrets can be structured data, or you can just have a little problem here. We'll talk more about that later. So effects library provides a bunch of batch functions to write these secrets to the locations where they're expected. That's useful because it doesn't look like much of an hassle, typically, at least when I'm writing this kind of thing. I forget about a make their call and I get the formatting wrong and whatever. So I think it's really nice to have these helper functions to make it just a little bit nicer. So that's actually sufficient to get a deployment up. Yeah, but we're now running effects on every run. So that means that all your feature ventures are going to be deploying to your environment all the time. So we need a solution for that. And luckily we have a programming language at our disposal. So we can just program a solution. So this is the run if function. Basically you give it a boolean and if it's true it's going to actually run the effect. And if it falls, it'll just build the dependencies. So when you integrate that, it'll look like this. The master version of the agent passes the web of the source. So you can check whether it's the right branch and then only actually deploy when you're building from master. And that'll look like this. I don't think the example of the pre-built attribute, that's also in there. So you can actually see when you're working on a feature that's not deploying, you can see whether it's the affiliate in your deployment specification or perhaps the build tooling has a build failure, which is also possible. You can see at a glance which one is the case. So deployment succeeded. But now we can't really use NixOps because we don't have the state file yet. You can download it from the UI, but that's going to be rather tedious. So instead we can use the Hercules CLI. You just need to log in. So HCI log in, you get a link and you click the authorize button and you can get your state on the command phone. I think this works and this syntax is great for scripting, but in the end what you really want is to have this behavior in NixOps, which is kind of close, I guess. I've seen the work on NixOps too. That also includes support for remote state files. That's still in progress. So I hope I can help out with that and really make this super easy so you don't have to write your own scripts to get this state and put it back if it changes so that I can all be automatically just have to pull this as usual. So yeah, to conclude, we now have a setup where we have a built agent that's running locally on the network and we can deploy from there. We don't have to maintain the CI server and we have a deployment configuration that's all Nix. So no YAML for taking care of. Work with CI just uses the KITL permissions. So to work with the state files, for example, you need write access to a repo. So just really not much to configure there. So effects aren't quite done yet. They're a beta feature. But the top priority right now is documentation. This effects feature is really fresh. So yeah, the next thing I'll be working on is making some excellent documentation for this. The goal is to make the easiest CI CD system for Nix users. I'll be making improvements to the secrets. So I mentioned that the state file isn't encrypted yet. So what I really want to do is encrypt both the state files and secrets using asymmetric cryptography. So it'll be much easier to share those secrets and make the state files more secure. I'll be writing more effect functions. Right now we have functions for Nixops, Nix Darwin and Arial. So I'm planning to also write one for Terraform soon and a simple NixOS deployer. You don't always need to use Nixops for deployments if you're just deploying one machine or maybe a couple of them. You can, if those machines already exist, you can just use NixOS rebuild. So I'm going to deploy to them by the SSH. I'll be making more improvements to the deployment class title. Like the possibility to run effects when the branch is deleted, which might be useful. So who requests? Can be modeled that way. Yeah. So I guess that's the future for effects and for Herculeci. It's been a bit of a thing for a while, but it's the obvious one. Herculeci has been a bit on the expensive side. I think right now it's best to increase adoption. The only way to do that is with competitive pricing. So prices will be about half. I'll announce the exact prices soon. We have a trial period of one month, so the prices will be updated well within that period. I think in the next week, it will definitely be on the side. On a higher level, I think it would be great to invest more in the Nix community. I've been doing so with some requests and whatnot, reviewing, co-maintaining some stuff. Hey Robert, we have two minutes off for Q&A. All right, thanks. So I hope that Herculeci will be successful. So I can invest more in the Nix community. I guess what I'm trying to say is both ways. Yeah. So I think we have time for maybe one or two questions. The first one I will tell you is, does your threat model for secrets explicitly address the case of correctly building malicious PRs with functionality designed to steal your secrets? And if so, how? I hope that's an okay question. So that's not built yet. Right now the threat model is basically to trust everyone with right access to the report. If you're using secrets with a public repo, actually external requests from folks, they won't be built. So as long as only trustworthy collaborators are on the repo, it should be fine. That's not the end state of course. I've mentioned the policies feature. I'm going to build that to make it possible to enforce that only certain branches or certain people can actually make use of those secrets. Right. Okay. And the last question we have is, what is the risk for vendor lock in when using your system? I don't think that's a great risk. Because well, it's really just mixed code. You're right in mixed code. So definitely your derivations are just regular ones and with effects. Well, I guess for HCI system this is different. I'd be happy to collaborate on some sort of standardization effort if people are interested in that kind of thing. Right. So I don't think the lock in is any more than any other. I would agree. Yeah. Thank you so much. We are out of time. I really enjoyed your talk. The interface you have going on here for developers is really nice. It's really sleek, such as the bash functions. You made it pretty easy for people to use this. Oh, good to hear. Thank you. Yep. Thank you very much. So everyone, please remember, put those ones, clap, show the love for this wonderful, wonderful talk. And I have one announcement after this is that I'm following this talk. We are going to be doing lightning talks. Okay. And we are cutting to the five minute break. See you then.
 Hello, we are back with automating deployments with Hercules CI and NixOps.  And you know, it's actually funny because a while back I did an Office R with Robert  on the same topic.  So this talk is about, is an introduction to Hercules CI, NixOps in a step-by-step demonstration  of setting up a continuous delivery pipeline with these tools.  Really cool.  So about Robert.  He started using Nix in 2015 and as you've heard, he is the founder of Hercules CI, the  CI CD service for Nix-based projects.  He also wrote Arian, a Docker compose wrapper for Nixify local development.  He also wrote KTignore.Nix and he co-maintains the NixOS module system, pre-commit hooks  and Docker tools.  Okay, with that out of the way, you can go ahead, Robert.  All right, thank you, Will.  Sorry, one second.  I should also mention.  Robert is having a bit of issue with background noise, so he may be a little bit quiet.  He's going to do his best to be as loud as possible so we can hear him.  All right, thank you, Will.  Yeah, so first of all, thank you so much for organizing this event.  It's been excellent so far.  Yeah, so let's get started.  So we'll be looking at NixOps first.  We have a very short introduction.  Now I'll explain a bit about Hercules CI and then we'll have a look at using Hercules  CI effects to achieve continuous deployment.  Yeah, so it won't be exactly the step by step demo that I promised, I guess.  I made a bit of a last-minute decision to, well, not do a live demo.  But instead I'll give a tour of how effects can be used and specifically the run NixOps  function.  We'll see some of the internals and how to use it.  And I'll end with a short section about the future of Hercules CI.  So NixOps is the deployment tool in the Nix ecosystem.  There's other deployment tools, but this is the regular one.  You can declare your deployment by specifying the nodes.  You can specify cloud resources as well.  And the attributes of those cloud resources like IP addresses of external services and  whatnot can be inserted into your NixOS configurations.  So it gives you a lot of flexibility with writing your deployments.  So this is a host name and this is just a NixOS configuration.  And some more of that resources will go into the NixOps file.  And then there's the NixOps command, which you can use to actually deploy it and perform  operations tasks like SSH to machines or copying files, that kind of thing.  And it's been around for a while.  And I think it kind of shows.  But there's also some progress being made with NixOps 2.0 so that there's a bunch of  commands that work with the local state of NixOps.  You can have multiple deployments, which you can list, create more of them.  And those deployments in the local state, they know where the Nix files are.  So you may have to modify those deployments in various ways.  And well, it's obviously from before mixed lakes, so it also relies on NixOps.  So these are going to be kind of challenges, I guess, when writing an effect for them.  So Hercule-CI is the Nix-based continuous integration service.  Of course, there's other solutions as well, like Hydra.  But the benefit of Hercule-CI is that you don't have to host the server, which is arguably  the most maintenance-intensive part of the CI system.  Instead, it works with just build agents, build agents contact the service in cloud,  and pick up work from there.  So it's a nice low maintenance setup.  It looks like this.  In your repo, you write a Nix file, CI.Nix, any packages that you put in there, they  will be built, and automatically upload it to Binarycache if you have one.  And the new feature is effects for stuff like continuous delivery.  We have live logs nowadays.  So effects are written using derivations, but they're not quite derivations.  It's actually similar to how Nix Shell isn't supposed to be built.  It's basically the same trick that we're using here to declare how some operations should  be executed.  It's executed in its own sandbox, which has network access, completes access to the root  file system.  It can read secrets, which will be configured on the agent machines.  So the CI service doesn't need to know about them.  It's great for security.  And they also have access to a state API, which is useful for, for example, the Nix  update that I mentioned earlier.  It's going to be a tour.  We're going to see quite a bit of code if you don't have to understand everything.  So I thought it was a good idea to basically show that it's really not that complicated.  That's why I point here that we're just using a function that's been written already.  So basically, too, to have an effect.  You create an attribute and you assign the result of the runNixHot function to it.  That means it will be picked up by the agent.  So the build succeeds, the effects will run.  Yeah, so you can see the basic arguments here.  You specify the source and which files to actually load and secrets and, yeah, a little  bit of setup, and that's about it.  So like I mentioned, they're basically derivations, but we're using a different set of phases.  And there's a special attribute to mark the effects as such.  So these are the essential ones.  You can have a source attribute.  That's great if you need to do evaluation, because you can do evaluation inside the effect  sandbox.  So there's the standard unpack and fetch phase.  And the effect phase, which is where most of the magic happens.  And there's some additional phases for stuff like getting state files and writing them out.  This also takes care of when there's an error.  You probably don't want to put the state back.  Well, it depends on your deployment strategy.  So let's start with the state files.  Practiciary effects, when you're using the library for it, you also get a bunch of bash  functions to get the state file from the server and put it back in your own.  You can use the hook system from standard end to add some extra behavior to it.  So in this case, with KnicksOps, we want to import that state into the KnicksOps state  file.  So this is an export.  Probably should have a problem with export.  We're treating it as the only state of the deployment.  So we import it, we export it.  Actually a bit more complicated, because when it's not there yet, we want to create it.  Well, we have to deal with the modifying the setRx.  But that's all taken care of.  Yeah, so the state files, they're uploaded to the server.  Actually they are not encrypted on the agent.  Yes, that's something I'm going to build.  So for now, it's usable, but it will be even more secure.  So they're visible here.  You can change them.  You can download the state file.  There's also a history of each state file.  So for example, if your state is corrupted for whatever reason, you'll be able to refer  to a known valid state.  All right.  So KnicksOps secrets.  We always have to deal with cloud providers and authentication for them.  It's unavoidable to have those.  They are configured on the agent in the file.  It's a JSON file.  And it looks like this.  It's a tiny bit boilerplating, because basically here there's going to be a way to  configure policies.  So not every repo can access every, well, not every branch can access every secret.  So the secrets can be structured data, or you can just have a little problem here.  We'll talk more about that later.  So effects library provides a bunch of batch functions to write these secrets to the locations  where they're expected.  That's useful because it doesn't look like much of an hassle, typically, at least when  I'm writing this kind of thing.  I forget about a make their call and I get the formatting wrong and whatever.  So I think it's really nice to have these helper functions to make it just a little  bit nicer.  So that's actually sufficient to get a deployment up.  Yeah, but we're now running effects on every run.  So that means that all your feature ventures are going to be deploying to your environment  all the time.  So we need a solution for that.  And luckily we have a programming language at our disposal.  So we can just program a solution.  So this is the run if function.  Basically you give it a boolean and if it's true it's going to actually run the effect.  And if it falls, it'll just build the dependencies.  So when you integrate that, it'll look like this.  The master version of the agent passes the web of the source.  So you can check whether it's the right branch and then only actually deploy when you're  building from master.  And that'll look like this.  I don't think the example of the pre-built attribute, that's also in there.  So you can actually see when you're working on a feature that's not deploying, you can  see whether it's the affiliate in your deployment specification or perhaps the build tooling  has a build failure, which is also possible.  You can see at a glance which one is the case.  So deployment succeeded.  But now we can't really use NixOps because we don't have the state file yet.  You can download it from the UI, but that's going to be rather tedious.  So instead we can use the Hercules CLI.  You just need to log in.  So HCI log in, you get a link and you click the authorize button and you can get your  state on the command phone.  I think this works and this syntax is great for scripting, but in the end what you really  want is to have this behavior in NixOps, which is kind of close, I guess.  I've seen the work on NixOps too.  That also includes support for remote state files.  That's still in progress.  So I hope I can help out with that and really make this super easy so you don't have to  write your own scripts to get this state and put it back if it changes so that I can all  be automatically just have to pull this as usual.  So yeah, to conclude, we now have a setup where we have a built agent that's running  locally on the network and we can deploy from there.  We don't have to maintain the CI server and we have a deployment configuration that's  all Nix.  So no YAML for taking care of.  Work with CI just uses the KITL permissions.  So to work with the state files, for example, you need write access to a repo.  So just really not much to configure there.  So effects aren't quite done yet.  They're a beta feature.  But the top priority right now is documentation.  This effects feature is really fresh.  So yeah, the next thing I'll be working on is making some excellent documentation for  this.  The goal is to make the easiest CI CD system for Nix users.  I'll be making improvements to the secrets.  So I mentioned that the state file isn't encrypted yet.  So what I really want to do is encrypt both the state files and secrets using asymmetric  cryptography.  So it'll be much easier to share those secrets and make the state files more secure.  I'll be writing more effect functions.  Right now we have functions for Nixops, Nix Darwin and Arial.  So I'm planning to also write one for Terraform soon and a simple NixOS deployer.  You don't always need to use Nixops for deployments if you're just deploying one machine or maybe  a couple of them.  You can, if those machines already exist, you can just use NixOS rebuild.  So I'm going to deploy to them by the SSH.  I'll be making more improvements to the deployment class title.  Like the possibility to run effects when the branch is deleted, which might be useful.  So who requests?  Can be modeled that way.  Yeah.  So I guess that's the future for effects and for Herculeci.  It's been a bit of a thing for a while, but it's the obvious one.  Herculeci has been a bit on the expensive side.  I think right now it's best to increase adoption.  The only way to do that is with competitive pricing.  So prices will be about half.  I'll announce the exact prices soon.  We have a trial period of one month, so the prices will be updated well within that period.  I think in the next week, it will definitely be on the side.  On a higher level, I think it would be great to invest more in the Nix community.  I've been doing so with some requests and whatnot, reviewing, co-maintaining some stuff.  Hey Robert, we have two minutes off for Q&A.  All right, thanks.  So I hope that Herculeci will be successful.  So I can invest more in the Nix community.  I guess what I'm trying to say is both ways.  Yeah.  So I think we have time for maybe one or two questions.  The first one I will tell you is, does your threat model for secrets explicitly address  the case of correctly building malicious PRs with functionality designed to steal your  secrets?  And if so, how?  I hope that's an okay question.  So that's not built yet.  Right now the threat model is basically to trust everyone with right access to the report.  If you're using secrets with a public repo, actually external requests from folks, they  won't be built.  So as long as only trustworthy collaborators are on the repo, it should be fine.  That's not the end state of course.  I've mentioned the policies feature.  I'm going to build that to make it possible to enforce that only certain branches or certain  people can actually make use of those secrets.  Right.  Okay.  And the last question we have is, what is the risk for vendor lock in when using your  system?  I don't think that's a great risk.  Because well, it's really just mixed code.  You're right in mixed code.  So definitely your derivations are just regular ones and with effects.  Well, I guess for HCI system this is different.  I'd be happy to collaborate on some sort of standardization effort if people are interested  in that kind of thing.  Right.  So I don't think the lock in is any more than any other.  I would agree.  Yeah.  Thank you so much.  We are out of time.  I really enjoyed your talk.  The interface you have going on here for developers is really nice.  It's really sleek, such as the bash functions.  You made it pretty easy for people to use this.  Oh, good to hear.  Thank you.  Yep.  Thank you very much.  So everyone, please remember, put those ones, clap, show the love for this wonderful, wonderful  talk.  And I have one announcement after this is that I'm following this talk.  We are going to be doing lightning talks.  Okay.  And we are cutting to the five minute break.  See you then.
NixCon_2020_Vid_13_0_251.png NixCon_2020_Vid_13_252_467.png NixCon_2020_Vid_13_468_695.png NixCon_2020_Vid_13_696_851.png NixCon_2020_Vid_13_852_971.png NixCon_2020_Vid_13_972_1247.png NixCon_2020_Vid_13_1248_1679.png NixCon_2020_Vid_13_1680_2075.png NixCon_2020_Vid_13_2076_4331.png NixCon_2020_Vid_13_4332_6695.png NixCon_2020_Vid_13_6696_7847.png NixCon_2020_Vid_13_7848_7931.png NixCon_2020_Vid_13_7932_7979.png NixCon_2020_Vid_13_7980_8027.png NixCon_2020_Vid_13_8028_9647.png NixCon_2020_Vid_13_9648_11219.png NixCon_2020_Vid_13_11220_20423.png NixCon_2020_Vid_13_20424_20663.png NixCon_2020_Vid_13_20664_21131.png NixCon_2020_Vid_13_21132_26891.png NixCon_2020_Vid_13_26892_27971.png NixCon_2020_Vid_13_27972_28811.png NixCon_2020_Vid_13_28812_30035.png NixCon_2020_Vid_13_30036_30479.png NixCon_2020_Vid_13_30480_32447.png NixCon_2020_Vid_13_32448_32699.png NixCon_2020_Vid_13_32700_35435.png NixCon_2020_Vid_13_35436_36827.png NixCon_2020_Vid_13_36828_37403.png NixCon_2020_Vid_13_37404_38183.png NixCon_2020_Vid_13_38184_38327.png NixCon_2020_Vid_13_38328_39179.png NixCon_2020_Vid_13_39180_39287.png NixCon_2020_Vid_13_39288_39347.png NixCon_2020_Vid_13_39348_39407.png NixCon_2020_Vid_13_39408_39493.png