Migrating a Hosting Infrastructure from Gentoo
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kauhaus, Christian
I would like to share our experiences while migrating a VM-based hosting infrastructure at Flying Circus Internet Operations from Gentoo to NixOS. It is a case study of migrating 500+ VM running diverse customer projects using a wide range of web technologies. I'll cover the following main points: 1. What was our motivation to change to NixOS? Which specific features of NixOS attracted us? 2. Creating a NixOS-based hosting platform - What did we have to implement? What is still missing? 3. What are the pain points? How could NixOS become more attractive? 4. What do our users think? - benefits and stumbling blocks. - Bio: Christian is a systems engineer working with Flying Circus Internet Operations in Halle (Saale), Germany. He works mainly on infrastructure which keeps customers' web applications running. Main programming languages are Python, Nix, Rust. Christian is a regular contributor to NixOS, most known for vulnix and vulnerability roundups.
 All righty, time for our next speaker. This is going to be Christian. And you might know Christian from his, sorry? Yes. And you might know Christian from his work that he has done on Valnex and on the vulnerability. Vulnerability, I can't even pronounce it. He did it, vulnerability roundups for NixOS. But today he is going to talk to us about this annoying situation in which you might end up where you're not actually yet running NixOS, but of course you want to. And he's going to explain us how to get out of the misery. Quick applause, of course, please. Thanks. Yeah, hi, everyone. So does it work? Yes. Sorry. OK, forget about it. Sorry. OK, as already, what's going on here? Used to be working. Sorry for the delay. OK, one of the silenced wrestling was his laptop. OK, we'll see. Sure. Okay. Sorry about that. Okay. As I already said, hi, my name is Christian. I'm Systems Engineer doing this for a bit more than 10 years. You can at least see it with the laser pointer. Yeah. Reach me by mail or find me at various social media or RSC or GitHub. And I'm wanting to talk about migrating a hosting infrastructure from Gen2 to LixOS and answer mostly two questions. First, what motivated us to migrate so you don't do that just out of a mood? And what experience did we make? So I'd like to share a bit of, well, real world insights so perhaps the community can profit from that. So when I say we, I mean Flagsorcus Internet Operations, we are the guys with the cool aircraft, but a bit more than that, we are a small company located in Halle, Zahler in Germany. And basically, we are taking care of customer applications. So there are customers which can't or won't concentrate on operational issues because for example, they are software shops or they are too small or whatever. And so they approach us for deployment concepts, for automation, for telemetry, for monitoring, for incident response, for load and resource management, for upgrades, strategies and much more. So some, someone said we are sort of DevOps for higher or DevOps as a service. So we are actually working closely together with the devs, usually on individual applications and we work together to keep the staff running. So actually for customer projects, we have more than 500 VMs with a lot of very diverse stuff installed and running on them. So why did we migrate? So where did we come from? We have been historically a gentle and puppet shop and before I am going to ditch this, it's important to state it went running for quite a while and it was not too bad. So during the time we started that, Gen2 was a good choice because of its extreme configurability and puppet was also a good choice because at that time I think it was one of the best tools you could get for automated systems management. But well, after doing that for 10 years or so, it got too complex. So puppet is not able to manage everything and so it was no problem. We are software engineers after all. So we say, well, let's write a script, let's write a wrapper, let's write a fix up and this is a partial view of our old system management stack. We see a lot of diverse steps and fixes and scripts and additional config runs and specific scripts for stuff and this is only one tenth of the whole thing. And what you see is that there are partly overlapping functionalities. So for example, puppet can manage 80% of something or Gen2 emerge, can do 90% of something but then we have to put another fix and another script and well, it was painfully slow. So for a full configuration run, we took more than five minutes on a typical VM. Too many moving parts so you perhaps have to emerge a package but then you have to run revdp rebuild. Don't know if anyone knows Gen2 and then you have a Python updater and then you have a Perl cleaner and so on and so on. And every step you add, you may get a bit closer to where you want but not really to the point and this small gap when you multiply that with 500 machines gets a real awful lot of work still to be solved manually and we didn't want to do that. So what was the problem? What is more like we did a bad job? So would we just have to improve the implementation or perhaps if the approach not optimal is our whole model of system management too weak? Perhaps yes and to answer this question we have to go a bit into theory. So perhaps some of you may know the real great paper by Steve Trager from NASA about why order matters and in this paper he defines three main models of system management. The divergent system management and congruent system management and I go to each of these in detail. So what does divergence mean? Aside from the paper divergence is characterized by the configuration of life hosts drifting away from any desired assumed baseline disc content. So what is this? For example this is when you configure a server by following a checklist. Let three admins do that. You get three different results but also like install our scripts for example while call to shell. Okay this is divergent system management but there's also a legitimate use for divergence. For example when you have user home dears or you have some sort of database content or any relevant productive data is by definition divergent. So if you wouldn't have that then you could burn your entire data center on DVDs and then it's static so no point about that. And the second model is the convergent model. So this is where most of the system management tools we have today live in. So you have some description of the desired state and then you measure the actual state and then you see where the delta is and then for each deviation you have a corrective action which puts the actual system state nearer to the desired system states of many things like puppet or ansible assault and you name it work this way. And this has also its use case and won't get away for example you have activation scripts in NixOS who are constructed exactly around this model or for example you have system these services which follow this model or when viewed as a larger scope you see for example a container orchestration which also follows this model. So this model is good but we can even get to a stronger model the congruent model. This means congruence is the practice of maintaining this state in complete compliance with a fully descriptive baseline. So for example system packages are exactly defined in every distribution and of course everything in the Nix star is exactly defined but also there are for example container images which are exactly defined and okay you could argue if several functions belong to that as well or not perhaps yes. So and well to get maximum control you want to make the congruent domain as large as possible and the other two domains were still there but just not growing too big. So we decided it's time to switch to another model we wanted to follow the congruent approach and the two interesting candidates for doing so were of course NixOS and of course going with a container thing. So well everyone is going with a container thing today but well they don't do well in a multi-tenant environment because they provide no great isolation and the other thing is that a container technology well has its strength but also has its weaknesses and if you read some larger Docker file then you know what I'm talking about. So what attracted us to NixOS I think in the first place the Nix language is so expressive that you can do nearly everything with it and this was of course very important for us as software developers. You get a lot of flexibility you can just mix everything together and all the dependencies are still in place and working binary substitution you get that basically for free when you manage to get Hydra running and while someone said Nix is a tool to rule them all so you have one approach to packaging, to systems management, to deployment and at least but the last the community is very approachable it was no problem for us to get into the community and for example place pull requests or something like that this is very great. Okay what did we do? So okay I'm just talking to some hackers here so the hacker would think no problem at all just install NixOS on the VMs and everyone will be happy so well every one of you would just start to put your whole project in one big default.nix and that would describe everything then you would just say Nix build and then the whole project just builds and everything is fine. Well this is what NixOS hackers think but we had a large installed base which was not constructed with NixOS in mind and we had to put that somehow over the fence and we'll see what's there how can we get that running and while doing so we talked to our customers we reviewed the code and we found out well not everyone is ready for NixOS so well NixOS sort of don't understand me wrong but it's a hacker thing it's technically very advanced so I like that very much but some of our users were frightened so they had quite a bit of hard time with this congruent or immutable approach so for example hey I want just to edit some etcc whatever file why can't I do that and where is the option for X and Z and do you have a GUI and what's going on here don't understand that and slash user bin is empty so where it's all of the stuff gone and I'm overwhelmed okay Nix Nix what's that funny language so I don't understand so can I program that like shell no I can't then I don't understand that so functional programming is very expressive once you get into it it's really fun and you don't want to go back and do something other but if you don't do that and for example you are more like an admin guy who started putting some CDs into some Microsoft Windows boxes then it's quite a steep learning curve that was for the one thing and the other thing well it depends on what you're running and we do a lot of web applications and they are well some of them insist of trying to build themselves in the very moment you are starting them up for example we've got some Node.js application which just starts to compile some C code in the moment you're trying to start that or we have some CMS applications who start to optimize themselves somehow don't know what they're doing but they I don't do know for sure they're going to fail if they cannot write the installation directory and we also have some applications which are sent out around this model of incremental build so for example some stuff which needs installed base and then needs to reinstall itself depending on the old installed base and if you try to install everything from scratch then it takes one and a half hours or so also some plugin system auto update systems and so on so well you could say that's all bad rule it out yeah but we are earning our money with running business critical customer applications and most of them have some tiny ugly core deep within which well isn't really good software design but is absolutely important and everyone knows okay no it's not the best but we cannot do without it and it's our job to keep it running so how to solve that obviously we cannot rewrite everything and nixify everything right away so we decided for flexible approach so we just took the application in the narrower sense so the custom code and see if that matches more convergent model or another model and we also separate components we also had that in our old gentle setup which bundle often mostly used things like engine x like post grass like riders like ma'am cashier like elastic search that many projects use and now we see which part of what goes into what model so the application deployment depends largely on the project we have some conventional deployment for example like ansible we didn't want to get other tools at that point we have even divergent stuff like data dears and but there are also congruent elements for example like customer supplied container images and of course there are a few already nixified projects and the color code is green is congruent yellow is convergent and red is divergent and as it comes to the premade components most of them have some glue code use some nix service modules which are in place already and have some integration points what does that mean these are directories or files where user deployments can typically put some snippets in the services the equivalent of fiddling with some config files under etc so I give an example for example our engine x component has an integration point this is some directory and you can just drop in some virtual hosting configuration files and so on and that stuff gets picked up by the nix as rebuild run using our glue code and gets incorporated in the configuration running in the nix store so that we still run everything out of the nix store but users find at least some points they can cope with so that was the plan was what was our experience for most projects with all of this in place it was quite doable and while you have to fit around a little bit but I think 90% of our customer projects went quite fine with that and we of course have some dependencies which used to come out of this out of the system installation on our jankto for example some libraries like SSL and so on and most of them are now placed into the user nix profile during whatever using whatever mechanism is fitting so it's in some project we installed them via nix and others have a default nix which builds an nf which is going to be installed there and in the end we see a large running code base mixing up nix code from as far back from 1509 until the current stuff and due to nix as we are able to mix it so it works out so this is really great so of course we made some tools to facilitate this our major wrapper and it's only one wrapper for everything is FC manage this is basically a thin layer above nix as we built and that runs regularly from a system the timer and pulse the channel sees if average changes occur and triggers them and then what's perhaps more interesting for the community it's a little tool called FC user scan that scans unmanaged installations for example user just compiles code in his home and that uses some libraries from the nix store but the nix store garbage collector doesn't know about that and deletes the libraries at some point and then the user code doesn't run anymore and so this scanner just goes through the installation and registers all the time dependency it finds with a nix store garbage collector and of course will next I think this is well known this scans an installation for open CVS so what were the main benefits from our point of view going from gen two to nix gave us significantly higher productivity so we can do stuff in no time which took days in the old environment we also have a test on the infrastructure level this quite good we got a lot more flexibility because we don't have to say okay this particular version of opens SSL is the system version and you have to use that and nothing else and also we see that we can scale things so for example most engine x-based conflicts are the same but also have the flexibility to bring in customer specific modifications without losing oversight so what do our users think well most users don't care so they just want to have some Linux server so well distro don't know so okay get it running that's fine so and we would like to have some more projects where customers are just shipping a default nix and say hey that's my default nix run that in production but it's only the case for maximum 10% of our projects so I think we profit more of the migration than our users profit directly but of course they profit indirectly because we are better off right now okay to finish on there are some things which well I would wanted to call the section pain points but I think there's a misnomer because working with nix is not painful at all it's fun so I called it things to improve and the security story so I've been busy providing vulnerability roundups every one week or two but when you look around you see quite a long list of open issues so I think we need a better approach to actually fixing stuff so discovering of what needs to be done is good but it's only first step the second step is to fix the stuff and in my opinion this is largely a problem of missing manpower so I was really happy to see a larger security team and I want to use Saturday I don't know for people who are still around there and just to see what could be done in the short term just to get this one better and we also should think about back porting important changes to older releases don't know if we need a formal process for that or just start by doing so informally the other point that is a constant source of well confusion with our customers is that nix likes to restart everything even for minor changes well it comes from the model if I change any input the hash changes the unit file changes and system D restarts the unit so when you see for example here for postgres it's well quite a problem because it terminates all connections free starts and then the application just has to reconnect and we've got one minute of useless downtime so also here the question is can we do better can we come up with some clever scheme to avoid that without breaking the overall nix-v-ass model and well the last point is I think the community has grown quite a bit over the last years nix-v-ass is attracting a larger user base this is great news but I think we have to keep up with community structure so from my perspective I would really be glad to see more teams with well clearly structures structured responsibilities well perhaps there are already exists some teams but then they are not well discoverable at least so I think this is a point where we can and should improve and so just to continue the success of nix-v-ass so as a final question if we would be in the same situation we've been three years ago right now would we choose nix-v-ass again probably yes I think it's a really great piece of software infrastructure thank you okay hello thanks so one of your slides was very interesting that just what first one of the first ones is nix-v-ass versus containers and well someone would have to ask that yeah yeah so with Docker and containers there is this huge pool about one containers runs one application and so on so it seems that it fits quite well that my understanding what you are doing because your customers have one application and they want to run one application in a container so would you like to elaborate a bit about why nix-v-ass for you was a bad choice yeah sure sure so well one point is that containers are only part of the solution because in a multi-tenant environment you have to provide in case encapsulation anyway and so we still need some management for the underlying underlying thing be it VMs whatever and we're not large enough to have dedicated hardware per customer and the other thing is that containers impose a specific way of deploying the application because you have to follow their standard or you aren't going to use it well and with our customer base this is not possible for example we have a lot we are historical a python shop and there are a lot of CC build out base projects and they don't work together very well in addition to that we are experimenting now with running containers inside VMs just to give the benefits of both worlds more question yes what was the first question yes so your 10% of your base of user base is already using default nix or kind of the nixify the project for you in a sense at least partly what is the difference in maintainability or between this 10% and let's say some some similar size project okay it largely depends on the project in my experience the maintainability is more dependent on the software code base as such so there are for example projects which save parts of the code in their databases and well that's a nightmare and and well we have quite good convergent deployment tools for example we use Batur this is an in-house tool or Ansible and other tools and if you do that well it works too and so and I think the major benefit of projects which are using nixified setup is that you've got the exact reproach usability on the developers computers so you know okay exactly what is running here will be running there I think this is the same promise that containers make and I think this is the main benefit for the most other projects we have a much more sort of staging setup with a lot of VMs that are solely there to see that everything still fits together in the deployment that we have that too for nixified projects but they don't need that much for staging second question so you remember do you still remember when you came back to your meeting with the idea let's try nix and see it and when your coworkers okay that was actually Domen the guy so and we had a sprint at then gocept this was the former company and we were supposed on hacking on some Python stuff and I know very well Domen just put there some nix stuff in and just said here look at that that's cool and we all looked at that and said that's weird but why you have to free yourself from 30 years of thinking that's the way how you next all Linux is and and well after seeing that presentation and we've been thinking about that for nearly a year so and then starting to try it maybe time for one last question that doesn't have some questions yes hello thanks for the presentation I want to ask you about the the emerge keywords in gen two compared to the module system that alcohol showed like how do you compare them and I think there's nothing to compare really because the whole emerge ecosystem is solely about installing packages and not really about doing service configuration so the whole nix as module systems is one abstraction layer up so did that answer a question not really okay perhaps have a personal talk yeah already yeah you can you can find him afterwards during what comes up next which is the next coffee break so a small round of applause again please for Chris and in this way talk thank you thank you
 All righty, time for our next speaker.  This is going to be Christian.  And you might know Christian from his, sorry?  Yes.  And you might know Christian from his work  that he has done on Valnex and on the vulnerability.  Vulnerability, I can't even pronounce it.  He did it, vulnerability roundups for NixOS.  But today he is going to talk to us  about this annoying situation in which you might end up  where you're not actually yet running NixOS,  but of course you want to.  And he's going to explain us how to get out of the misery.  Quick applause, of course, please.  Thanks.  Yeah, hi, everyone.  So does it work?  Yes.  Sorry.  OK, forget about it.  Sorry.  OK, as already, what's going on here?  Used to be working.  Sorry for the delay.  OK, one of the silenced wrestling was his laptop.  OK, we'll see.  Sure.  Okay. Sorry about that. Okay. As I already said,  hi, my name is Christian. I'm Systems Engineer doing this for a bit more than 10 years. You  can at least see it with the laser pointer. Yeah. Reach me by mail or find me at various  social media or RSC or GitHub. And I'm wanting to talk about migrating a hosting infrastructure  from Gen2 to LixOS and answer mostly two questions. First, what motivated us to migrate so you  don't do that just out of a mood? And what experience did we make? So I'd like to share  a bit of, well, real world insights so perhaps the community can profit from that. So when  I say we, I mean Flagsorcus Internet Operations, we are the guys with the cool aircraft, but  a bit more than that, we are a small company located in Halle, Zahler in Germany. And basically,  we are taking care of customer applications. So there are customers which can't or won't  concentrate on operational issues because for example, they are software shops or they  are too small or whatever. And so they approach us for deployment concepts, for automation,  for telemetry, for monitoring, for incident response, for load and resource management,  for upgrades, strategies and much more. So some, someone said we are sort of DevOps for  higher or DevOps as a service. So we are actually working closely together with the devs, usually  on individual applications and we work together to keep the staff running. So actually for  customer projects, we have more than 500 VMs with a lot of very diverse stuff installed  and running on them. So why did we migrate? So where did we come from? We have been historically  a gentle and puppet shop and before I am going to ditch this, it's important to state it  went running for quite a while and it was not too bad. So during the time we started  that, Gen2 was a good choice because of its extreme configurability and puppet was also  a good choice because at that time I think it was one of the best tools you could get  for automated systems management. But well, after doing that for 10 years or so, it got  too complex. So puppet is not able to manage everything and so it was no problem. We are  software engineers after all. So we say, well, let's write a script, let's write a wrapper,  let's write a fix up and this is a partial view of our old system management stack. We  see a lot of diverse steps and fixes and scripts and additional config runs and specific scripts  for stuff and this is only one tenth of the whole thing. And what you see is that there  are partly overlapping functionalities. So for example, puppet can manage 80% of something  or Gen2 emerge, can do 90% of something but then we have to put another fix and another  script and well, it was painfully slow. So for a full configuration run, we took more  than five minutes on a typical VM. Too many moving parts so you perhaps have to emerge  a package but then you have to run revdp rebuild. Don't know if anyone knows Gen2 and then you  have a Python updater and then you have a Perl cleaner and so on and so on. And every  step you add, you may get a bit closer to where you want but not really to the point  and this small gap when you multiply that with 500 machines gets a real awful lot of  work still to be solved manually and we didn't want to do that. So what was the problem?  What is more like we did a bad job? So would we just have to improve the implementation  or perhaps if the approach not optimal is our whole model of system management too  weak? Perhaps yes and to answer this question we have to go a bit into theory. So perhaps  some of you may know the real great paper by Steve Trager from NASA about why order  matters and in this paper he defines three main models of system management. The divergent  system management and congruent system management and I go to each of these in detail. So what  does divergence mean? Aside from the paper divergence is characterized by the configuration  of life hosts drifting away from any desired assumed baseline disc content. So what is  this? For example this is when you configure a server by following a checklist. Let three  admins do that. You get three different results but also like install our scripts for example  while call to shell. Okay this is divergent system management but there's also a legitimate  use for divergence. For example when you have user home dears or you have some sort of database  content or any relevant productive data is by definition divergent. So if you wouldn't  have that then you could burn your entire data center on DVDs and then it's static so  no point about that. And the second model is the convergent model. So this is where  most of the system management tools we have today live in. So you have some description  of the desired state and then you measure the actual state and then you see where the  delta is and then for each deviation you have a corrective action which puts the actual  system state nearer to the desired system states of many things like puppet or ansible  assault and you name it work this way. And this has also its use case and won't get away  for example you have activation scripts in NixOS who are constructed exactly around  this model or for example you have system these services which follow this model or  when viewed as a larger scope you see for example a container orchestration which also  follows this model. So this model is good but we can even get to a stronger model the  congruent model. This means congruence is the practice of maintaining this state in complete  compliance with a fully descriptive baseline. So for example system packages are exactly  defined in every distribution and of course everything in the Nix star is exactly defined  but also there are for example container images which are exactly defined and okay you could  argue if several functions belong to that as well or not perhaps yes. So and well to  get maximum control you want to make the congruent domain as large as possible and the other  two domains were still there but just not growing too big. So we decided it's time to  switch to another model we wanted to follow the congruent approach and the two interesting  candidates for doing so were of course NixOS and of course going with a container thing.  So well everyone is going with a container thing today but well they don't do well in  a multi-tenant environment because they provide no great isolation and the other thing is that  a container technology well has its strength but also has its weaknesses and if you read  some larger Docker file then you know what I'm talking about. So what attracted us to  NixOS I think in the first place the Nix language is so expressive that you can do nearly everything  with it and this was of course very important for us as software developers. You get a lot  of flexibility you can just mix everything together and all the dependencies are still  in place and working binary substitution you get that basically for free when you manage  to get Hydra running and while someone said Nix is a tool to rule them all so you have  one approach to packaging, to systems management, to deployment and at least but the last the  community is very approachable it was no problem for us to get into the community and for example  place pull requests or something like that this is very great. Okay what did we do? So  okay I'm just talking to some hackers here so the hacker would think no problem at all  just install NixOS on the VMs and everyone will be happy so well every one of you would  just start to put your whole project in one big default.nix and that would describe everything  then you would just say Nix build and then the whole project just builds and everything  is fine. Well this is what NixOS hackers think but we had a large installed base which was  not constructed with NixOS in mind and we had to put that somehow over the fence and  we'll see what's there how can we get that running and while doing so we talked to our  customers we reviewed the code and we found out well not everyone is ready for NixOS  so well NixOS sort of don't understand me wrong but it's a hacker thing it's technically  very advanced so I like that very much but some of our users were frightened so they  had quite a bit of hard time with this congruent or immutable approach so for example hey I  want just to edit some etcc whatever file why can't I do that and where is the option  for X and Z and do you have a GUI and what's going on here don't understand that and slash  user bin is empty so where it's all of the stuff gone and I'm overwhelmed okay Nix Nix  what's that funny language so I don't understand so can I program that like shell no I can't  then I don't understand that so functional programming is very expressive once you get  into it it's really fun and you don't want to go back and do something other but if you  don't do that and for example you are more like an admin guy who started putting some  CDs into some Microsoft Windows boxes then it's quite a steep learning curve that was  for the one thing and the other thing well it depends on what you're running and we do  a lot of web applications and they are well some of them insist of trying to build themselves  in the very moment you are starting them up for example we've got some Node.js application  which just starts to compile some C code in the moment you're trying to start that or  we have some CMS applications who start to optimize themselves somehow don't know what  they're doing but they I don't do know for sure they're going to fail if they cannot  write the installation directory and we also have some applications which are sent out  around this model of incremental build so for example some stuff which needs installed  base and then needs to reinstall itself depending on the old installed base and if you try to  install everything from scratch then it takes one and a half hours or so also some plugin  system auto update systems and so on so well you could say that's all bad rule it out yeah  but we are earning our money with running business critical customer applications and most of  them have some tiny ugly core deep within which well isn't really good software design  but is absolutely important and everyone knows okay no it's not the best but we cannot do  without it and it's our job to keep it running so how to solve that obviously we cannot rewrite  everything and nixify everything right away so we decided for flexible approach so we  just took the application in the narrower sense so the custom code and see if that matches  more convergent model or another model and we also separate components we also had that  in our old gentle setup which bundle often mostly used things like engine x like post  grass like riders like ma'am cashier like elastic search that many projects use and now  we see which part of what goes into what model so the application deployment depends largely  on the project we have some conventional deployment for example like ansible we didn't want to  get other tools at that point we have even divergent stuff like data dears and but there  are also congruent elements for example like customer supplied container images and of  course there are a few already nixified projects and the color code is green is congruent yellow  is convergent and red is divergent and as it comes to the premade components most of  them have some glue code use some nix service modules which are in place already and have  some integration points what does that mean these are directories or files where user  deployments can typically put some snippets in the services the equivalent of fiddling  with some config files under etc so I give an example for example our engine x component  has an integration point this is some directory and you can just drop in some virtual hosting  configuration files and so on and that stuff gets picked up by the nix as rebuild run using  our glue code and gets incorporated in the configuration running in the nix store so that  we still run everything out of the nix store but users find at least some points they can  cope with so that was the plan was what was our experience for most projects with all  of this in place it was quite doable and while you have to fit around a little bit but I  think 90% of our customer projects went quite fine with that and we of course have some  dependencies which used to come out of this out of the system installation on our jankto  for example some libraries like SSL and so on and most of them are now placed into the  user nix profile during whatever using whatever mechanism is fitting so it's in some project  we installed them via nix and others have a default nix which builds an nf which is going  to be installed there and in the end we see a large running code base mixing up nix code  from as far back from 1509 until the current stuff and due to nix as we are able to mix  it so it works out so this is really great so of course we made some tools to facilitate  this our major wrapper and it's only one wrapper for everything is FC manage this is basically  a thin layer above nix as we built and that runs regularly from a system the timer and  pulse the channel sees if average changes occur and triggers them and then what's perhaps  more interesting for the community it's a little tool called FC user scan that scans  unmanaged installations for example user just compiles code in his home and that uses some  libraries from the nix store but the nix store garbage collector doesn't know about that and  deletes the libraries at some point and then the user code doesn't run anymore and so this  scanner just goes through the installation and registers all the time dependency it finds  with a nix store garbage collector and of course will next I think this is well known  this scans an installation for open CVS so what were the main benefits from our point  of view going from gen two to nix gave us significantly higher productivity so we can  do stuff in no time which took days in the old environment we also have a test on the  infrastructure level this quite good we got a lot more flexibility because we don't have  to say okay this particular version of opens SSL is the system version and you have to  use that and nothing else and also we see that we can scale things so for example most  engine x-based conflicts are the same but also have the flexibility to bring in customer  specific modifications without losing oversight so what do our users think well most users  don't care so they just want to have some Linux server so well distro don't know so  okay get it running that's fine so and we would like to have some more projects where  customers are just shipping a default nix and say hey that's my default nix run that in  production but it's only the case for maximum 10% of our projects so I think we profit more  of the migration than our users profit directly but of course they profit indirectly because  we are better off right now okay to finish on there are some things which well I would  wanted to call the section pain points but I think there's a misnomer because working  with nix is not painful at all it's fun so I called it things to improve and the security  story so I've been busy providing vulnerability roundups every one week or two but when you  look around you see quite a long list of open issues so I think we need a better approach  to actually fixing stuff so discovering of what needs to be done is good but it's only  first step the second step is to fix the stuff and in my opinion this is largely a problem  of missing manpower so I was really happy to see a larger security team and I want to  use Saturday I don't know for people who are still around there and just to see what could  be done in the short term just to get this one better and we also should think about  back porting important changes to older releases don't know if we need a formal process for  that or just start by doing so informally the other point that is a constant source of  well confusion with our customers is that nix likes to restart everything even for minor  changes well it comes from the model if I change any input the hash changes the unit  file changes and system D restarts the unit so when you see for example here for postgres  it's well quite a problem because it terminates all connections free starts and then the application  just has to reconnect and we've got one minute of useless downtime so also here the question  is can we do better can we come up with some clever scheme to avoid that without breaking  the overall nix-v-ass model and well the last point is I think the community has grown quite  a bit over the last years nix-v-ass is attracting a larger user base this is great news but I  think we have to keep up with community structure so from my perspective I would really be glad  to see more teams with well clearly structures structured responsibilities well perhaps there  are already exists some teams but then they are not well discoverable at least so I think  this is a point where we can and should improve and so just to continue the success of nix-v-ass  so as a final question if we would be in the same situation we've been three years ago  right now would we choose nix-v-ass again probably yes I think it's a really great piece of software  infrastructure thank you  okay hello thanks so one of your slides was very interesting that just what first one  of the first ones is nix-v-ass versus containers and well someone would have to ask that yeah  yeah so with Docker and containers there is this huge pool about one containers runs one  application and so on so it seems that it fits quite well that my understanding what you are  doing because your customers have one application and they want to run one application in a  container so would you like to elaborate a bit about why nix-v-ass for you was a bad choice  yeah sure sure so well one point is that containers are only part of the solution because in a  multi-tenant environment you have to provide in case encapsulation anyway and so we still need  some management for the underlying underlying thing be it VMs whatever and we're not large  enough to have dedicated hardware per customer and the other thing is that containers impose a  specific way of deploying the application because you have to follow their standard or you aren't  going to use it well and with our customer base this is not possible for example we have a lot  we are historical a python shop and there are a lot of CC build out base projects and they don't  work together very well in addition to that we are experimenting now with running containers  inside VMs just to give the benefits of both worlds more question yes what was the first  question yes so your 10% of your base of user base is already using default nix or  kind of the nixify the project for you in a sense at least partly what is the difference in  maintainability or between this 10% and let's say some some similar size project okay it largely  depends on the project in my experience the maintainability is more dependent on the software  code base as such so there are for example projects which save parts of the code in their  databases and well that's a nightmare and and well we have quite good convergent deployment  tools for example we use Batur this is an in-house tool or Ansible and other tools and if you do  that well it works too and so and I think the major benefit of projects which are using nixified  setup is that you've got the exact reproach usability on the developers computers so you  know okay exactly what is running here will be running there I think this is the same  promise that containers make and I think this is the main benefit for the most other projects  we have a much more sort of staging setup with a lot of VMs that are solely there to see that  everything still fits together in the deployment that we have that too for nixified projects  but they don't need that much for staging second question so you remember do you still remember  when you came back to your meeting with the idea let's try nix and see it and when your  coworkers okay that was actually Domen the guy so and we had a sprint at then gocept this was the  former company and we were supposed on hacking on some Python stuff and I know very well Domen  just put there some nix stuff in and just said here look at that that's cool and we all looked  at that and said that's weird but why you have to free yourself from 30 years of thinking that's  the way how you next all Linux is and and well after seeing that presentation and we've been  thinking about that for nearly a year so and then starting to try it maybe time for one last  question that doesn't have some questions yes hello thanks for the presentation I want to ask you  about the the emerge keywords in gen two compared to the module system that alcohol showed like  how do you compare them and I think there's nothing to compare really because the whole  emerge ecosystem is solely about installing packages and not really about doing service  configuration so the whole nix as module systems is one abstraction layer up so did that answer  a question not really okay perhaps have a personal talk yeah already yeah you can you can find him  afterwards during what comes up next which is the next coffee break so a small round of applause  again please for Chris and in this way talk thank you thank you
Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_0_179.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_180_395.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_396_455.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_456_635.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_636_839.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_840_2879.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_2880_2927.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_2928_3023.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_3024_3911.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_3912_4199.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_4200_8051.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_8052_11759.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_11760_12071.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_12072_12119.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_12120_12347.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_12348_20291.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_20292_21899.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_21900_37895.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_37896_39683.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_39684_42491.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_42492_43919.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_43920_45251.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_45252_46295.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_46296_47375.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_47376_47795.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_47796_48419.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_48420_50291.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_50292_50447.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_50448_50747.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_50748_51959.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_51960_52739.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_52740_52979.png Christian_Kauhaus_-_Migrating_a_Hosting_Infrastructure_from_Gentoo_5GtOAaqqNGU_52980_52989.png