Building development environments using Vagrant
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Blomberg, Mikael
Building development environments using VagrantDuring Oskari.org development we noticed releases usually where delayed due to environment differences. Our desktop and server environments are quite different. We believe there should be no differences between the production and development environments. In our research we discovered Vagrant, which utilizes Virtualbox for creating virtual environments. It was a perfect solution and we wondered into the world of DevOps. We noticed our lack of automation and began our search for a good Configuration Management tool. Our research showed many promising alternatives and we ended up choosing Ansible. Armed with Vagrant and Ansible we began to automate our development environment in a reproducible way. This talk is about our journey into DevOps and why you should join us!
 I'll start in a minute, but let's start with being responsive and tweet like that. Done? We can close that and then let's start the presentation. So welcome everybody. This is not the slide we're supposed to see yet. All right. Let's start like this. I'm a little bit nervous if you haven't noticed it yet. And this is not going well at all. But that's okay. All right. All right. We're good to go. My name is Mikael Bloomberg. I come from Finland. I work for a company called SC5. And this is to talk about our journey into using DevOps. So I've been working on this project, OSCORI. And it's an open source project. Basically, it uses a lot of open source software and it's coordinated by the National Land Survey of Finland. And we're a team of eight developers and there's many more organizations that use the same software and you can check out OSCORI.org to find out more. So now let's go back in time. Let's start in January. So our development environment is Windows 7 and SIGVIN. If you are a developer, you probably know that it's not a very good starting point considering that the servers are usually Linux. And that's what we also have. We have Linux servers. And the software we mainly had at that time was Paikka Tietoikkuna. And in that case, we had the usual staging them on production servers. I think most of you know where this is heading. So all of these are a little bit different, every environment. And because they're a little bit different, it was difficult to test in advance. So basically, you need to know a little bit more about the environment to make releases. And you also usually had to tinker a little bit, which means it's a little bit painful. And in January, we were presented with a new challenge. We had two more offerings and two new platforms. But this time only production servers. So that means basically we cannot test in advance. So we had to figure out something. Now, we sat down and come up with our requirements for this thing. We wanted the environment to be reproducible for testing and for development and to get everything working as it should. And because we don't like to do that manually, we wanted it to be automated. That way, we can kind of remove the human factor from the deployments and it would be successful every time. And on our wish list, we also had that the tools would be easy to learn and easy to use. So we started with doing an evaluation. We're reproducible server environments. Well, that's basically virtualization. That's pretty common these days. We started to check out the different options and the most common ones is VMware and the virtual box. And, well, the free version of Fusion, it doesn't have so many built in, well, a command line interface that you can do a lot of stuff with. So the only alternative really is virtual box at this time unless you pay for the VMware version. In that case, you can use a tool called Vagrant. So we found Vagrant. It uses virtual box. Has somebody in the audience used Vagrant? Great. I really like it. It makes life easier. The kind of thing with Vagrant is that you write a single command, Vagrant up and you have it up and running. So if you used virtual box and you couldn't quite get it working, try out Vagrant. It will make life easier. It comes with a set of promises like synced folders. It makes data transformations easier. It doesn't always work in my experience, but most of the time it works. You can use provisioning to get your virtual environment to a specific point. And that's the topic for our next slide. So we checked out the provisioning alternatives in Vagrant. It was version 1.4 at the time. So we had four different alternatives. Shell Script, Chef, Poppet and Ensemble. Well, I don't know how much you are familiar with these tools, but Chef and Poppet are quite old. Well, Shell Scripts are even older. But Shell Scripts, well, because they're like been there always, they're easy to write. But in our experience, they were a little bit unreliable and it's not important unless you specifically take actions to make them important. And that means that you cannot rerun it several times after a chatter. So basically, if you run a Shell Script and move something around, it will be moved and you kind of need to notice that, hey, it has already been moved to the place where it's supposed to be moved. Chef and Poppet, they are built in a way that they can notice, for example, if something has been moved already. And but they have some issues which in our case was mainly that they're difficult to write. And when we checked out Ensemble, we were pleasantly surprised that it was easy to write and simple to use. So we took that. These are the values and promises of Ensemble. It's reliable. It has a low learning curve and it's easy to install. It's consistent and secure. The consistency comes from the way the configuration files are built. It's just another markup language, Yamle. And it's quite easy to read and understand what's going on even if you never seen it before. It's secure because it uses SSH for the transactions. And it has a nice feature of both push and pull mode. So push means that you have a server, a master, and then you push it into the target computer, the things you want to get installed. And pull mode means that you have, you go into the target computer and you pull it from some central repository what you want to have installed. So let's go back in time again. We started out building environments so that we can use Vagrant. We used Baccaro I.O. to build the environments because we wanted it to have a finished locale and it was based on Ubuntu. The next one was a Gen2 box and then we built a CentOS box because if you remember, our offerings were supposed to target Gen2 and CentOS, we started out with Ubuntu because Vagrant has this default box based on Ubuntu. So just to get to learn the tool, we did that first. While we did this, we learned that our setup is kind of complicated. We have a lot of, our stack contains a lot of different pieces of software and this is where the fact that we had to automate it made it very, very kind of clear that what are the steps that needs to be done to get everything working and of course we noticed we have some missing documentation which has been fixed now. So after this, we're in a situation we have one script that is able to do the whole installation process for three platforms which is kind of nice. But every platform has its own challenges. They're a little bit different and unfortunately our virtualized environments are still different from production. That's basically because when we have it in Amazon, we have no idea how they built their images and we haven't been able to extract the images from Amazon and put it into virtual box yet. And these three projects are rapidly involving, so sometimes they break and we need to fix that along the lines. But we still seen benefits of this. We know the state of the servers which is nice because it's easier to improve. We were able to prepare and test before we went to production. So our deployments were very nice to do. We just run a command and there it was. And I have to say we almost have saved time when doing deployments. We will once we get more environments and do more automation. But at this point, we're kind of breaking even. And what's next? We're going to evaluate Docker. It's promising that we can make an image and insert it in any environment we want to have. So the server can run CentOS, it can run Red Hat, it can run Gentoo, it can run Ubuntu, it doesn't really matter. Docker will abstract that away and we can kind of just focus on creating our own stack and build it in advance and use that for development and testing and deployment production. Then we need to figure out how to formalize our migration processes and practices. And once we have done this, we can start to measure and improve our server scalability and of course do automated testing. It seems like I've been talking way, way faster than I originally assumed. So there's a lot of time for questions if you have any. Yes. Yeah, just a moment. So we started looking into using Vagrant and I'm not sure which exact version of VMware we're running but it is the paid, you know, whatever enterprise version. And actually I kind of got the impression that it was available with, you know, like the free VMware player for Vagrant but we seem to have hit a wall. So I was curious, you said actually that it works with the enterprise version or the... Yes, but you need to buy the plug-in, the Vagrant plug-in for VMware. So that's kind of the open software. You can get it working but if you want to use Fusion then you need to pay for the plug-in for Vagrant and then you're all set. Okay, right. That's the brick wall that we hit. Yeah. And then actually the follow-up question, or actually totally unrelated question, we have a kind of mixed Linux Windows environment and I don't know if you could speak to what portions of these tools work well with Windows? Well... If you know. I unfortunately do. These don't work very well on Windows at all. Unsymbol uses very low-level Linux function calls so unsymbol won't work on Windows. What you can do is that you can fire up a Linux box in, well, in your case, Fusion or VMware and then you use that as the kind of orchestrating machine. So you can use unsymbol from your virtualized environment to fire up more virtual environments. I've never used Vagrant before which is partly the reason why I'm here to learn. One thing that wasn't quite clear to me is that you articulated that your server environment was Linux-based but then you mentioned it was going through several different versions of the operating system that you had to support in production and then you said you moved, you brought in Vagrant. Did Vagrant run on those operating systems? No. Vagrant is running on my development box computer so that I can simulate the production environments. So basically what I did is I have a local version of the server environment so that I can run the testing using Vagrant and kind of get the environment there. Okay. Yeah, you mentioned using Ubuntu and I wasn't sure if Ubuntu was, you put Ubuntu into virtual box and Vagrant is managing virtual box running Ubuntu. I got that much. But then you referred to CentOS and Gentoo and that's why I wasn't confused about, were you staying with Ubuntu or did you switch from Ubuntu? Yes, I switched from Ubuntu to Gentos and from Gentos to Centos and my script is kind of aware of which distribution it's going to install so it kind of runs on Gentoo, it uses the emerge commands and on Ubuntu up get and in Centos, yum install. Okay. Thanks. Do you care elaborate a little bit more on the differences between Chef and Ansible and why the ladder is more, is easier to use? Because I know that Chef is hard to use. I'm just curious about the alternative. I haven't used Chef. I have to say that in advance. What I did was I was checking out the examples, how to use Chef. So if Chef Reset was like this long, the similar Ansible script is that long and compared to Puppet which was like this long, Ansible is still only the same. So it's more verbose and it's quite easy to use because we have time. I can try to show you a playbook which is the name for this Ansible. Let's see if we can get it like so. So basically what we're telling is that, hey, we have defined something called web servers. That's the target for this. These are our variables. This is our remote user and these are the tasks. This is the interesting part because this is where everything happens. In this case, we have a name key and the value will, in this case, describe what will happen. And here we have the module name which exists already in Ansible which is the yum software. And what we are telling here is that there's a package called HTTPD, Apache in other words, and we want to install the latest version of it. And well, that's what this task is doing. Next task, it's the next minus line. We write the config file and if it has changed, then we notify that, hey, we need to restart Apache. If it doesn't change, it won't notify it, so it won't be restarted. And these notifications are stacked up. So if you have a lot of changes, then it will only once restart it and restart will happen after the playbooks have ran. Well, the notifications are mapped to handlers which is basically just a service called at hey, you need to restart this service. And here's the third task which is to ensure it's up and running. So it's quite understandable, I think. The first section about variables, right, properties, right? Could it be now they are defined, they are coded there? Could they be defined in a more flexible way, like reading them from a properties file, from a database, from whatever? Because we had some issues with Chef being a little bit rigid on this. Database, I'm not so sure. But a separate file, yes, that's possible. If you have some files that are sensitive, like passwords or whatever, you can use this Ansible Vault tool and you encrypt it and it will be decrypted when it's fetched from the repository or how you ever feel to get the data, the settings into the computer. So we were going to develop and shop for a special software and one of the things, so we've started using a big Ansible. Okay. One of the two nice things is for describing machine environments to our system administrators, whereas before it's kind of a clunky way of trying to describe, you know, what servers we need, whatever. So at this point we just pass on a vagrant and Ansible configuration set of files and get that going. The other nice thing is being able to onboard new developers to applications really fast, whereas previously they'd go through documents to step by step set up. And at this point they can have development environments with the applications that they're developing on their system, the same as it is on our Dev QA and Prod stack within a minute or two. So that's really nice. And then of course on how fast they can download the vagrant books. Right, exactly. So the nice thing with that is that we've standardized on AscentOS for everything and then we have internal Git repository that measures download from there. So thanks for coming.
 I'll start in a minute, but let's start with being responsive and tweet like that. Done?  We can close that and then let's start the presentation. So welcome everybody. This is  not the slide we're supposed to see yet. All right. Let's start like this. I'm a little bit  nervous if you haven't noticed it yet. And this is not going well at all. But that's okay.  All right. All right. We're good to go. My name is Mikael Bloomberg. I come from Finland. I  work for a company called SC5. And this is to talk about our journey into using DevOps. So I've  been working on this project, OSCORI. And it's an open source project. Basically, it uses a lot of  open source software and it's coordinated by the National Land Survey of Finland. And we're a team  of eight developers and there's many more organizations that use the same software and you  can check out OSCORI.org to find out more. So now let's go back in time. Let's start in January.  So our development environment is Windows 7 and SIGVIN. If you are a developer, you probably know  that it's not a very good starting point considering that the servers are usually Linux. And that's  what we also have. We have Linux servers. And the software we mainly had at that time was Paikka  Tietoikkuna. And in that case, we had the usual staging them on production servers. I think most  of you know where this is heading. So all of these are a little bit different, every environment.  And because they're a little bit different, it was difficult to test in advance. So basically,  you need to know a little bit more about the environment to make releases. And you also usually  had to tinker a little bit, which means it's a little bit painful. And in January, we were  presented with a new challenge. We had two more offerings and two new platforms. But this time  only production servers. So that means basically we cannot test in advance. So we had to figure  out something. Now, we sat down and come up with our requirements for this thing. We wanted  the environment to be reproducible for testing and for development and to get everything working  as it should. And because we don't like to do that manually, we wanted it to be automated.  That way, we can kind of remove the human factor from the deployments and it would be successful  every time. And on our wish list, we also had that the tools would be easy to learn and  easy to use. So we started with doing an evaluation. We're reproducible server environments. Well,  that's basically virtualization. That's pretty common these days. We started to check out  the different options and the most common ones is VMware and the virtual box. And, well,  the free version of Fusion, it doesn't have so many built in, well, a command line interface  that you can do a lot of stuff with. So the only alternative really is virtual box at  this time unless you pay for the VMware version. In that case, you can use a tool called Vagrant.  So we found Vagrant. It uses virtual box. Has somebody in the audience used Vagrant?  Great. I really like it. It makes life easier. The kind of thing with Vagrant is that you  write a single command, Vagrant up and you have it up and running. So if you used virtual  box and you couldn't quite get it working, try out Vagrant. It will make life easier.  It comes with a set of promises like synced folders. It makes data transformations easier.  It doesn't always work in my experience, but most of the time it works. You can use provisioning  to get your virtual environment to a specific point. And that's the topic for our next slide.  So we checked out the provisioning alternatives in Vagrant. It was version 1.4 at the time.  So we had four different alternatives. Shell Script, Chef, Poppet and Ensemble. Well, I  don't know how much you are familiar with these tools, but Chef and Poppet are quite  old. Well, Shell Scripts are even older. But Shell Scripts, well, because they're like been  there always, they're easy to write. But in our experience, they were a little bit unreliable  and it's not important unless you specifically take actions to make them important. And that  means that you cannot rerun it several times after a chatter. So basically, if you run  a Shell Script and move something around, it will be moved and you kind of need to notice  that, hey, it has already been moved to the place where it's supposed to be moved. Chef  and Poppet, they are built in a way that they can notice, for example, if something has  been moved already. And but they have some issues which in our case was mainly that they're  difficult to write. And when we checked out Ensemble, we were pleasantly surprised that  it was easy to write and simple to use. So we took that. These are the values and promises  of Ensemble. It's reliable. It has a low learning curve and it's easy to install. It's consistent  and secure. The consistency comes from the way the configuration files are built. It's  just another markup language, Yamle. And it's quite easy to read and understand what's  going on even if you never seen it before. It's secure because it uses SSH for the transactions.  And it has a nice feature of both push and pull mode. So push means that you have a server,  a master, and then you push it into the target computer, the things you want to get installed.  And pull mode means that you have, you go into the target computer and you pull it from some  central repository what you want to have installed.  So let's go back in time again. We started out building environments so that we can use  Vagrant. We used Baccaro I.O. to build the environments because we wanted it to have  a finished locale and it was based on Ubuntu. The next one was a Gen2 box and then we built  a CentOS box because if you remember, our offerings were supposed to target Gen2 and  CentOS, we started out with Ubuntu because Vagrant has this default box based on Ubuntu.  So just to get to learn the tool, we did that first. While we did this, we learned that  our setup is kind of complicated. We have a lot of, our stack contains a lot of different  pieces of software and this is where the fact that we had to automate it made it very, very  kind of clear that what are the steps that needs to be done to get everything working  and of course we noticed we have some missing documentation which has been fixed now.  So after this, we're in a situation we have one script that is able to do the whole installation  process for three platforms which is kind of nice. But every platform has its own challenges.  They're a little bit different and unfortunately our virtualized environments are still different  from production. That's basically because when we have it in Amazon, we have no idea  how they built their images and we haven't been able to extract the images from Amazon  and put it into virtual box yet. And these three projects are rapidly involving, so  sometimes they break and we need to fix that along the lines. But we still seen benefits  of this. We know the state of the servers which is nice because it's easier to improve.  We were able to prepare and test before we went to production. So our deployments were  very nice to do. We just run a command and there it was. And I have to say we almost  have saved time when doing deployments. We will once we get more environments and do  more automation. But at this point, we're kind of breaking even. And what's next? We're  going to evaluate Docker. It's promising that we can make an image and insert it in any  environment we want to have. So the server can run CentOS, it can run Red Hat, it can  run Gentoo, it can run Ubuntu, it doesn't really matter. Docker will abstract that away  and we can kind of just focus on creating our own stack and build it in advance and  use that for development and testing and deployment production. Then we need to figure out how  to formalize our migration processes and practices. And once we have done this, we can start to  measure and improve our server scalability and of course do automated testing. It seems  like I've been talking way, way faster than I originally assumed. So there's a lot of  time for questions if you have any. Yes. Yeah, just a moment.  So we started looking into using Vagrant and I'm not sure which exact version of VMware  we're running but it is the paid, you know, whatever enterprise version. And actually  I kind of got the impression that it was available with, you know, like the free VMware player  for Vagrant but we seem to have hit a wall. So I was curious, you said actually that it  works with the enterprise version or the... Yes, but you need to buy the plug-in, the Vagrant  plug-in for VMware. So that's kind of the open software. You can get it working but  if you want to use Fusion then you need to pay for the plug-in for Vagrant and then you're  all set. Okay, right. That's the brick wall that we hit.  Yeah. And then actually the follow-up question,  or actually totally unrelated question, we have a kind of mixed Linux Windows environment  and I don't know if you could speak to what portions of these tools work well with Windows?  Well... If you know. I unfortunately do. These don't work very well on Windows at all. Unsymbol  uses very low-level Linux function calls so unsymbol won't work on Windows. What you  can do is that you can fire up a Linux box in, well, in your case, Fusion or VMware and  then you use that as the kind of orchestrating machine. So you can use unsymbol from your  virtualized environment to fire up more virtual environments.  I've never used Vagrant before which is partly the reason why I'm here to learn. One thing  that wasn't quite clear to me is that you articulated that your server environment was  Linux-based but then you mentioned it was going through several different versions of  the operating system that you had to support in production and then you said you moved,  you brought in Vagrant. Did Vagrant run on those operating systems?  No. Vagrant is running on my development box computer so that I can simulate the production  environments. So basically what I did is I have a local version of the server environment  so that I can run the testing using Vagrant and kind of get the environment there.  Okay. Yeah, you mentioned using Ubuntu and I wasn't sure if Ubuntu was, you put Ubuntu  into virtual box and Vagrant is managing virtual box running Ubuntu. I got that much. But then  you referred to CentOS and Gentoo and that's why I wasn't confused about, were you staying  with Ubuntu or did you switch from Ubuntu?  Yes, I switched from Ubuntu to Gentos and from Gentos to Centos and my script is kind  of aware of which distribution it's going to install so it kind of runs on Gentoo, it  uses the emerge commands and on Ubuntu up get and in Centos, yum install.  Okay. Thanks.  Do you care elaborate a little bit more on the differences between Chef and Ansible and  why the ladder is more, is easier to use? Because I know that Chef is hard to use. I'm just  curious about the alternative.  I haven't used Chef. I have to say that in advance. What I did was I was checking out  the examples, how to use Chef. So if Chef Reset was like this long, the similar Ansible  script is that long and compared to Puppet which was like this long, Ansible is still  only the same. So it's more verbose and it's quite easy to use because we have time. I  can try to show you a playbook which is the name for this Ansible. Let's see if we can  get it like so. So basically what we're telling is that, hey, we have defined something called  web servers. That's the target for this. These are our variables. This is our remote user  and these are the tasks. This is the interesting part because this is where everything happens.  In this case, we have a name key and the value will, in this case, describe what will happen.  And here we have the module name which exists already in Ansible which is the yum software.  And what we are telling here is that there's a package called HTTPD, Apache in other words,  and we want to install the latest version of it. And well, that's what this task is doing.  Next task, it's the next minus line. We write the config file and if it has changed, then  we notify that, hey, we need to restart Apache. If it doesn't change, it won't notify it,  so it won't be restarted. And these notifications are stacked up. So if you have a lot of changes,  then it will only once restart it and restart will happen after the playbooks have ran.  Well, the notifications are mapped to handlers which is basically just a service called at  hey, you need to restart this service. And here's the third task which is to ensure it's  up and running. So it's quite understandable, I think.  The first section about variables, right, properties, right? Could it be now they are  defined, they are coded there? Could they be defined in a more flexible way, like reading  them from a properties file, from a database, from whatever? Because we had some issues  with Chef being a little bit rigid on this.  Database, I'm not so sure. But a separate file, yes, that's possible. If you have some  files that are sensitive, like passwords or whatever, you can use this Ansible Vault  tool and you encrypt it and it will be decrypted when it's fetched from the repository or how  you ever feel to get the data, the settings into the computer.  So we were going to develop and shop for a special software and one of the things, so  we've started using a big Ansible. Okay.  One of the two nice things is for describing machine environments to our system administrators,  whereas before it's kind of a clunky way of trying to describe, you know, what servers  we need, whatever. So at this point we just pass on a vagrant and Ansible configuration  set of files and get that going. The other nice thing is being able to onboard new developers  to applications really fast, whereas previously they'd go through documents to step by step  set up. And at this point they can have development environments with the applications that they're  developing on their system, the same as it is on our Dev QA and Prod stack within a minute  or two. So that's really nice.  And then of course on how fast they can download the vagrant books.  Right, exactly. So the nice thing with that is that we've standardized on AscentOS for  everything and then we have internal Git repository that measures download from there.  So thanks for coming.
09_Mikael_Blomberg_720p_0_431.png 09_Mikael_Blomberg_720p_432_695.png 09_Mikael_Blomberg_720p_696_1007.png 09_Mikael_Blomberg_720p_1008_1139.png 09_Mikael_Blomberg_720p_1140_1235.png 09_Mikael_Blomberg_720p_1236_15323.png 09_Mikael_Blomberg_720p_15324_19667.png 09_Mikael_Blomberg_720p_19668_21167.png 09_Mikael_Blomberg_720p_21168_21323.png 09_Mikael_Blomberg_720p_21324_21467.png 09_Mikael_Blomberg_720p_21468_23339.png 09_Mikael_Blomberg_720p_23340_24875.png 09_Mikael_Blomberg_720p_24876_24923.png 09_Mikael_Blomberg_720p_24924_25139.png 09_Mikael_Blomberg_720p_25140_25295.png 09_Mikael_Blomberg_720p_25296_25895.png 09_Mikael_Blomberg_720p_25896_26087.png 09_Mikael_Blomberg_720p_26088_26147.png 09_Mikael_Blomberg_720p_26148_33275.png 09_Mikael_Blomberg_720p_33276_33375.png