Improving your virtualization development workflow with Lago
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Martins, Rafael
Lago is an ad-hoc virtual framework which helps you build virtualizedenvironments on your server or laptop for various use cases. It creates andorchestrates virtual machines that can be used to run test suites and allow adeveloper to quickly test his code change on non-trivial flows such as livemigration even before starting the CI process. In this session, we will walkthrough the usage of Lago in the oVirt project and how it can be extended toassist virtualization developers. Virtualization software became a vital part of the IT infrastructure of mostcompanies nowadays. Making sure that the software works as expected in "realworld" tasks is hard, mainly because virtualization environments are usuallyhard to reproduce automatically. Lago is an ad-hoc virtual framework whichhelps you build virtualized datacenter on your server or laptop to run testsuites. Being a framework, it can be extended to implement any requiredworkflows needed by the tested software. We will discuss how Lago works with KVM and Libvirt to create a testenvironment, how it can be extended to help with automated system tests ofvirtualization products and how it is currently used by oVirt project to runautomated system tests and help developers testing their patches duringdevelopment phase. We will also review and compare Lago with tools likeVagrant, when applied to the virtualization development workflow.
 I'm pleased to introduce Rafael Martins from Red Hat. Thank you. Hi everyone. First of all, thanks for coming. I'm Rafael Martins. I'm from Red Hat. I work on the integration team for Overt, where we basically do all the releases and we maintain all the installation scripts for Overt. And I am also a gentle developer for more than six years, so most of my background is on distributing software and packaging software. And this talk is to show you how we use some tools in Overt to improve the way we test software and we release software. So... It's not working. So let's start by describing what's the problem we are actually trying to solve. Basically, virtualization products are complex. They are complex in the sense that they always have lots of integrations. For example, we need to integrate with storage, with network. And when you get some bug, some new feature that you need to implement that will require you to rebuild the environment to test, you need to reproduce the environment. And sometimes you can take a really long time building this environment. Personally, I already spent more than one day trying to reproduce the environment to fix a bug. So most of my research on this topic and the tools, the plugin I wrote for Lago were because I was really tired of spending so much time on reproducing bugs and wanted something automated. So the objective of this talk is to show you how you can do the same for your project if you want to and if you find it viable for your project. And showing how we do it for Overt. So I will use Overt as a new case for the talk. How many people here in the room already use it? Overt or Rav? Or at least know what it is? Okay. So Overt basically is a virtualization management system. You have a web application that you call Overt Engine where you can create data centers, clusters, new virtual machines and stuff. And it can manage the VMs for you. It's based on KVM and Libvert that are technologies that are well known in the open source and Linux community. And this is what the Overt Engine looks like for those that don't know it. This is a screenshot from one of our internal instances that Red Hat that we use for testing. It is not the biggest deployment we know but it's a quite large deployment with three data centers. And this is the main dashboard. We have options and screens to change about everything on the machines. And now we will talk about the solution we found to make testing and deployment and verifying of patches easier. That is Lago, Lago Project. It is basically a network framework for virtualization that can build environments for you using Libvert and KVM so you can buy environments with how many machines you want and it will allow you to integrate those machines and do the tasks needed by using plugins right in Python. It's totally extensible. Almost all of the layers of Lago are extensible by plugins in Python and the core itself just carries the very basic functionality that is needed to work. Now we will talk about how we use Lago in Overt. All the tasks, all the code that connects to Overt and all of the related stuff is on a splitted plugin that is called Overt Lago and Lago itself does not carry any code that talks to Overt. So in the same way that we have Overt Lago for Lago you can have a specific plugin for another product and another project and use it with Lago. Lago is just responsible by orchestrating the machines and calling the plugins to do the job. We to solve the problem of automated tasks because as release managers we need to be able to trust the software we are releasing. We need some frequent tasks to make sure that anything big is broken and one of the solutions for that was an automated test suite that we have. It is run by Jenkins frequently and with this we can make sure that the code is ready to release and that the main flows we support are not broken. For example we have Lago and one engine and two hosts. This is all we virtualized by Lago by using the Overt Lago plugin and Lago outside just managing the VMs and the Overt Lago plugin creating everything that is needed. This is automated. We have scripts and repository descriptions to cache the RPM files. Lago can create a local repository that is used to install packages quickly without downloading from internet every time. So with this we can run end to end tasks in like 25 minutes, 20 minutes depending on what we are doing for a really big test suite. The good things about system tasks is that having automated tasks for you is that you can quickly see big breakages or big issues on your project. If you have a patch that is really breaking the basic flows and it is not really bad it may still pass unit tasks but it won't pass system tasks because it is actually creating the machines, it is actually creating the VMs, actually trying to add hosts, trying to add storage and so it won't pass. And it is very well maintained. We have a lot of people writing tests for it and keeping an eye on it to see if it is really working and make sure that everything is fine. And a last point that is important to what I will tell you after is that the virtual machines are left to be used after the tests are running so you can still, after running the test suite, you can still log in the machines and do something else like run some command, read some logs or do something else with the machine that you want to verify your patch. And this is a screenshot of our Jenkins instance. We have lots of jobs for all the over supported versions. We can look at it and quickly see that, for example, in this day we had some issue on 3.6 that was breaking most of the builds. And we can look at the logs and see what's wrong and talk with the person that sent the patch and fix it. That is more important. Now I will talk about the real point of this talk that is doing manual testing because automated tests are good, are needed, but we also need manual tests to improve our workflow when testing patches. We have several jobs on Jenkins that can build custom RPMs from a Jared patch. We use Jared on Ovid. And you can easily build some RPMs from any Jared patch and use those RPMs on Ovid system tests to create an environment with the new functionality you implemented or with the fix you are doing. An important point is that the developer can do it on its own laptop. So the process is a bit memory hungry because we are running some heavy stuff, but it's definitely possible to run it on a laptop and build your RPMs and just test them. And most of the time a lot of actions, more simple patches, don't even need some manual intervention to be tested. The automated test may just be verifying how the flow that you are affecting with your patch. And sometimes you won't even need to do any manual test to test it. But like anything in software basically we had some trade-offs to have good automated tests on Ovid system tests, we had to do some choices that sometimes are not really, how can I say, they make the manual testing a bit harder than developers wanted. For example, when we are running, when we are using Ovid system tests to test a custom patch, you need to always run the full test to it. It means that if your test is correct, it's fine, you just run it once and be done with it. But if your test has some issue, you need to iterate. Any time you change the patch, you need to rerun the full test to get it fixed, to get it to work. And if the patch changed the behavior of the test, like if you change it, either the new option to install it, or you have some, change the behavior of some part of the system, it will break the test. And to be able to use Ovid system tests to test it, you need to fix the test. This is not a bad thing because this means that Ovid system tests will keep working because the developer will need to fix the test. And it's a good thing because the developer will use it to write in tests. So this is not really a downside, but it will spend time because instead of just fixing your patch, you still need to fix the test, too. But this is a good thing for the project. And sometimes the environment that is deployed by the automated test is not enough for your test. For example, you may need more hosts or you may need more engines. And in this case, in this case, the environment is not really what you wanted to test. To try to solve part of this problem, I created a local plugin that is called Ovid Partware Fire. And it is very simple. It will just create VMs for you based on what you define on the common line. Like you have a declarative way to say how you want your machines to be created, and it will just create them, link them, and get them ready for usage. The good thing about it is that you don't need to run the full test suite. You can have how many machines you want. But sometimes it may have, like Ovid system tests, it may also have some downsides or some tradeoffs that you have to accept. So it's your job to find what of the tools is better for you or extend Logo to create your own and fix your issue. So these are some common examples. The first one is the deploy itself. We are like creating one engine machine with 8 gigabytes of RAM and two hosts. And we are using a custom patch from Jarrett and from Jenkins. This command will create 3 VMs, one for Ovid engine and two for Ovid hosts, using the RPMs from the build I mentioned. So after that, I will have to run the command to deploy the engine itself. This command will run the engine setup command that is the script that might maintain that will actually deploy the engine and install the RPMs and make sure that everything is running. Or if the automated setup is not good for you, for example, if you have a new option to the install script, you can still run the login to the shell and run engine setup manually using whatever options you want, whatever options you need and doing any other manual testing you want too. I prepared a demo for you. It is like showing this process. I was demonstrating. I should say that the video is not full here. I cut some parts of it to make it more quick to explain here. But I will focus on the times it took to do the tasks. Let me just open it here. Let me just open it here. Okay. So this is the command that we will run. This command will create three VMs, one is engine, one is Ovid engine and it will be named engine two. And the other two ones are two hosts, one with one gigabyte of RAM and the other with one gigabyte of RAM. This is just an example. Nobody would create an host with one gigabyte of RAM because it would be like crazy. But this is the environment we are creating to test. For example, if you just see if the host is correctly attached to the engine, you don't need to create a big host just to see that. This is more than enough. So it's creating network, creating bootstrapping the VMs and we will start deploying. Those steps are all based on scripts or Python code from the Lago plugin or the Lago code itself. All the parts that are marked like doing something related to Ovid are from the Ovid Lago plugin, not for Lago itself. So now it's actually deploying. It's running some shell scripts that are installing RPMs, importing stuff from the repository, et cetera. And this is one of the steps that takes more time because it's download a lot of stuff from internet. Because the Lago, the Ovid patch verifier plugin, it's not, it's one of the trade-offs that it has is that it's not really optimized for downloading. We can't maintain a list of RPM packages that are needed for the testing because it changes a lot. So it usually downloads a lot of stuff that's not needed, but it's the price you pay for being automated. So the deployment took almost five minutes and now I will actually run the engine setup step. It also takes some time because it will actually install the RPMs. And this command we will use a pre-created answer file that works fine for most of the cases, but sometimes it may not work. So if you have a custom answer file that is the configuration file for engine setup, you can pass it for this command and it will be used instead of the default one. So it run engine setup, it took three minutes, 338, and now it is adding the host to the engine automatically for you. It's important to say that everything is based on templates. In this case, as I run the default commands, it uses EL7 templates that are essential seven in for lago, but we have templates for Fedora and it won't work with Overit, but if you have some other project and you create a template for some other distribution, it would work too. But your plugin would need to be able to install packages for that distribution. Now I am running the interface to show that it works. Authenticating. And engine is running. The two hosts I created are there and they are currently installing the host as expected. I have two here, but I could have as many as I wanted without any issue. Doing this manually to test a patch, it takes a lot of time. It's something that really saves a lot of time for us. I got like from zero to the system running in less than ten minutes. It's really a good progress. Yeah. As everything in software, I have filled out slides. I already started commenting about it before, but having it fully automated makes it hard to cache. And the script can install more than one engine simultaneously. The engines attack script from the plugin. But if you want to create more than one engine from the command line and then you have to install it, you can do it. And then you have to create a new one. And then you have to create a new script from the plugin. But if you want to create more than one engine from the command line and do the setup manually running engines attack on both engines, it is possible, it will work. But just the bonus you have from having automatic configuration of hosts won't work with more than one engine, but you can still do it. And one important point of this task of this talk is to show that this is not just for over it. We want to have this used by other people. And the tool was created with being as agnostic as possible from the start. This is why we don't have any virtual code in the core of Lago. So it can be used for several purposes, like testing other virtualization managers, testing some applies, doing some end-to-end testing on some software that is run as a virtual machine. This is all possible to do with Lago by writing plugins. And another topic is we frequently get some questions on, what should I use Lago instead of use some other project that is already popular in the community? The most common question we get is about Vagran. When we started Lago, Vagran wasn't in the best state yet, like it is now. So it was not really an option at that point. But personally, I see Vagran more as a tool for develop code instead of a tool to test code. Like if I have a website and I need Postgres and MongoDB and something like that, I have a Vagran file that built that environment for me to run my website without configuring stuff. I personally don't see it as a tool to build virtualized environments with lots of machines that are integrating there. But it is possible to do. The other tool we frequently get asked is about Avocado. But Avocado is more targeted at testing the hypervirus. For example, it is at a different level than Lago. It is more used to test KVM and KMU. And Lago is more to test the full environment. But still, if you want, you can write a Lago plugin to use Avocado as a test runner instead of the default test runner and use Avocado to run your tests. The same tool is true for Vagran. It is possible to write a Lago plugin that will use Vagran to provide on the VMs instead of Libvert. And so if you really want to use Vagran, if you really want to use Avocado, it is possible to integrate and benefit from the best side of both worlds. Another option is Lava. It is the linear validation tool. It is very similar to Lago, but it is targeted on testing real hardware, like ARM devices and Intel devices. It will run the tests on the real device, not in virtual machines. As far as I know, it is possible to use virtual machines as the target device for Lava, but this is not really popular in the community. That's it. Before I finish the talk, I have a few comments. If you follow those people that say that don't know Lago, don't know Ovid, sorry, we have a stand here in the conference with some demos and stuff. You can bring some gadgets. And also I was talking with some developers. Some of them are using gentle. You also have a stand here. You can compile it around the bottom if you want. You can visit both. If you want to talk with me later, I should be around one of those stands. Also, the slides are in the website of the conference. And that's it. If you have some questions, feel very nice. Thanks! I will take that part off for now. Good luck if it will get me煎아아 WAZ
 I'm pleased to introduce Rafael Martins from Red Hat.  Thank you.  Hi everyone. First of all, thanks for coming.  I'm Rafael Martins. I'm from Red Hat.  I work on the integration team for Overt, where we basically do all the releases and  we maintain all the installation scripts for Overt.  And I am also a gentle developer for more than six years, so most of my background is on  distributing software and packaging software.  And this talk is to show you how we use some tools in Overt to improve the way we test  software and we release software.  So...  It's not working.  So let's start by describing what's the problem we are actually trying to solve.  Basically, virtualization products are complex.  They are complex in the sense that they always have lots of integrations.  For example, we need to integrate with storage, with network.  And when you get some bug, some new feature that you need to implement that will require  you to rebuild the environment to test, you need to reproduce the environment.  And sometimes you can take a really long time building this environment.  Personally, I already spent more than one day trying to reproduce the environment to  fix a bug.  So most of my research on this topic and the tools, the plugin I wrote for Lago were  because I was really tired of spending so much time on reproducing bugs and wanted something  automated.  So the objective of this talk is to show you how you can do the same for your project  if you want to and if you find it viable for your project.  And showing how we do it for Overt.  So I will use Overt as a new case for the talk.  How many people here in the room already use it?  Overt or Rav?  Or at least know what it is?  Okay.  So Overt basically is a virtualization management system.  You have a web application that you call Overt Engine where you can create data centers,  clusters, new virtual machines and stuff.  And it can manage the VMs for you.  It's based on KVM and Libvert that are technologies that are well known in the open source and  Linux community.  And this is what the Overt Engine looks like for those that don't know it.  This is a screenshot from one of our internal instances that Red Hat that we use for testing.  It is not the biggest deployment we know but it's a quite large deployment with three  data centers.  And this is the main dashboard.  We have options and screens to change about everything on the machines.  And now we will talk about the solution we found to make testing and deployment and verifying  of patches easier.  That is Lago, Lago Project.  It is basically a network framework for virtualization that can build environments for you using  Libvert and KVM so you can buy environments with how many machines you want and it will  allow you to integrate those machines and do the tasks needed by using plugins right  in Python.  It's totally extensible.  Almost all of the layers of Lago are extensible by plugins in Python and the core itself just  carries the very basic functionality that is needed to work.  Now we will talk about how we use Lago in Overt.  All the tasks, all the code that connects to Overt and all of the related stuff is on  a splitted plugin that is called Overt Lago and Lago itself does not carry any code that  talks to Overt.  So in the same way that we have Overt Lago for Lago you can have a specific plugin for  another product and another project and use it with Lago.  Lago is just responsible by orchestrating the machines and calling the plugins to do  the job.  We to solve the problem of automated tasks because as release managers we need to be  able to trust the software we are releasing. We need some frequent tasks to make sure that  anything big is broken and one of the solutions for that was an automated test suite that  we have.  It is run by Jenkins frequently and with this we can make sure that the code is ready to  release and that the main flows we support are not broken.  For example we have Lago and one engine and two hosts. This is all we virtualized by Lago  by using the Overt Lago plugin and Lago outside just managing the VMs and the Overt Lago plugin  creating everything that is needed.  This is automated. We have scripts and repository descriptions to cache the RPM files. Lago  can create a local repository that is used to install packages quickly without downloading  from internet every time.  So with this we can run end to end tasks in like 25 minutes, 20 minutes depending on what  we are doing for a really big test suite.  The good things about system tasks is that having automated tasks for you is that you  can quickly see big breakages or big issues on your project. If you have a patch that  is really breaking the basic flows and it is not really bad it may still pass unit tasks  but it won't pass system tasks because it is actually creating the machines, it is actually  creating the VMs, actually trying to add hosts, trying to add storage and so it won't pass.  And it is very well maintained. We have a lot of people writing tests for it and keeping  an eye on it to see if it is really working and make sure that everything is fine.  And a last point that is important to what I will tell you after is that the virtual machines  are left to be used after the tests are running so you can still, after running the test suite,  you can still log in the machines and do something else like run some command, read some logs  or do something else with the machine that you want to verify your patch.  And this is a screenshot of our Jenkins instance. We have lots of jobs for all the over supported  versions. We can look at it and quickly see that, for example, in this day we had some  issue on 3.6 that was breaking most of the builds. And we can look at the logs and see  what's wrong and talk with the person that sent the patch and fix it. That is more important.  Now I will talk about the real point of this talk that is doing manual testing because  automated tests are good, are needed, but we also need manual tests to improve our  workflow when testing patches. We have several jobs on Jenkins that can build custom RPMs  from a Jared patch. We use Jared on Ovid. And you can easily build some RPMs from any  Jared patch and use those RPMs on Ovid system tests to create an environment with the new  functionality you implemented or with the fix you are doing.  An important point is that the developer can do it on its own laptop. So the process is  a bit memory hungry because we are running some heavy stuff, but it's definitely possible  to run it on a laptop and build your RPMs and just test them. And most of the time a lot of actions, more simple patches, don't even need some manual intervention to be tested.  The automated test may just be verifying how the flow that you are affecting with your patch.  And sometimes you won't even need to do any manual test to test it. But like anything in  software basically we had some trade-offs to have good automated tests on Ovid system  tests, we had to do some choices that sometimes are not really, how can I say, they make the  manual testing a bit harder than developers wanted. For example, when we are running, when  we are using Ovid system tests to test a custom patch, you need to always run the full test  to it. It means that if your test is correct, it's fine, you just run it once and be done  with it. But if your test has some issue, you need to iterate. Any time you change the  patch, you need to rerun the full test to get it fixed, to get it to work. And if the  patch changed the behavior of the test, like if you change it, either the new option to  install it, or you have some, change the behavior of some part of the system, it will break  the test. And to be able to use Ovid system tests to test it, you need to fix the test.  This is not a bad thing because this means that Ovid system tests will keep working because  the developer will need to fix the test. And it's a good thing because the developer will  use it to write in tests. So this is not really a downside, but it will spend time because  instead of just fixing your patch, you still need to fix the test, too. But this is a good  thing for the project. And sometimes the environment that is deployed by the automated  test is not enough for your test. For example, you may need more hosts or you may need more  engines. And in this case, in this case, the environment is not really what you wanted  to test. To try to solve part of this problem, I created a local plugin that is called Ovid  Partware Fire. And it is very simple. It will just create VMs for you based on what you  define on the common line. Like you have a declarative way to say how you want your  machines to be created, and it will just create them, link them, and get them ready for usage.  The good thing about it is that you don't need to run the full test suite. You can have  how many machines you want. But sometimes it may have, like Ovid system tests, it may  also have some downsides or some tradeoffs that you have to accept. So it's your job to  find what of the tools is better for you or extend Logo to create your own and fix your  issue. So these are some common examples. The first one is the deploy itself. We are  like creating one engine machine with 8 gigabytes of RAM and two hosts. And we are using a custom  patch from Jarrett and from Jenkins. This command will create 3 VMs, one for Ovid engine and  two for Ovid hosts, using the RPMs from the build I mentioned. So after that, I will have  to run the command to deploy the engine itself. This command will run the engine setup command  that is the script that might maintain that will actually deploy the engine and install  the RPMs and make sure that everything is running. Or if the automated setup is not good  for you, for example, if you have a new option to the install script, you can still run the  login to the shell and run engine setup manually using whatever options you want, whatever options  you need and doing any other manual testing you want too. I prepared a demo for you. It  is like showing this process. I was demonstrating. I should say that the video is not full here.  I cut some parts of it to make it more quick to explain here. But I will focus on the  times it took to do the tasks. Let me just open it here. Let me just open it here.  Okay. So this is the command that we will run. This command will create three VMs, one  is engine, one is Ovid engine and it will be named engine two. And the other two ones  are two hosts, one with one gigabyte of RAM and the other with one gigabyte of RAM. This  is just an example. Nobody would create an host with one gigabyte of RAM because it would  be like crazy. But this is the environment we are creating to test. For example, if you  just see if the host is correctly attached to the engine, you don't need to create a  big host just to see that. This is more than enough. So it's creating network, creating  bootstrapping the VMs and we will start deploying. Those steps are all based on scripts or Python  code from the Lago plugin or the Lago code itself. All the parts that are marked like  doing something related to Ovid are from the Ovid Lago plugin, not for Lago itself. So  now it's actually deploying. It's running some shell scripts that are installing RPMs,  importing stuff from the repository, et cetera. And this is one of the steps that takes more  time because it's download a lot of stuff from internet. Because the Lago, the Ovid  patch verifier plugin, it's not, it's one of the trade-offs that it has is that it's not  really optimized for downloading. We can't maintain a list of RPM packages that are needed  for the testing because it changes a lot. So it usually downloads a lot of stuff that's  not needed, but it's the price you pay for being automated. So the deployment took almost  five minutes and now I will actually run the engine setup step. It also takes some time  because it will actually install the RPMs. And this command we will use a pre-created  answer file that works fine for most of the cases, but sometimes it may not work. So if  you have a custom answer file that is the configuration file for engine setup, you can  pass it for this command and it will be used instead of the default one. So it run engine  setup, it took three minutes, 338, and now it is adding the host to the engine automatically  for you. It's important to say that everything is based on templates. In this case, as I  run the default commands, it uses EL7 templates that are essential seven in for lago, but  we have templates for Fedora and it won't work with Overit, but if you have some other  project and you create a template for some other distribution, it would work too. But  your plugin would need to be able to install packages for that distribution. Now I am running  the interface to show that it works. Authenticating. And engine is running. The two hosts I created  are there and they are currently installing the host as expected. I have two here, but  I could have as many as I wanted without any issue. Doing this manually to test a patch,  it takes a lot of time. It's something that really saves a lot of time for us. I got like  from zero to the system running in less than ten minutes. It's really a good progress.  Yeah. As everything in software, I have filled out slides. I already started commenting about  it before, but having it fully automated makes it hard to cache. And the script can install  more than one engine simultaneously. The engines attack script from the plugin. But if you  want to create more than one engine from the command line and then you have to install  it, you can do it. And then you have to create a new one. And then you have to create a new  script from the plugin. But if you want to create more than one engine from the command  line and do the setup manually running engines attack on both engines, it is possible, it  will work. But just the bonus you have from having automatic configuration of hosts won't  work with more than one engine, but you can still do it.  And one important point of this task of this talk is to show that this is not just for  over it. We want to have this used by other people. And the tool was created with being  as agnostic as possible from the start. This is why we don't have any virtual code in the  core of Lago. So it can be used for several purposes, like testing other virtualization  managers, testing some applies, doing some end-to-end testing on some software that is  run as a virtual machine. This is all possible to do with Lago by writing plugins.  And another topic is we frequently get some questions on, what should I use Lago instead  of use some other project that is already popular in the community? The most common question  we get is about Vagran. When we started Lago, Vagran wasn't in the best state yet, like  it is now. So it was not really an option at that point. But personally, I see Vagran  more as a tool for develop code instead of a tool to test code. Like if I have a website  and I need Postgres and MongoDB and something like that, I have a Vagran file that built  that environment for me to run my website without configuring stuff. I personally don't  see it as a tool to build virtualized environments with lots of machines that are integrating  there. But it is possible to do. The other tool we frequently get asked is about Avocado.  But Avocado is more targeted at testing the hypervirus. For example, it is at a different  level than Lago. It is more used to test KVM and KMU. And Lago is more to test the full  environment. But still, if you want, you can write a Lago plugin to use Avocado as a test  runner instead of the default test runner and use Avocado to run your tests. The same  tool is true for Vagran. It is possible to write a Lago plugin that will use Vagran to  provide on the VMs instead of Libvert. And so if you really want to use Vagran, if you  really want to use Avocado, it is possible to integrate and benefit from the best side  of both worlds. Another option is Lava. It is the linear validation tool. It is very similar  to Lago, but it is targeted on testing real hardware, like ARM devices and Intel devices.  It will run the tests on the real device, not in virtual machines. As far as I know,  it is possible to use virtual machines as the target device for Lava, but this is not  really popular in the community. That's it. Before I finish the talk, I have a few comments.  If you follow those people that say that don't know Lago, don't know Ovid, sorry, we have  a stand here in the conference with some demos and stuff. You can bring some gadgets. And  also I was talking with some developers. Some of them are using gentle. You also have a  stand here. You can compile it around the bottom if you want. You can visit both. If  you want to talk with me later, I should be around one of those stands. Also, the slides  are in the website of the conference. And that's it. If you have some questions, feel  very nice. Thanks! I will take that part off  for now. Good luck if it will get me煎아아 WAZ
Improving_your_virtualization_development_workflow_with_Lago_0_3467.png Improving_your_virtualization_development_workflow_with_Lago_3468_12947.png Improving_your_virtualization_development_workflow_with_Lago_12948_15371.png Improving_your_virtualization_development_workflow_with_Lago_15372_18575.png Improving_your_virtualization_development_workflow_with_Lago_18576_21251.png Improving_your_virtualization_development_workflow_with_Lago_21252_23159.png Improving_your_virtualization_development_workflow_with_Lago_23160_23219.png Improving_your_virtualization_development_workflow_with_Lago_23220_23975.png Improving_your_virtualization_development_workflow_with_Lago_23976_26171.png Improving_your_virtualization_development_workflow_with_Lago_26172_29843.png Improving_your_virtualization_development_workflow_with_Lago_29844_30191.png Improving_your_virtualization_development_workflow_with_Lago_30192_30287.png Improving_your_virtualization_development_workflow_with_Lago_30288_30359.png Improving_your_virtualization_development_workflow_with_Lago_30360_30551.png Improving_your_virtualization_development_workflow_with_Lago_30552_30623.png Improving_your_virtualization_development_workflow_with_Lago_30624_34187.png Improving_your_virtualization_development_workflow_with_Lago_34188_34235.png Improving_your_virtualization_development_workflow_with_Lago_34236_34355.png Improving_your_virtualization_development_workflow_with_Lago_34356_34451.png Improving_your_virtualization_development_workflow_with_Lago_34452_34823.png Improving_your_virtualization_development_workflow_with_Lago_34824_35087.png Improving_your_virtualization_development_workflow_with_Lago_35088_35351.png Improving_your_virtualization_development_workflow_with_Lago_35352_35831.png Improving_your_virtualization_development_workflow_with_Lago_35832_36311.png Improving_your_virtualization_development_workflow_with_Lago_36312_36455.png Improving_your_virtualization_development_workflow_with_Lago_36456_36707.png Improving_your_virtualization_development_workflow_with_Lago_36708_36887.png Improving_your_virtualization_development_workflow_with_Lago_36888_36935.png Improving_your_virtualization_development_workflow_with_Lago_36936_37787.png Improving_your_virtualization_development_workflow_with_Lago_37788_39419.png Improving_your_virtualization_development_workflow_with_Lago_39420_39540.png