Public Cloud Watcher
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Smorodsky, Anton
In this presentation I would like to talk about Public Cloud Watcher. Tool used by SUSE SLE QE team to monitor Public Cloud providers ( Azure , AWS , GCE ) for testing leftovers and delete them. I will describe: 1. tool itself ( internal architecture and features it provides ) 2. how we maintain running instance of PCW ( it is deployed in 3 docker containers maintained by mixture of docker-compose, docker files and some bash scripts on top it )
 Hello, my name is Anton Smorodskiy and today I will do a short introduction into framework for monitor and cleanup cloud service providers. Public Cloud to watch. Let's start. First of all, a few words about myself. I am working in IT since 2005 using Linux's main tool for work and funds since 2007. Main areas of interest before joining Susie was Java and automated testing, mainly in area of online retail stores. After joining Susie, focus shifted to Perl, Python and testing of Sli and OpenSusi. One of the areas is testing of Sli in public cloud. And currently my favorite distro is OpenSusi Lib. So I am installing it everywhere I can. We will start from stating the problem which Public Cloud to watchers is trying to solve, then switch to the tool itself in going through its main features. Also we will speak about internals of Public Cloud to watcher. Next we will speak about HashiCorp Vault, what this tool generally for and about our very specific use case of this tool. And we will discuss how we currently maintain running instance of Public Cloud to watcher. What setup we have done to keep it running and not worry about any potential environment breakage. The last topic will be about future plans for this project. At my daily work, one of the main things what I am doing is testing how Sli behaves as a virtual machine in different cloud service providers. Azure, AWS and GC at the moment. All testing related to Public Cloud providers is happening in OpenQA in automated way. I assume majority of people who attending OpenSusi conferences know at least something about OpenQA. For those who don't know or want to know more, I will recommend to visit Open.QA or find some nice talks about OpenQA in previous conferences. I am sure that there were more than one talk like this. But let's get back to our main topic. So OpenQA uploading Sli image into dedicated cloud, then creating VM using this image and then running some testing against it. And after that trying to clean up after itself by deleting all created entities. Keep in mind that I said creating VM, but actually it's much more than that. Sli service VM, every public cloud provider creating a lot of different entities, subnets, resource groups, disk and disk images, etc. After successful test execution, we have logic which clean up all created entities. But of course there is plenty of options how things might go wrong. And regardless that we keep improving our code to do clean up always, there is still room for some unexpected behavior from OpenQA side or from provider side which would not allow to finish clean up. All what I just described needs to be multiplied by diversity of public cloud providers. Like I said, VM creation usually means creation of some additional resources for this VM. And every cloud service provider have their own understanding what kind of resources will be created and how they will behave when you trying to delete VM because some of them will be automatically clean it up but others know and there is a lot of different cases when this behavior is differs. So basically whenever something goes wrong in OpenQA, you not simply check that VM was deleted but also check some other entities and you need to keep in mind all differences of current provider. Currently we starting around 300 VMs daily in OpenQA in all three providers together and this number will suppose to grow in nearest future. So obviously there is no chance to keep all this in manual check up. So we need to have some logic which will double check that our test don't burn extra money. Also it would be nice to have it outside OpenQA to not get into a trap that same bug which would invalidate clean up in test level will do the same with our less chance clean up logic. Also it would be nice to have everything in one place to ease maintenance. Another problem not directly related to what I just described is coming from the fact that creation of VM requires credentials which you need to bypass to the test in secure way and make sure that unattended person will not be able to use them. To address first problem we created public cloud watcher. It's published in GitHub so feel free to use it, learn from it or contribute. It's written in Python using Django framework. Django was selected because of the cheap way to get web UI. Public cloud watcher monitor clean up and notify about leftovers in supported providers. Currently we support three cloud service providers. Microsoft Azure, Amazon AWS and Google Cloud Enterprise. For each provider it using native Python API bindings from provider to communicate with the cloud. Public cloud watcher is currently used by several teams inside our company. Each team has its own credentials to access clouds, its own images naming convention. Its own flows which images need to be deleted and when. This is why in public cloud watcher we have notion of namespaces which allow us define these differences per team. Also in some cases public cloud accounts may be used not only by our open QA automation but also some other people and other automated workflows may create resources in the cloud. And we should not touch them obviously so we need to implement something smarter than delete everything older than next days. Do not interfere with work happening outside open QA. For this we decided to use feature available in all three providers. We decided to set tags on the resource. So when open QA creates VM in its VM it settings two tags open QA created by which generally give a hint for public cloud watcher that this VM should be monitored and open QA TTL which store amount of time after which public cloud watcher may delete VM if open QA fail it to do so after the test run. Internals of public cloud watcher may be logically divided into several groups. One group represented by classes responsible for actual interaction with providers. It contains dedicated classes for each provider which holds all provider specific logic. Each class knows how to authenticate with certain provider how to query for resources and which resources actually needs to be queried. Another group responsible for actual VM cleanup. It has process which periodically loops over all providers in all defined namespaces and collect VMs according to tags which I mentioned before. Then this VM serialized into local SQL light database. Then it tries to delete VMs living together, sorry, living longer than TTL defined for them. And at the end it send email notification about VMs which was not deleted so potential human involvement is needed. Also there is a group responsible for cleanup of everything else. This group holds knowledge about specialties of every provider and what kind of additional entities are created together with VMs. To clean up them separately because there is a lot of cases that after VM deletion provider not delete these entities. All this cleanup and notification flows can be turned on and off on global level or per namespace. Such configuration stored in PCW e-file which public cloud watcher reads on startup. Also local SQL light database with cache list of VMs can be accessed via web interface which allows to browse through list of VMs and do some basic search and manually trigger deletion of some VMs which was not yet cleaned up automatically. Now let's discuss second-raised problem, credentials. After some discussions around the problem we realized that to provide really secure flow to operate credentials within OpenQA we would need to build pretty complex flow which still might be broken easily. So we choose another path. We decided not to hide credentials at all but define short tl and very limited permission set for them. To achieve this we pick the project from company which created Terraform which we also used quite a lot in the internals of our testing approach. The name of the project is Vault. The project page describes it as a following. Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using UI, CLI or HTTP API. Vault also has notion of namespaces so it perfectly fit in our model. Another useful feature from Vault is ability to request temporary credentials from provider with certain set of permissions. Vault will keep track on this requested tokens and may request token deletion from the provider. Vault is used by OpenQA tests so they have ability to create, delete VMs and public cloud watcher so it can query our accounts and delete leftovers in case there are any. We using very basic Vault setup which actually describe it as four testing purposes only and not using prod environments. We not using any offensive Vault storage types which would give plenty of options where persistently store credentials. Indeed we using cram storage which means 100% data loss in case of reboot shutdown of Vault. But it's totally fine for us because we generating temporary credentials anyway. Another not recommended way of using Vault is that we allowing not encrypted connections to Vault. But I would tell more about that part when I will describe our prod instance setup. So last but not least let's speak about our prod instance where we have public cloud watcher and Vault running. At the beginning it was just random VM running somewhere. But over time with more people start using it and more tests run rely on it. We start realizing that we need more stable ground. If Vault is down no test will be able to finish because test need Vault to get temporary credentials which will allow create delete entities in the cloud. Also if public cloud watcher is down test are able to run but there is a risk of getting big bills from Microsoft, Amazon and Google. So at some point we decided to move to state where we can recreate whole setup on clean machine within 10 minutes. There is plenty of options how one can back up some infrastructure in the code. Each one of these options has some pros and each one has some cons. Also sometimes it's simply matter of taste. After going through several different options we end up on choosing containers as building material for our setup. The idea end up as separate repository which contains Docker files for public cloud watcher, Vault and Engines, Docker compose file which tie containers together, some bash scripts which automate everything what should be done outside Docker files like installing Docker on the host, getting SSL certificate into host which would be needed for communication with OpenQA, check out of latest version of public cloud watcher from GitHub and etc. Set of vault click commands which to vault configuration from scratch. Set up all needed namespaces and upload root credentials for each cloud service provider in each namespace which vault will later use to generate delete temporary credential on request. Public cloud watcher container based on Python 3 container. Whenever new image is built it will take latest version of public cloud watcher from GitHub and PCVW.ini with details about which namespaces needs to be monitored and how. Also container has attached storage to a low SQL lite database with instances cache, survive container recreation. Vault container is based on official container from HashiCorp and contains small config which just defines in-mem storage and disables web interface of vault. Most of business logic is in bash scripts with vault click commands which will do initial init of fresh vault instance and set up all needed namespaces and secrets. So basically our approach is that we are doing full automated install of both entities, public cloud watcher and vault every time from scratch on clean container but because it's fully automated we basically can do it fast and because it's based on stable container versions we are in a safe mode that we don't get so basically we have reproducible builds at some point. To simplify vault setup but keep it secure we introduce third container with engines. In Jinx is playing reverse proxy role here by encapsulating HTTP into HTTPS and a single entry point which broker request appropriate container, public cloud watcher and vault. This trick allows us to keep vault unsecured from setting up point of view but secure from usability point of view. This is how we work around what vault is not recommending to do by allowing plain HTTP connections. Now let's speak about future plans. First and in my opinion most important we need to improve our code coverage with unit tests. Not that we don't have them at all but let's say coverage is far away from 100%. Another problem which posers me as a person who maintain public cloud watcher is that whenever there is some issue in the cloud for example Azure changed API in non-backward compatible way which already happened several times. I'm start getting email notification about exception every x minutes. From one side I do want to be notified about the problem so I would start acting as soon as possible but on other hand after I have read first email I can skip next dozen which would not say something new. So notification flow needs to become smarter and be aware whether it is some new problem which I need to be notified or it is something what I have already seen. Also currently web interface allow only manage VMs but in effect public cloud watcher is cleaning a lot of other entities. It would be nice to see them in the web UI and have better understanding what actually clean it and when. For now to solve this problem I usually grab in log file of public cloud watcher. That's all what I wanted to say. I will be happy to answer any additional questions during conference or any time after and I would be even more happy if someone will consider using public cloud watcher together with us.
 Hello, my name is Anton Smorodskiy and today I will do a short introduction into framework  for monitor and cleanup cloud service providers.  Public Cloud to watch.  Let's start.  First of all, a few words about myself.  I am working in IT since 2005 using Linux's main tool for work and funds since 2007.  Main areas of interest before joining Susie was Java and automated testing, mainly in  area of online retail stores.  After joining Susie, focus shifted to Perl, Python and testing of Sli and OpenSusi.  One of the areas is testing of Sli in public cloud.  And currently my favorite distro is OpenSusi Lib.  So I am installing it everywhere I can.  We will start from stating the problem which Public Cloud to watchers is trying to solve,  then switch to the tool itself in going through its main features.  Also we will speak about internals of Public Cloud to watcher.  Next we will speak about HashiCorp Vault, what this tool generally for and about our very  specific use case of this tool.  And we will discuss how we currently maintain running instance of Public Cloud to watcher.  What setup we have done to keep it running and not worry about any potential environment  breakage.  The last topic will be about future plans for this project.  At my daily work, one of the main things what I am doing is testing how Sli behaves as a  virtual machine in different cloud service providers.  Azure, AWS and GC at the moment.  All testing related to Public Cloud providers is happening in OpenQA in automated way.  I assume majority of people who attending OpenSusi conferences know at least something  about OpenQA.  For those who don't know or want to know more, I will recommend to visit Open.QA or find  some nice talks about OpenQA in previous conferences.  I am sure that there were more than one talk like this.  But let's get back to our main topic.  So OpenQA uploading Sli image into dedicated cloud, then creating VM using this image and  then running some testing against it.  And after that trying to clean up after itself by deleting all created entities.  Keep in mind that I said creating VM, but actually it's much more than that.  Sli service VM, every public cloud provider creating a lot of different entities, subnets,  resource groups, disk and disk images, etc.  After successful test execution, we have logic which clean up all created entities.  But of course there is plenty of options how things might go wrong.  And regardless that we keep improving our code to do clean up always, there is still  room for some unexpected behavior from OpenQA side or from provider side which would not  allow to finish clean up.  All what I just described needs to be multiplied by diversity of public cloud providers.  Like I said, VM creation usually means creation of some additional resources for this VM.  And every cloud service provider have their own understanding what kind of resources will  be created and how they will behave when you trying to delete VM because some of them  will be automatically clean it up but others know and there is a lot of different cases  when this behavior is differs.  So basically whenever something goes wrong in OpenQA, you not simply check that VM was  deleted but also check some other entities and you need to keep in mind all differences  of current provider.  Currently we starting around 300 VMs daily in OpenQA in all three providers together  and this number will suppose to grow in nearest future.  So obviously there is no chance to keep all this in manual check up.  So we need to have some logic which will double check that our test don't burn extra money.  Also it would be nice to have it outside OpenQA to not get into a trap that same bug which  would invalidate clean up in test level will do the same with our less chance clean up  logic.  Also it would be nice to have everything in one place to ease maintenance.  Another problem not directly related to what I just described is coming from the fact that  creation of VM requires credentials which you need to bypass to the test in secure way  and make sure that unattended person will not be able to use them.  To address first problem we created public cloud watcher.  It's published in GitHub so feel free to use it, learn from it or contribute.  It's written in Python using Django framework.  Django was selected because of the cheap way to get web UI.  Public cloud watcher monitor clean up and notify about leftovers in supported providers.  Currently we support three cloud service providers.  Microsoft Azure, Amazon AWS and Google Cloud Enterprise.  For each provider it using native Python API bindings from provider to communicate with  the cloud.  Public cloud watcher is currently used by several teams inside our company.  Each team has its own credentials to access clouds, its own images naming convention.  Its own flows which images need to be deleted and when.  This is why in public cloud watcher we have notion of namespaces which allow us define  these differences per team.  Also in some cases public cloud accounts may be used not only by our open QA automation  but also some other people and other automated workflows may create resources in the cloud.  And we should not touch them obviously so we need to implement something smarter than  delete everything older than next days.  Do not interfere with work happening outside open QA.  For this we decided to use feature available in all three providers.  We decided to set tags on the resource.  So when open QA creates VM in its VM it settings two tags open QA created by which generally  give a hint for public cloud watcher that this VM should be monitored and open QA TTL  which store amount of time after which public cloud watcher may delete VM if open QA fail  it to do so after the test run.  Internals of public cloud watcher may be logically divided into several groups.  One group represented by classes responsible for actual interaction with providers.  It contains dedicated classes for each provider which holds all provider specific logic.  Each class knows how to authenticate with certain provider how to query for resources  and which resources actually needs to be queried.  Another group responsible for actual VM cleanup.  It has process which periodically loops over all providers in all defined namespaces and  collect VMs according to tags which I mentioned before.  Then this VM serialized into local SQL light database.  Then it tries to delete VMs living together, sorry, living longer than TTL defined for  them.  And at the end it send email notification about VMs which was not deleted so potential  human involvement is needed.  Also there is a group responsible for cleanup of everything else.  This group holds knowledge about specialties of every provider and what kind of additional  entities are created together with VMs.  To clean up them separately because there is a lot of cases that after VM deletion provider  not delete these entities.  All this cleanup and notification flows can be turned on and off on global level or per  namespace.  Such configuration stored in PCW e-file which public cloud watcher reads on startup.  Also local SQL light database with cache list of VMs can be accessed via web interface  which allows to browse through list of VMs and do some basic search and manually trigger  deletion of some VMs which was not yet cleaned up automatically.  Now let's discuss second-raised problem, credentials.  After some discussions around the problem we realized that to provide really secure  flow to operate credentials within OpenQA we would need to build pretty complex flow  which still might be broken easily.  So we choose another path.  We decided not to hide credentials at all but define short tl and very limited permission  set for them.  To achieve this we pick the project from company which created Terraform which we also used  quite a lot in the internals of our testing approach.  The name of the project is Vault.  The project page describes it as a following.  Secure, store and tightly control access to tokens, passwords, certificates, encryption  keys for protecting secrets and other sensitive data using UI, CLI or HTTP API.  Vault also has notion of namespaces so it perfectly fit in our model.  Another useful feature from Vault is ability to request temporary credentials from provider  with certain set of permissions.  Vault will keep track on this requested tokens and may request token deletion from the provider.  Vault is used by OpenQA tests so they have ability to create, delete VMs and public  cloud watcher so it can query our accounts and delete leftovers in case there are any.  We using very basic Vault setup which actually describe it as four testing purposes only  and not using prod environments.  We not using any offensive Vault storage types which would give plenty of options where persistently  store credentials.  Indeed we using cram storage which means 100% data loss in case of reboot shutdown of Vault.  But it's totally fine for us because we generating temporary credentials anyway.  Another not recommended way of using Vault is that we allowing not encrypted connections  to Vault.  But I would tell more about that part when I will describe our prod instance setup.  So last but not least let's speak about our prod instance where we have public cloud  watcher and Vault running.  At the beginning it was just random VM running somewhere.  But over time with more people start using it and more tests run rely on it.  We start realizing that we need more stable ground.  If Vault is down no test will be able to finish because test need Vault to get temporary credentials  which will allow create delete entities in the cloud.  Also if public cloud watcher is down test are able to run but there is a risk of getting  big bills from Microsoft, Amazon and Google.  So at some point we decided to move to state where we can recreate whole setup on clean  machine within 10 minutes.  There is plenty of options how one can back up some infrastructure in the code.  Each one of these options has some pros and each one has some cons.  Also sometimes it's simply matter of taste.  After going through several different options we end up on choosing containers as building  material for our setup.  The idea end up as separate repository which contains Docker files for public cloud watcher,  Vault and Engines, Docker compose file which tie containers together, some bash scripts  which automate everything what should be done outside Docker files like installing Docker  on the host, getting SSL certificate into host which would be needed for communication  with OpenQA, check out of latest version of public cloud watcher from GitHub and etc.  Set of vault click commands which to vault configuration from scratch.  Set up all needed namespaces and upload root credentials for each cloud service provider  in each namespace which vault will later use to generate delete temporary credential on  request.  Public cloud watcher container based on Python 3 container.  Whenever new image is built it will take latest version of public cloud watcher from GitHub  and PCVW.ini with details about which namespaces needs to be monitored and how.  Also container has attached storage to a low SQL lite database with instances cache, survive  container recreation.  Vault container is based on official container from HashiCorp and contains small config which  just defines in-mem storage and disables web interface of vault.  Most of business logic is in bash scripts with vault click commands which will do initial  init of fresh vault instance and set up all needed namespaces and secrets.  So basically our approach is that we are doing full automated install of both entities, public  cloud watcher and vault every time from scratch on clean container but because it's fully  automated we basically can do it fast and because it's based on stable container versions  we are in a safe mode that we don't get so basically we have reproducible builds at some  point.  To simplify vault setup but keep it secure we introduce third container with engines.  In Jinx is playing reverse proxy role here by encapsulating HTTP into HTTPS and a single  entry point which broker request appropriate container, public cloud watcher and vault.  This trick allows us to keep vault unsecured from setting up point of view but secure from  usability point of view.  This is how we work around what vault is not recommending to do by allowing plain HTTP  connections.  Now let's speak about future plans.  First and in my opinion most important we need to improve our code coverage with unit  tests.  Not that we don't have them at all but let's say coverage is far away from 100%.  Another problem which posers me as a person who maintain public cloud watcher is that  whenever there is some issue in the cloud for example Azure changed API in non-backward  compatible way which already happened several times.  I'm start getting email notification about exception every x minutes.  From one side I do want to be notified about the problem so I would start acting as soon  as possible but on other hand after I have read first email I can skip next dozen which  would not say something new.  So notification flow needs to become smarter and be aware whether it is some new problem  which I need to be notified or it is something what I have already seen.  Also currently web interface allow only manage VMs but in effect public cloud watcher is  cleaning a lot of other entities.  It would be nice to see them in the web UI and have better understanding what actually  clean it and when.  For now to solve this problem I usually grab in log file of public cloud watcher.  That's all what I wanted to say.  I will be happy to answer any additional questions during conference or any time after and I  would be even more happy if someone will consider using public cloud watcher together  with us.
openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_0_131.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_132_179.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_180_263.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_264_575.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_576_1595.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_1596_2435.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_2436_2735.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_2736_8627.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_8628_10115.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_10116_12335.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_12336_15887.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_15888_19715.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_19716_22019.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_22020_27251.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_27252_30107.png openSUSE_Virtual_Conference_2021_-_Public_Cloud_Wa_2d3ks3T0q_E_30108_30258.png