RobotMK: Extend Checkmk with Robot Framework E2E tests
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Meggle, Simon
Robotmk: How to extend the monitoring system Checkmk with checks from the user's perspective So you think you are comprehensively monitoring your business-critical applications? Are you using Nagios, Naemon, Icinga2, Checkmk et cetera for this? Then... let me say that, alas, you only rely on a hypothesis, because all these IT infrastructure monitoring tools have their natural limits. Robotmk extends the monitoring capabilities of Checkmk to the application level. It integrates the results of End2End tests done with Robot Framework - which is by far the most popular and versatile tool for automated testing. My talk first covers the basic concepts of Robot Framework and Checkmk. You will learn the added value by RobotMK and the different monitoring strategies it offers. Robot Framework/RobotMK has been chosen as the E2E monitoring solution for the application landscape of the swiss government.
 Hi, my name is Simon Meckle. I'm very happy to be a speaker at this FOSSTEM conference this year. The topic of my presentation is Robot MK, how you can integrate the results from Robot Framework into the monitoring system CheckMK. Let's get started. I'm coming from the south of Germany, close to Munich, and I've been working with open source monitoring systems like Nagios for 20 years now and the last 10 years I've been working as a consultant. My focus topics are data center automation with Ansible, monitoring, I'm specialized on CheckMK, and test automation with focus on Robot Framework. And last but not least, I'm generally interested in Python development. So first let's talk about IT infrastructure monitoring. In my opinion, the best solution for this is CheckMK. CheckMK was written 2008 by Matthias Kettner as a kind of Nagios add-on, first of all to only solve problems with large Nagios installations and its configuration limitations. But it's gradually evolved and became a fully integrated monitoring system, which aims to be an all-in-one solution for monitoring everything while still being user-friendly and very flexible. It comes with more than 2,800 curated check plug-ins and is available in two editions. This is the raw edition, open source, and the enterprise edition with some additional features and enterprise support. The most important fact you should know about CheckMK is that monitoring systems like Nagios, Neiman, CheckMK, Isinga, etc. All together are able to monitor up to OC layer 7. And strictly speaking, they all start on OC layer 3 by sending ICMP packets to check if a host is up. But didn't we forget the applications? How do we monitor applications? How do we monitor if users can really work with an application and how performant? The answer is by default there is no such monitoring available. And don't get me wrong, this is not a criticism of any of these monitoring systems. It's simply too complicated that these systems could provide a kind of generic way to monitor applications, to monitor what the user really uses and sees. Layer 7 is called the application layer, but this is a kind of false friend, I think. It's only the interface layer to the applications. Layer 7 only contains the protocol, for example HTTP, to be able to contact a web server. The content of HTTP packets is the full responsibility of the application itself. And this is what we should check. And this is my second disclaimer. I don't say that we do not need checks in layers 3 to 7. It's absolutely necessary to have them, but they are not the whole truth. What I want to say is, in the end, it's all based on a hypothesis. I found a nice paragraph in this book here, weniger schlecht programmieren, less bad coding, which I think perfectly fits. Observations want to become hypotheses. It's not easy to only record what you can see. An untrained observer tends to see a man running after the bus, but in reality, the man and the bus are moving in the same direction accidentally. It's nearly impossible to observe something freed from hypothesis. This means we are monitoring through the best of our ability the application preconditions and hope that the applications are happy with that. And if the application is OK, we tend to believe that everything we already monitor is everything the application needs to work. We are displacing the question if there could be more reasons to break the application, and unfortunately, there are many more. A great way to test applications automatically is the open source tool Robot Framework. Robot Framework is a generic test automation framework written 2005 by Pekka Klärk at Nokia Siemens Networks. And since 2008, open source. It's completely written in Python. And there are two features that set Robot Framework apart from other testing tools. It's the keyword-driven approach, which I will explain in the next slide, and the library concept. Robot Framework itself doesn't test anything. You import libraries with special keywords for web testing, image recognition, REST calls, and so on and so on. And this makes Robot Framework for me to a Swiss Army knife. In my opinion, one of the best reasons to take Robot Framework as a test tool is that it's open source and independent from any software vendor. The decision for Robot Framework is not a decision for a company. It's the decision for the lingua franca of test automation. Robot Framework will always be the property of the community. So what are keywords now? Simply explain keywords abstract source code. Source codes can be written in Java or in Python, mainly in Python. And they can be written and used like functions. They can take parameters and return values. They are case insensitive. They allow spaces, which is not a big feature, but makes the code much more readable. And a question I often get asked is, why should I use Robot Framework? There's Selenium for Python. Why should I use Robot Framework? It's absolutely OK. You can see here a small snippet of Python code using Selenium. And for sure, this code would need some better exception handling and methods to integrate the data into other systems. And if you are writing hundreds of those tests, for example, you will of course try to separate the common code from the test code. And in the end, you will have some kind of a self-written framework. And this is exactly what Robot Framework already does for you. In my opinion, there is no reason not to use Robot Framework. The same test as you can see here in Python can also be written in Robot Framework using the Selenium library. It's a much more cleaner code. The yellow words here are the keywords, which I would say is also readable by anyone who is not a real programmer. But Robot Framework is not a programming language itself. It's more like an additional abstraction layer for existing Python modules. So in theory now, we know CheckMK and Robot Framework. Let's talk about the integration of Robot Framework into CheckMK. And this can be done with my project RobotMK. RobotMK comes as a MKP extension, MK package. It's a collection of files in the end for CheckMK to integrate the results of Robot Framework. I've started this project in November 2019 and presented a proof of concept at the RobotCon conference in January 2020 in Helsinki. I got a lot of good feedback, even though hardly anyone knew Nagyus or CheckMK. And I continued my work. And thanks to some customers of mine, I always could further develop and improve the module. This slide normally takes up to five minutes to explain. Time is rare. For now, it's only too important to know for you that RobotMK consists of mainly two components. Like any other CheckMK agent plug-in, the RobotMK plug-in gets triggered by the agent on the monitored host where the Robot Framework tests are. Other plug-ins collect system data, memory consumption, or something. And the RobotMK plug-in executes Robot Framework tests and transfers the XML result back to the CheckMK server. On the CheckMK server, the check script is responsible then to evaluate that Robot XML. By default, a Robot Framework test already has a state. This is if the whole test was successful or not. And the RobotMK rules, these are normal CheckMK rules, enable you to enrich the result by adding more monitoring tests. For example, monitoring the run times of sweet tests and keywords. In the following demo, I will show you a start from scratch. We will integrate a small Robot Framework test on a Windows 10 machine and integrate this result into CheckMK. And I will show you how to install RobotMK and how to set some settings and show you features of RobotMK. Let's start. Okay, here we are on a Windows 10 VM. The first thing I want to show you is the Robot Framework test itself. It's a very simple test. As you can see here, it consists of two test cases. Each of those is connected to seleniumeasy.com and executing a dummy test. I can execute this test by hand so that you see what it's doing. It's starting Chrome. Enter some text and expect the page to display the entered text in a separate field. So that's all. The first thing I have to do now is to copy the directory of this RobotTest. This is seleniume test into the agent directory of the CheckMK agent. The RobotMK plugin expects the Robot Framework test case in a certain directory. This is called Robot. And within the Robot directory, I'm pasting this directory. We only need this here. So this is the first step. With RobotMK, you can control where RobotTest cases should be executed and how. Let me show you how this works. This is a fresh installation of CheckMK with one monitored host. This is my Windows 10 VM where the seleniume test should be executed by the CheckMK agent. Now I open the host and service parameters and search for the rule Robot. This brings up three rules. I'm taking the agent plugin rule. This rule controls how agent installation packages should look like individually per host. This individually means the MSI package I'm now creating for this Windows host in a few minutes will contain everything. The RobotMK plugin and a YAML definition file which is read by the plugin and which tells the plugin which Robot suites should be executed. I create this rule here. The first thing I do is choose explicit host. I only want to apply this rule on my Windows VM. Here I can choose to deploy the plugin. Of course there are two modes of execution. This is the spoolier mode which means the plugin gets executed externally for example by a task scheduler or the asynchronous mode which is default. This means the CheckMK agent executes the plugin but in another interval than the normal CheckMK interval which is one minute per default. I'm choosing this interval and I'm setting let's say two. Robot suites directory is an option where I can override the default directory. We do not need this. Robot framework test suites allows me to add a list of test suites I want to execute. This takes the directory names of suites I want to execute. Remember this is the name of the directory we already have copied into the Robot directory. All the options you can see here are translated at runtime to Robot framework command line options. So you can choose another name for the suite. You can black and white list by tags. You can load variables, a variable file into the Robot test. And the last option is a special CheckMK option. This allows you to assign this result to another host defined in CheckMK. We don't need this at the moment. This is everything we set up here. The last two options define how the agent output should be encoded. If there is too much content you can also choose ZLIP to compress the agent output. And this is a setting which controls how many days you want to keep the log files on this host. I click on save. And now you can see that CheckMK offers me to bake a new agent version. I choose bake agents. And now watch this list. Currently we only have one version of the agent. This is the generic version which is applicable to every host. And now CheckMK will bake a special version for my Windows VM. This is what we now have. This is the MSI package with a new hash. And here I see the settings which come from the RobotMK rule. The next step is now to install this agent on the Windows VM. Now I'm back on the Windows machine. I'm logging into CheckMK. And in the VATO configuration I choose monitoring agents and download the new MSI package. You can also set up the CheckMK agent to do automatic unattended updates. I didn't do this here for demonstration purposes. So let's open this new MSI and install it. Remember this package already contains the RobotMK plugin and the YAML file. Okay. Let's check this. Here I have the CheckMK agent directory. And within the config directory we see a RobotMK YAML file which contains all information the RobotMK plugin needs to execute the test. And the key suites contains the directory names of Robot suites. The brackets here are a placeholder for command line options we saw before. For example variables and black and white listing of tags and so on and so on. This is enough for now. Now we switch back to CheckMK and do the first inventory. Okay, this is the service list or the checklist of our Windows VM in CheckMK. And now I want to see if the agent already can get data from the RobotMK plugin. I choose edit services to inspect the agent output. And here we can see that the agent contains RobotMK information which CheckMK doesn't already have created a check for this. I choose fix all missing vanished to add this check to the set of checks of this host. Save the changes. And now we have a new check SeleniumTest and if I now reschedule the CheckMK check which is responsible for fetching the agent data I will get a result. For now the RobotTest SeleniumTest runs continuously in the background on the Windows host. I can reschedule this because it's asynchronous check, I see this in the action menu of this service. CheckMK tells me that this is scheduled in an interval of 120 seconds and it's independent from the normal check interval of CheckMK. If I have now configured the basic notification settings in CheckMK I will already get notified if this SeleniumTest fails. Now let's say I want to get alarmed by CheckMK if the second test case SeleniumInputBarDemo runs for a too long time. Then I can use this button here and choose parameters for the service and click RobotFramework. We can see that for RobotFramework there's currently no parameters set, I create a new rule, a specific rule for this host and this service. And I choose runtime thresholds and as we want to limit the runtime of a test I choose test thresholds and click Add. This allows me to define a pattern and each test which matches this pattern will be set this threshold. I want 3 and 1 second for example. And this is an option which I would recommend. Normally RobotMK only shows the runtime in the service output if it was exceeded. If you set this to Yes you will always get the runtime of monitored nodes. I would suggest also to use this pattern for the performance data creation. We also get data to draw a graph out of this. This is an option which allows you to include the last execution date in the first output line also useful and you can choose this here to display error messages coming from sub nodes which then appear on upper nodes. If you want to learn something more about the options you can choose this book icon here. RobotMK comes with a lot of documentation in line also with references to the RobotFramework documentation. Then I click on Save, Apply and then go back to my host and reschedule the check-in-k service. This means fetch new data from the agent. And now the Selenium test turned into warning. Let's open the details and see what happened exactly. The first line contains kind of compressed information so the whole suite failed, it was last executed and it has the warning state. Then what follows is the reason. The reason is the test which we monitor the runtime was warning. Its runtime was this here and it was exceeded. Also we can see the warning state in the test element itself and we also get a graph of exactly this metric. And I am free to define more and more runtime thresholds and performance data patterns. Everything gets collected in this check. For now we have our first robot test integrated into check-in-k with the help of RobotMK. As already explained you can now get alarms for the state of this suite. This means that whatever goes wrong in this suite, one specific contact or contact group will be notified. This is not always what you perhaps want. Let's say that you have two teams and team A is responsible for the first test case and team B for the second. They do not want to receive alarms for the whole suite because the suite can also contain alarms they are not interested in. You on the other hand cannot or want to split these robot tests into two separate robot files. The so-called service discovery level in RobotMK allows you to solve this problem. For this we have to understand how a robot result is constructed. A suite in general is either a collection of test cases. This is why the RobotFrameberg created SeleniumTest which is the file name or it can be a collection of directories containing those robot files. This is here SeleniumTest with an underscore. So from top to bottom we have here the folder suite, the file suite, the test cases and the key birds which then can be nested. You will always by default get one check in K check per robot suite like here but you can change the discovery level for example to two to create checks out from the test cases. Let's try this. We go into the host and service parameters search again for robot and now we choose the last rule the discovery rule, the service discovery rule and create a new one. And this option allows to set a pattern. I leave this on the catch all and to define a level from where we want to create the services or the check in K checks in other terms. I choose to here to generate them from the test cases. I can also specify a regular expression pattern to blacklist some nodes which I do not want to have in the monitoring. The default prefix is always robots and a space. I can also change this if I want. Now I click on save and now if everything works, now I am doing a new inventory. I will see that the Selenium test we first have created which has its origin from the folder suite has disappeared. Instead we got two new checks for the test cases. The button fix all missing vanished solve this for us. So it deletes the old one and creates the two new one when I click on save and activate. I should now have two new pending checks. And now we can see that these two checks have their data from the test cases and look like this. You have seen now how easy it is to integrate robot framework tests into check in K. If you want give it a try and give me feedback and report issues on GitHub. So what's next on the roadmap? I am currently working on a special robot MK keyboard library which allows you then to kind of scrape metrics out of the system under test and to monitor it completely from check in K which currently is only possible for runtimes as we have seen. This will make it possible to monitor application internal values you probably won't have access otherwise. Another cool plan feature is to transfer not only the XML results but also HTML reports from the robot agent to the check in K server and then to link them to the check in K checks. This is an example of such a log and in case of an alarm the user can then directly open this HTML log on the check in K server. It's also planned to also selectively record the desktop and integrate those clips into the report so that it's better understandable what's going on and to debug failed tests. So thanks for your interest in my presentation and I'm looking forward to hearing your questions now. Hey Simon, thank you for your talk, it was pretty nice. Sadly we don't have any question right now but let's start. Do you want to mention anything that you didn't mention in your presentation? I don't think so. I have explained everything to my best and if there are any questions go on. You can just wait a few seconds or minutes. If you got any questions, write them in chat. So there's already applause in this chat. Thank you for your nice talk. I don't know if you mentioned this. I have used the bakery in the check in K enterprise edition but everything can also be done in the raw edition of check in K in the opposite source version. We saw that in your talk and thought about we should ask you this but we have googled something and we saw that everything is available on the opposite. And Robert Fremburg is open source also. I see one question by Martin. What are current projects which you are doing with Robert M.K. One thing I want to change or to expand is the information which are transferred to the check in K server. At the moment this is only the XML output of Robert Fremburg which contains all the runtime information of suites, tests and key words but Robert Fremburg also produces a nice HTML log file and this is only available at the moment on the test client and currently I am reorganizing the structure. I am introducing some metadata and there will also be a field for HTML content and this will be the complete HTML file which gets transferred to the check in K server and the check then splits the content and saves the HTML file into the file system and I want to build some landing page or something so that you can click on the action menu on the check in K service to go to the Robert Fremburg log and all the images and screenshots which were taken during the test are base 64 encoded within the HTML so it is transportable. And another feature I want to implement is some kind of meta monitoring so that there is some check on the client side which can monitor the staleness of spools files because if there is some Robert Fremburg test which didn't end successfully and there is no robot mk section in the agent output this is a situation you never should have and I want to have some meta monitoring which can report how many suites ran and how old are the spools files and how healthy the whole client is. Okay, studio is asking I worked with data.symptatic browser test on the past which also uses Selenium testing does robol mk offers the same possibility. Yes, I have used the robot framework Selenium library in my demo. So Selenium is only a part of the possibilities you can have with robot framework. Another very cool technique for testing websites with robot framework is the browser library. You should name it playwright library because it's based on the playwright technology, which is a kind of successor of puppeteer which I think is more famous. And I think this is more powerful and even more faster than the Selenium with Selenium you have to use a web driver which is kind of translator between robot framework and the browser and with playwright you are controlling the browser if you were the browser itself and you can do rest calls and monitor the network traffic starts different browser instances with certain permissions and so on it's very powerful. Yeah. Right, thank you. There's one more question by Alex. Can you give some more real world examples that you monitored with robot mk already? For one client I have written a test, the goal was to test a very old document management system based on Windows and for this I didn't use a library like Selenium or playwright because it's a native Windows application and there are many more libraries for robot framework to accomplish this and one very powerful library is the image horizon library which is based on a pure Python library and it works like this you are taking screenshots of a button and within the robot test you write wait for a pixel region which looks like this screenshot and if you have found it within 10 seconds do a click on it and this is a test which runs very long it's very complicated and yeah this is very stable. Another test I've written is for government IT system which it's a map based web application which shows up the weather forecast and snowfalls, water levels of rivers and so on for the whole Switzerland and this application shows the metrics of these measurement stations and the test is based on playwright and clicks through all these measurement stations and checks for the valid metrics. Okay, thank you Simon I think for your talk. I think you are available for questions in the next couple of days or so on. I am. Yeah and I want to thank you the whole community that watched our deaf room today. Thank you guys have a nice day and stay healthy. Yes, also from my side thank you all for visiting our deaf room and yes thank you to the whole deaf room team I think Stefan had a lot of effort in making this whole thing possible. So, yes of course. So again thank you everybody for being here and well maybe we can see each other physically on maybe another conference. Another conference next year. Would you agree if we see us? Yes, of course. Thank you guys. See you. Bye. Thank you. Bye.
 Hi, my name is Simon Meckle.  I'm very happy to be a speaker at this FOSSTEM conference this year.  The topic of my presentation is Robot MK, how you can integrate the results from Robot  Framework into the monitoring system CheckMK.  Let's get started.  I'm coming from the south of Germany, close to Munich, and I've been working with open  source monitoring systems like Nagios for 20 years now and the last 10 years I've been  working as a consultant.  My focus topics are data center automation with Ansible, monitoring, I'm specialized  on CheckMK, and test automation with focus on Robot Framework.  And last but not least, I'm generally interested in Python development.  So first let's talk about IT infrastructure monitoring.  In my opinion, the best solution for this is CheckMK.  CheckMK was written 2008 by Matthias Kettner as a kind of Nagios add-on, first of all to  only solve problems with large Nagios installations and its configuration limitations.  But it's gradually evolved and became a fully integrated monitoring system, which aims to  be an all-in-one solution for monitoring everything while still being user-friendly and very flexible.  It comes with more than 2,800 curated check plug-ins and is available in two editions.  This is the raw edition, open source, and the enterprise edition with some additional features  and enterprise support.  The most important fact you should know about CheckMK is that monitoring systems like Nagios,  Neiman, CheckMK, Isinga, etc.  All together are able to monitor up to OC layer 7.  And strictly speaking, they all start on OC layer 3 by sending ICMP packets to check if  a host is up.  But didn't we forget the applications?  How do we monitor applications?  How do we monitor if users can really work with an application and how performant?  The answer is by default there is no such monitoring available.  And don't get me wrong, this is not a criticism of any of these monitoring systems.  It's simply too complicated that these systems could provide a kind of generic way to monitor  applications, to monitor what the user really uses and sees.  Layer 7 is called the application layer, but this is a kind of false friend, I think.  It's only the interface layer to the applications.  Layer 7 only contains the protocol, for example HTTP, to be able to contact a web server.  The content of HTTP packets is the full responsibility of the application itself.  And this is what we should check.  And this is my second disclaimer.  I don't say that we do not need checks in layers 3 to 7.  It's absolutely necessary to have them, but they are not the whole truth.  What I want to say is, in the end, it's all based on a hypothesis.  I found a nice paragraph in this book here, weniger schlecht programmieren, less bad coding,  which I think perfectly fits.  Observations want to become hypotheses.  It's not easy to only record what you can see.  An untrained observer tends to see a man running after the bus, but in reality, the  man and the bus are moving in the same direction accidentally.  It's nearly impossible to observe something freed from hypothesis.  This means we are monitoring through the best of our ability the application preconditions  and hope that the applications are happy with that.  And if the application is OK, we tend to believe that everything we already monitor  is everything the application needs to work.  We are displacing the question if there could be more reasons to break the application,  and unfortunately, there are many more.  A great way to test applications automatically is the open source tool Robot Framework.  Robot Framework is a generic test automation framework written 2005 by Pekka Klärk at Nokia  Siemens Networks.  And since 2008, open source.  It's completely written in Python.  And there are two features that set Robot Framework apart from other testing tools.  It's the keyword-driven approach, which I will explain in the next slide, and the library  concept.  Robot Framework itself doesn't test anything.  You import libraries with special keywords for web testing, image recognition, REST  calls, and so on and so on.  And this makes Robot Framework for me to a Swiss Army knife.  In my opinion, one of the best reasons to take Robot Framework as a test tool is that  it's open source and independent from any software vendor.  The decision for Robot Framework is not a decision for a company.  It's the decision for the lingua franca of test automation.  Robot Framework will always be the property of the community.  So what are keywords now?  Simply explain keywords abstract source code.  Source codes can be written in Java or in Python, mainly in Python.  And they can be written and used like functions.  They can take parameters and return values.  They are case insensitive.  They allow spaces, which is not a big feature, but makes the code much more readable.  And a question I often get asked is, why should I use Robot Framework?  There's Selenium for Python.  Why should I use Robot Framework?  It's absolutely OK.  You can see here a small snippet of Python code using Selenium.  And for sure, this code would need some better exception handling and methods to integrate  the data into other systems.  And if you are writing hundreds of those tests, for example, you will of course try to separate  the common code from the test code.  And in the end, you will have some kind of a self-written framework.  And this is exactly what Robot Framework already does for you.  In my opinion, there is no reason not to use Robot Framework.  The same test as you can see here in Python can also be written in Robot Framework using  the Selenium library.  It's a much more cleaner code.  The yellow words here are the keywords, which I would say is also readable by anyone who  is not a real programmer.  But Robot Framework is not a programming language itself.  It's more like an additional abstraction layer for existing Python modules.  So in theory now, we know CheckMK and Robot Framework.  Let's talk about the integration of Robot Framework into CheckMK.  And this can be done with my project RobotMK.  RobotMK comes as a MKP extension, MK package.  It's a collection of files in the end for CheckMK to integrate the results of Robot Framework.  I've started this project in November 2019 and presented a proof of concept at the RobotCon  conference in January 2020 in Helsinki.  I got a lot of good feedback, even though hardly anyone knew Nagyus or CheckMK.  And I continued my work.  And thanks to some customers of mine, I always could further develop and improve the module.  This slide normally takes up to five minutes to explain.  Time is rare.  For now, it's only too important to know for you that RobotMK consists of mainly two  components.  Like any other CheckMK agent plug-in, the RobotMK plug-in gets triggered by the agent  on the monitored host where the Robot Framework tests are.  Other plug-ins collect system data, memory consumption, or something.  And the RobotMK plug-in executes Robot Framework tests and transfers the XML result back to  the CheckMK server.  On the CheckMK server, the check script is responsible then to evaluate that Robot XML.  By default, a Robot Framework test already has a state.  This is if the whole test was successful or not.  And the RobotMK rules, these are normal CheckMK rules, enable you to enrich the result by  adding more monitoring tests.  For example, monitoring the run times of sweet tests and keywords.  In the following demo, I will show you a start from scratch.  We will integrate a small Robot Framework test on a Windows 10 machine and integrate  this result into CheckMK.  And I will show you how to install RobotMK and how to set some settings and show you  features of RobotMK.  Let's start.  Okay, here we are on a Windows 10 VM.  The first thing I want to show you is the Robot Framework test itself.  It's a very simple test.  As you can see here, it consists of two test cases.  Each of those is connected to seleniumeasy.com and executing a dummy test.  I can execute this test by hand so that you see what it's doing.  It's starting Chrome.  Enter some text and expect the page to display the entered text in a separate field.  So that's all.  The first thing I have to do now is to copy the directory of this RobotTest.  This is seleniume test into the agent directory of the CheckMK agent.  The RobotMK plugin expects the Robot Framework test case in a certain directory.  This is called Robot.  And within the Robot directory, I'm pasting this directory.  We only need this here.  So this is the first step.  With RobotMK, you can control where RobotTest cases should be executed and how.  Let me show you how this works.  This is a fresh installation of CheckMK with one monitored host.  This is my Windows 10 VM where the seleniume test should be executed by the CheckMK agent.  Now I open the host and service parameters and search for the rule Robot.  This brings up three rules.  I'm taking the agent plugin rule.  This rule controls how agent installation packages should look like individually per  host.  This individually means the MSI package I'm now creating for this Windows host in a few  minutes will contain everything.  The RobotMK plugin and a YAML definition file which is read by the plugin and which tells  the plugin which Robot suites should be executed.  I create this rule here.  The first thing I do is choose explicit host.  I only want to apply this rule on my Windows VM.  Here I can choose to deploy the plugin.  Of course there are two modes of execution.  This is the spoolier mode which means the plugin gets executed externally for example  by a task scheduler or the asynchronous mode which is default.  This means the CheckMK agent executes the plugin but in another interval than the normal  CheckMK interval which is one minute per default.  I'm choosing this interval and I'm setting let's say two.  Robot suites directory is an option where I can override the default directory.  We do not need this.  Robot framework test suites allows me to add a list of test suites I want to execute.  This takes the directory names of suites I want to execute.  Remember this is the name of the directory we already have copied into the Robot directory.  All the options you can see here are translated at runtime to Robot framework command line  options.  So you can choose another name for the suite.  You can black and white list by tags.  You can load variables, a variable file into the Robot test.  And the last option is a special CheckMK option.  This allows you to assign this result to another host defined in CheckMK.  We don't need this at the moment.  This is everything we set up here.  The last two options define how the agent output should be encoded.  If there is too much content you can also choose ZLIP to compress the agent output.  And this is a setting which controls how many days you want to keep the log files on this  host.  I click on save.  And now you can see that CheckMK offers me to bake a new agent version.  I choose bake agents.  And now watch this list.  Currently we only have one version of the agent.  This is the generic version which is applicable to every host.  And now CheckMK will bake a special version for my Windows VM.  This is what we now have.  This is the MSI package with a new hash.  And here I see the settings which come from the RobotMK rule.  The next step is now to install this agent on the Windows VM.  Now I'm back on the Windows machine.  I'm logging into CheckMK.  And in the VATO configuration I choose monitoring agents and download the new MSI package.  You can also set up the CheckMK agent to do automatic unattended updates.  I didn't do this here for demonstration purposes.  So let's open this new MSI and install it.  Remember this package already contains the RobotMK plugin and the YAML file.  Okay.  Let's check this.  Here I have the CheckMK agent directory.  And within the config directory we see a RobotMK YAML file which contains all information  the RobotMK plugin needs to execute the test.  And the key suites contains the directory names of Robot suites.  The brackets here are a placeholder for command line options we saw before.  For example variables and black and white listing of tags and so on and so on.  This is enough for now.  Now we switch back to CheckMK and do the first inventory.  Okay, this is the service list or the checklist of our Windows VM in CheckMK.  And now I want to see if the agent already can get data from the RobotMK plugin.  I choose edit services to inspect the agent output.  And here we can see that the agent contains RobotMK information which CheckMK doesn't  already have created a check for this.  I choose fix all missing vanished to add this check to the set of checks of this host.  Save the changes.  And now we have a new check SeleniumTest and if I now reschedule the CheckMK check which  is responsible for fetching the agent data I will get a result.  For now the RobotTest SeleniumTest runs continuously in the background on the Windows host.  I can reschedule this because it's asynchronous check, I see this in the action menu of this  service.  CheckMK tells me that this is scheduled in an interval of 120 seconds and it's independent  from the normal check interval of CheckMK.  If I have now configured the basic notification settings in CheckMK I will already get notified  if this SeleniumTest fails.  Now let's say I want to get alarmed by CheckMK if the second test case SeleniumInputBarDemo  runs for a too long time.  Then I can use this button here and choose parameters for the service and click RobotFramework.  We can see that for RobotFramework there's currently no parameters set, I create a new  rule, a specific rule for this host and this service.  And I choose runtime thresholds and as we want to limit the runtime of a test I choose  test thresholds and click Add.  This allows me to define a pattern and each test which matches this pattern will be set  this threshold.  I want 3 and 1 second for example.  And this is an option which I would recommend.  Normally RobotMK only shows the runtime in the service output if it was exceeded.  If you set this to Yes you will always get the runtime of monitored nodes.  I would suggest also to use this pattern for the performance data creation.  We also get data to draw a graph out of this.  This is an option which allows you to include the last execution date in the first output  line also useful and you can choose this here to display error messages coming from sub  nodes which then appear on upper nodes.  If you want to learn something more about the options you can choose this book icon  here.  RobotMK comes with a lot of documentation in line also with references to the RobotFramework  documentation.  Then I click on Save, Apply and then go back to my host and reschedule the check-in-k  service.  This means fetch new data from the agent.  And now the Selenium test turned into warning.  Let's open the details and see what happened exactly.  The first line contains kind of compressed information so the whole suite failed, it  was last executed and it has the warning state.  Then what follows is the reason.  The reason is the test which we monitor the runtime was warning.  Its runtime was this here and it was exceeded.  Also we can see the warning state in the test element itself and we also get a graph of  exactly this metric.  And I am free to define more and more runtime thresholds and performance data patterns.  Everything gets collected in this check.  For now we have our first robot test integrated into check-in-k with the help of RobotMK.  As already explained you can now get alarms for the state of this suite.  This means that whatever goes wrong in this suite, one specific contact or contact group  will be notified.  This is not always what you perhaps want.  Let's say that you have two teams and team A is responsible for the first test case and  team B for the second.  They do not want to receive alarms for the whole suite because the suite can also contain  alarms they are not interested in.  You on the other hand cannot or want to split these robot tests into two separate robot files.  The so-called service discovery level in RobotMK allows you to solve this problem.  For this we have to understand how a robot result is constructed.  A suite in general is either a collection of test cases.  This is why the RobotFrameberg created SeleniumTest which is the file name or it can be a collection  of directories containing those robot files.  This is here SeleniumTest with an underscore.  So from top to bottom we have here the folder suite, the file suite, the test cases and  the key birds which then can be nested.  You will always by default get one check in K check per robot suite like here but you  can change the discovery level for example to two to create checks out from the test  cases.  Let's try this.  We go into the host and service parameters search again for robot and now we choose the  last rule the discovery rule, the service discovery rule and create a new one.  And this option allows to set a pattern.  I leave this on the catch all and to define a level from where we want to create the services  or the check in K checks in other terms.  I choose to here to generate them from the test cases.  I can also specify a regular expression pattern to blacklist some nodes which I do not want  to have in the monitoring.  The default prefix is always robots and a space.  I can also change this if I want.  Now I click on save and now if everything works, now I am doing a new inventory.  I will see that the Selenium test we first have created which has its origin from the  folder suite has disappeared.  Instead we got two new checks for the test cases.  The button fix all missing vanished solve this for us.  So it deletes the old one and creates the two new one when I click on save and activate.  I should now have two new pending checks.  And now we can see that these two checks have their data from the test cases and look like  this.  You have seen now how easy it is to integrate robot framework tests into check in K.  If you want give it a try and give me feedback and report issues on GitHub.  So what's next on the roadmap?  I am currently working on a special robot MK keyboard library which allows you then to  kind of scrape metrics out of the system under test and to monitor it completely from check  in K which currently is only possible for runtimes as we have seen.  This will make it possible to monitor application internal values you probably won't have access  otherwise.  Another cool plan feature is to transfer not only the XML results but also HTML reports  from the robot agent to the check in K server and then to link them to the check in K checks.  This is an example of such a log and in case of an alarm the user can then directly open  this HTML log on the check in K server.  It's also planned to also selectively record the desktop and integrate those clips into  the report so that it's better understandable what's going on and to debug failed tests.  So thanks for your interest in my presentation and I'm looking forward to hearing your questions  now.  Hey Simon, thank you for your talk, it was pretty nice.  Sadly we don't have any question right now but let's start.  Do you want to mention anything that you didn't mention in your presentation?  I don't think so.  I have explained everything to my best and if there are any questions go on.  You can just wait a few seconds or minutes.  If you got any questions, write them in chat.  So there's already applause in this chat.  Thank you for your nice talk.  I don't know if you mentioned this.  I have used the bakery in the check in K enterprise edition but everything can also be done in  the raw edition of check in K in the opposite source version.  We saw that in your talk and thought about we should ask you this but we have googled  something and we saw that everything is available on the opposite.  And Robert Fremburg is open source also.  I see one question by Martin.  What are current projects which you are doing with Robert M.K.  One thing I want to change or to expand is the information which are transferred to the  check in K server.  At the moment this is only the XML output of Robert Fremburg which contains all the  runtime information of suites, tests and key words but Robert Fremburg also produces a  nice HTML log file and this is only available at the moment on the test client and currently  I am reorganizing the structure.  I am introducing some metadata and there will also be a field for HTML content and this  will be the complete HTML file which gets transferred to the check in K server and the  check then splits the content and saves the HTML file into the file system and I want  to build some landing page or something so that you can click on the action menu on the  check in K service to go to the Robert Fremburg log and all the images and screenshots which  were taken during the test are base 64 encoded within the HTML so it is transportable.  And another feature I want to implement is some kind of meta monitoring so that there  is some check on the client side which can monitor the staleness of spools files because  if there is some Robert Fremburg test which didn't end successfully and there is no robot  mk section in the agent output this is a situation you never should have and I want  to have some meta monitoring which can report how many suites ran and how old are the spools  files and how healthy the whole client is.  Okay, studio is asking I worked with data.symptatic browser test on the past which also uses Selenium  testing does robol mk offers the same possibility.  Yes, I have used the robot framework Selenium library in my demo.  So Selenium is only a part of the possibilities you can have with robot framework.  Another very cool technique for testing websites with robot framework is the browser library.  You should name it playwright library because it's based on the playwright technology, which  is a kind of successor of puppeteer which I think is more famous.  And I think this is more powerful and even more faster than the Selenium with Selenium  you have to use a web driver which is kind of translator between robot framework and  the browser and with playwright you are controlling the browser if you were the browser itself  and you can do rest calls and monitor the network traffic starts different browser instances  with certain permissions and so on it's very powerful.  Yeah.  Right, thank you.  There's one more question by Alex.  Can you give some more real world examples that you monitored with robot mk already?  For one client I have written a test, the goal was to test a very old document management  system based on Windows and for this I didn't use a library like Selenium or playwright  because it's a native Windows application and there are many more libraries for robot  framework to accomplish this and one very powerful library is the image horizon library  which is based on a pure Python library and it works like this you are taking screenshots  of a button and within the robot test you write wait for a pixel region which looks  like this screenshot and if you have found it within 10 seconds do a click on it and  this is a test which runs very long it's very complicated and yeah this is very stable.  Another test I've written is for government IT system which it's a map based web application  which shows up the weather forecast and snowfalls, water levels of rivers and so on for the whole  Switzerland and this application shows the metrics of these measurement stations and  the test is based on playwright and clicks through all these measurement stations and  checks for the valid metrics.  Okay, thank you Simon I think for your talk.  I think you are available for questions in the next couple of days or so on.  I am.  Yeah and I want to thank you the whole community that watched our deaf room today.  Thank you guys have a nice day and stay healthy.  Yes, also from my side thank you all for visiting our deaf room and yes thank you to the whole  deaf room team I think Stefan had a lot of effort in making this whole thing possible.  So, yes of course.  So again thank you everybody for being here and well maybe we can see each other physically  on maybe another conference.  Another conference next year.  Would you agree if we see us?  Yes, of course.  Thank you guys.  See you.  Bye.  Thank you.  Bye.
nemorobotmk_0_15875.png nemorobotmk_15876_16703.png nemorobotmk_16704_18143.png nemorobotmk_18144_18683.png nemorobotmk_18684_23615.png nemorobotmk_23616_24035.png nemorobotmk_24036_24851.png nemorobotmk_24852_24899.png nemorobotmk_24900_25331.png nemorobotmk_25332_28523.png nemorobotmk_28524_29075.png nemorobotmk_29076_30407.png nemorobotmk_30408_31343.png nemorobotmk_31344_33383.png nemorobotmk_33384_33695.png nemorobotmk_33696_35771.png nemorobotmk_35772_36671.png nemorobotmk_36672_38927.png nemorobotmk_38928_39143.png nemorobotmk_39144_40403.png nemorobotmk_40404_40727.png nemorobotmk_40728_40775.png nemorobotmk_40776_41363.png nemorobotmk_41364_41507.png nemorobotmk_41508_41555.png nemorobotmk_41556_43391.png nemorobotmk_43392_43943.png nemorobotmk_43944_44159.png nemorobotmk_44160_45023.png nemorobotmk_45024_47159.png nemorobotmk_47160_47207.png nemorobotmk_47208_47375.png nemorobotmk_47376_49547.png nemorobotmk_49548_50615.png nemorobotmk_50616_57131.png nemorobotmk_57132_57298.png