Abusing GitLab CI to Test Kernel Patches
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kondrashov, Nikolai
Hofmann, Michael
See how Red Hat’s CKI project uses GitLab CI to test kernel patches as soon as they're posted to maillists. Red Hat's CKI project uses GitLab CI to organize and track its pipelines, lint, patch, and build Linux kernels, and oversee testing. It also uses a number of supporting systems to discover kernel patches and commits, maintain hardware inventory, provision hardware and VMs, run tests, and finally record and report results. See which tricks the project is pulling to tie all these parts together, and test patches posted to several maillists, commits to 15+ git repos, builds done by other build systems, as well as weird things like stable kernel patch queue, and parts of its own software stack. Making such extensive use of a CI system inevitably uncovers its limitations, and a list of these will also be presented along with some possible solutions.
 So we are going to talk about this thing. And us is me, Nikolai Kondrashov, and Michael is sitting here. He'll join me later. Neither of us does actually look like that anymore. So we are coming from Red Hat CKI team or CKI project, and we are a distributed team which are doing the kernel CI at Red Hat. So why do we need kernel CI? Well, as most of you probably know, we do releases of distributions, and each release has different kernel versions. There's a lot of kernel versions. And moreover, we are one of the major contributors to the kernel and certainly the biggest one among distributions. So this shows the comparison of unique email addresses from SUSE, Red Hat, and Canonical contributed to the kernel. And this is commits. Each of those is per year, and the Red Hat is the blue one. So we somehow got to make it consistent and reliable. And if you look at this, this is a big cue of how code goes through the pipeline towards Red Hat. And until not so long ago, our tests were only there at the end. So the developers would throw the commits over to the builds over to the QA, and they would test it and then come back, and they would retest it. That takes a long time. And if you consider how long it takes for the whole code pipeline to execute one patch to digest the patch through that and get to a release, it's a long way. So what we want to do is we want to do this. And we want to do it fast and provide as much as possible feedback so that ideally the bugs are caught before even maintainers see them at the moment that the developers submit those. And that is hard because there is just so much email. I mean, look at this. So we somehow are supposed to, I'm not having anything against this message that just shows the complexity of these things. So somehow we are supposed to put our web hooks in here. This is patch number 62 out of 114. And there is this amount of discussion going on. So somehow we are supposed to test those and provide feedback to developers. So this has been an ongoing discussion recently in the kernel circles. And for example, the last Linux plumbers, there was one of those presentations that are happening recently by Dmitry Bujkov, who did a very good take on those issues. And I recommend you watch it if you're interested in the kernel development process. He makes good points. So what we've built is something like this. This is simplified. I actually lost a slightly more complex slide on the plane here because of how Slice.com works. But no mind, this is very simple. And we'll go around. So normally, if you wanted to check just the changes in the Git repo being committed there, that's kind of easy. So we have a bunch of repos with track. And we check if there are new commits and we test those. That's fairly trivial. We actually have, inside of that, we have a bunch of Git repos for different releases like RL7, RL6, RL5, RL8. And we also track upstream repos, mainly stable at this moment. But we track also a few others. And we test those commits, of course. And that's relatively easy. We just pull the repo and run our test about which I'll tell you a little later. Then there's the interesting part which I started with. Turns out it's not that hard. So it's hard. It's hard. Don't get me wrong. But you can do it easier. So there's a typical mail list, like Linux USB mail list. There's a message from a series. It looks like this. Turns out there is a project called Patchwork, probably many of you know, which is used by main trainers. Well, most of all, they use it to track the patches as they've been processed, reviewed, and tested. And they check which patches were merged and which not. So it looks like this. If you go to patchwork.kernel.org, there is a bunch of projects. And those projects can be mapped to a particular mail list or even a particular tag being used, like tagging the subject being used in the mail list. Well, at least the thread head is done this way. I'm not sure if that's upstream actually. So if you go to the same Linux USB mail list here, you can see there is again those patches. But this time they're organized into series. If you click on one of those links, you get to the particular patch series. There are two patches in there. We can go to specific patch and see what's going on, what's the patch, and we can download the inbox there on the right or the whole series. And the main thing that concerns us about this is that patchwork has REST API. So we can go through those projects. We can extract the patch series, the patches, and everything. And we can track when they are peering. This is, of course, sounds very simple, but the devil is in the details where you have to kind of expect that not all messages come through at the same time. When you go check and the series might not be complete, there are bugs, people sending all kinds of messages in there. Sometimes they're not picked up and things like that. Patch series can have cover letter, don't have cover letter and things like that. But you can make it work. So the typical patchwork trigger would be tied to a particular patchwork instance and a particular project there and associated with the corresponding Git repo. Further on, we also have triggers from our package builds in Koji and the developers builds in copper for Fedora and internally for L, although it's called a little differently. Pardon. Let's go back. So Fedora build system looks like this. There are a bunch of packages. They are being built, prepared for releases and reviewed. So we can look for kernel. There's our kernel. Let's take this one. And here is information on the build of the kernel, like specific revision and everything and all the packages that were built for all the architectures. And copper looks like this. Oh, wonderfully. Finally connected. So copper is more for developers. You can have your package build and put into our RPM repo and picked up by your users or by other developers. We track those as well. So you can look for the kernel there and find one of those and go in there and into builds and see there's been a build and here's the packages. But we don't talk through the web UI, of course. There's the packages. We talk through the, we listen to the Fedora message bus, which is used both by Codege and copper and internally at Red Hat. There is a message bus as well. We just listen to the message. There is a log from our trigger and checks like, okay, there's build completed. There's a message coming through the bus. We're not interested in this one, neither in this one. But here is the kernel. We pick it up and we trigger our pipeline. So we also have to test our own CI. So we have two special kinds of triggers for GitHub and GitLab for testing contributions to our CI repo, so which we have many. And as you can see, we have some on GitHub and some on GitLab because of historically. So it looks simply like this. You submit a PR or an MR and the bot comes and says like, hey, I'm a bot. Send me a message and I'll test this for you. So the developer says like a test please bot. And bot says testing and then it triggers the pipelines through for the various, for the various repos that we have. Developer can go have lunch or two and then finishes and the bot says like passed or failed. Well, as it happens, here's an example of my food request. So the bot comes in and tells us like, here I am. This is what you can do. And it's a little different for GitLab and for various repos and can say like, oh, add this keyword or that keyword and we'll test this and that and things like that. And yeah, I am asking the bot to test. So the bot says, yeah, I'm going. Then it's supposed to resolve or failed, of course. So that's how we do testing for our CI because yeah, many repos and because our CI is in a separate pipeline and we have like, we can certainly have. Two GitLab repos actually handling this bot this not repos GitLab instances. That's a complication. A fun one. Yeah. So further on that's and these are two major parts of our pipeline. Of course, the test database and the tool which lets us pick which tests to run and organizes everything into trees and other dependencies like where we can put we can run with this build with that build on this architecture and that architecture and et cetera, whether developers want to test this or that. So basically the data flow is very simple. They have Kpdb, which is a repo with the test information, which is currently private because there is like all kinds of stuff for rail internal ones. And we are still intending to open it up, but it's difficult to separate the upstream tests from downstream tests in a complicated data structure and somehow merge it together. So we had the database, which is basically YAML and we have a tool which takes it and a bunch of parameters and then speeds out XML for bigger system, which actually runs our tests. And this system is installed internally, but it's open source, but you will have a very hard time actually trying to install it, which we are working on right now very hard and hopefully will be able to let you guys enjoy it. But it's all open source. It's out there, goes into documentation. It's just nobody succeeded installing it by themselves. So the databases can contain information about architectures again, of which there are five right now. Kind of host types and this describe what do we want the host to have, like do we want to have this much RAM or this many CPUs or this much storage or even a particular PCI card or a network card. And we organize them into host types because that's easier. They have three, so obviously particular repos or types of repos we want to test and those effect which tests will run where like for example one test can run on REL7 but not on REL5 and some on upstream and some tests are still internal. Not many though. Most of them are actually out there. Components which describe what things the build contains, like upstream only contains kernel image, but internal builds, they are built using RPMs and there could be debugging for headers, the internal kernel headers and things like that or tools that some tests need like for example some tests need debugging for things like that or some tests don't run on debug build. And then we organize tests into sets of course like for network tests, for file system memory, etc., etc., virtual machines and of course the description of the tests with themselves of which there are quite a big number, assume to be 100. And these range from simple tests like just a shell script which just restarts the kernel test something and it's done to very big ones like LTP, USEX and the top ones are listed there I guess but those are not all which can contain like thousands of tests. So an example of tests with data like a description where it is in the report is actually quite outdated. Anyway, the essence is there. Where it is located and for example this one is in our test repository on GitHub where most of our tests are, which host type it runs on, additional information of like I want like this very specific host for this like and it could be down to a specific host name on our in our bigger system like I want to run it like exactly here on this machine because there's only this hardware there. This description could look like this actually this says don't run on these ARM systems because they don't work. So information on maintainers and what's not well this is a discussion for upstream that's not the depth for now so these are test maintainers who look after the test and check that it's working and it's failing and they actually receive copies of failure reports and they're supposed to take a look at as soon as something happens and tell the developer okay sorry that's my bag that's a failure or say like this is your problem and they are responsible for those tests which is going to upstream important because upstream developers they don't see that much into our machines and things like that so they have a hard time figuring out what actually happened which we are working on. So there's a can there's the conditions for the test to run on like the sets which it belongs to and this is also outdated my gosh. This is also an interesting part that we specify which source files the particular test covers more or less so that we can avoid running it when there is a patch that doesn't touch those files and that's why we can kind of contain the runtime at least a little bit make it shorter when we don't need it. And this allows us to kind of describe when to run them for which code. Architectures the test will run on and which trees it belongs to but there are no components here because this is old and there could be like multiple cases of this RIP I wanted to run it with this file system or the dot file system for example a file system test or additional parameters and in walking the K-pad to normally people don't invoke it by hand but it runs in the pipeline so you can say like okay generate me the XML for this run for this kernel turbo for the upstream tree AR64 with this patch and highlight the output and it would look something like this and it goes on and on and on and I'm not going to bore you with those details. So this is the input to beaker and saying how to run it and where to run it and in which order. So going to beaker it's a big system which maintains inventory for the hardware including down to the components. Let's you match that hardware has access control like particular groups have access to this hardware those to this hardware and for example some NDA hardware could be there and protected it also does the provisioning and puts up the machines, installs the operating system from scratch using an account normally because we don't support running from images because that's hard to do and we are distribution so we have to test the whole distribution from install. So it installs everything, it talks to the test harness, extracts test results and things like that and looks after the machine so that if it does lock up it then it releases the machine and erases everything. So it could be like the system inventory we can find of those machines like these are not very useful right now this is a titanium we still have those. You can go into the machine and take a look at the details like this is just one tab about the host information there's the CPU info storage peripherals things like that and this is an example of some of our jobs running for stable repository of the Linux kernel. This one job is just for one architecture and it has four hosts and here's an example of one hosting executing those tests this is a bigger UI this is a bunch of tests there. Further on now we are approaching the user visible stuff so we have a reporter which watches over the pipelines and checks which stage they are on which job they are approaching and sends the email reports to developers or whoever is interested and sometimes it can send an early email saying like okay have we started this test and like watch out or where we did the testing or something failed in the pipeline. So there's an example of a successful report that was sent to a stable mail list here's the it starts with the saying like we took this repo we took this commit and there's the summary everything went fine and we were actually compiled and use those comments and then we run them on these hosts like this architecture is ARC64, cross-host second host, PPC64, two hosts, X86, X86 got more hosts, four hosts and we also have a notion of waved tests, a test which you mark in that KPEB DB and saying this test is waved which means run this test as normal maybe at the end of the run but ignore the result and don't take it into account when giving a verdict whether it failed or not we say we ignore it and we use this to test the test which were just introduced into the system or were being fixed so that we can track like how are they performing like they doing okay and the test maintainer can look after it until it stabilizes then remove the waved status and this is done manually because tests are different you have to look after them. And that's an example of report that was sent upstream our internal reports are a little more elaborate you get to see actually links to the bigger results and explore the logs and everything but those tests actually have yeah artifacts there is a blue link there this contain the binaries, config files and logs things like that. So and then finally we have the data warehouse it's a system which uses PostgreSQL and collects all the information about our runs also similar to reporter and this kind of application of the effort at this moment but we are working on that. So it watches over all the jobs and collects the information like how it went and what's the status and much tests run and there is a web UI which looks something like this and provides statistics how much we failed how much we succeeded as pipelines and various statuses this has been pulled like from GitLab using the GitLab API and there's a particular pipeline and listing all the tests and all the hosts and you can go and see the results in Beaker how it went. And we maintain the test statistics how tests were failing or passing for exactly for the purposes of deciding like when to waive the test if it's been doing bad and then send it back to the maintainer and say like okay deal with it or we can actually take it out of wave state if it's doing okay. And same for hosts like if some host is behaving in Beaker and that's a problem because there are just so many hosts that they break and you have to watch out and the host maintainer like whoever maintains that host they don't have time to look after it so we look at those and we say okay this host should be excluded from the runs and we add like don't run on that host. So finally the title of this talk no not yet this actually concerns Guillaume's talk so there is this thing you probably seen it at the last slide. It's the kernelci.org and there are lots of tests and they were recently approved as accepted as a links foundation project to advance the state of kernelci and we joined that effort and right now we are working on a database and the system used to aggregate testing information from various CI systems so that at the ultimate goal so that there is a single place to go and check kernelci results from whoever runs those tests and so that the developers get on a single email with those results and not just five emails from everyone. Right now this is mostly the kernelci folks and us but others are joining hopefully soon we start aggregating more data but we already have a tool. You can take a look like how this looks I don't know if Guillaume showed this but this is an example of how test reports like top level could look there. So we took a Google BigQuery system for storing those results so they are more readily publicly available and so that people can go and explore the data and see how kernel is doing and do research if they need to. So this is our repo with the code for that and it looks something like this when it's pushing and it's the data and we are working on a dashboard to show this off and provide the developers. This is very resimentary at the moment. So finally the interesting part I took a little while. So we store our CI pipeline inside YAML but we store it in separate repos because of the way we trigger those. So to trigger GitLab we are actually doing commits to the repo and I'll show that in a moment. Basically these are two repos and one the repo on the left is only including pieces from the other repo. And this lets us to let the triggers do commits with the information about what we want to test inside that repo. And we need two repos so that these commits don't interfere with the development commits we have. So because there is like every time you want to test something there is a new commit and that's an empty commit and it doesn't have any data in it so we use it just to identify the particular pipeline and GitLab view. So for example the baseline trigger like the Git repo trigger comes in and does and sees that there are two changes in the like in one repo and another and does commits to separate branches in that repo. This trigger actually is retired now but it was quite kind of interesting. So it also does and checks and finds something and does the commit. Because the trigger that finds patches and does commit in that same branch and finally the GitHub board comes and finds like okay there is a new merge request and I put it in all the branches that we're interested in testing. And it might look like this for example the stable branch has those commits all with the pipelines running and this branch has its own commits here. And the commits can look like this. So there is data there but it's not for GitLab consumption only for us as debugging like this says. All the variables that we put in there all the descriptions like what we are triggering on things like that this one is huge and this one is big as well it's abbreviated. So we use a lot of GitLab extends property which lets us separate the general pipeline into the pipeline into the menu of jobs and stages and into three specific information like our pipeline specific information. This is our shortest pipeline and it says okay pick those four stages from the pipeline and we have ten maybe or more and this says okay pick this prepare step where we download all the stuff for the execution all the dependencies and this says like this is the prepare and this is stage prepare and this one we say okay again pick this create repo x8664 and extend it and this is one of those same okay take this repo take this template of the job and a bunch of variables and conditions and create a particular job for this specific pipeline here. And this time we are using the merge keys to merge those. So we use an extend here extends here because this is a separate YAML file so we cannot use merge keys. And we use merge keys here it's the same one big as YAML file there and this is what be create repo creating a RPM repository with build results for testing which are then installed in biker. Next stage is composed a little differently so we have a huge script which is split into a few YAML objects and finally the last stage looks similar to that and so on so we have pipelines which are much longer than that and more involved. So why we took GitLab well we started out with Jenkins so we had a Python script controlling Jenkins which had a Joe Britton Groovy which controlled another Python script which checked out the kernel and build it and then fed it off to biker. So that was not very reliable hard to debug, hard to understand, hard to maintain as a contrast with GitLab we had relatively straightforward system you could keep everything in the Git repo and keep changing it faster and it's more reliable than Jenkins for us. And I hope Michael is able to say something, we have ten minutes left. Now you will hear me complaining about GitLab but so I don't know how many people used GitLab here and how many people have used GitHub. So it's very familiar system so it's nicely documented, it has a huge API surface, people are familiar with it so if I say GitLab people actually know what I'm talking about, if I mention some other CI technology then people look at me like this, if I mention Jenkins people just go away and say like no you don't want to be on your team, hopefully there's nobody from Jenkins here, sorry. But then we are actually testing kernel so that's quite similar to other software to test in some aspects but for other aspects especially like testing there are some interesting issues that you will see there. The one is that GitLab, most of these general CI systems don't have any concept of a failed pipeline because of infrastructure issues so if you look into what distributions do for gating most of those actually have a test fail thing, maintain or fix it and then they have something like oh our infrastructure failed or our test system and then it's actually for somebody else to fix. Now most of you might know that kernel maintainers don't react too well if you email them without any good reasons or if you email the Linux kernel list with infrastructure issues they get pissed quite easily so we really want to avoid that and that is not very easy so you actually need to put stuff around GitLab to make that happen. On the slide on the left you see what the test system actually gives you which is Beaker in our case and then there's this one missing so a panic code in Beaker is actually the hardware messed up, the kernel didn't boot but for whatever reason or distribution didn't boot or there was some power surge or whatever or actually we messed up or our general infrastructure or had the networking issues stuff like that and that's not in the system, might never get in there because it's not something that you would normally have for your average. Just a little bit about that, yeah, sorry. So one consequence of that is that GitLab, yeah, okay, sorry, ah this is the line, sorry. So the thing is GitLab has infrastructure issues, GitLab, GitLab CI has infrastructure issues and you can select okay you can restart on this failure, on this failure, on this failure but it actually doesn't matter to me which failure GitLab restarts on because it's GitLab's thing, it can fail in various things but it doesn't allow me to tell like okay restart on this issue or on this issue, I can only say test passed or failed and that probably comes from where GitLab is intended to be used in, it's like test ran and nothing can happen, it's just running tests on simple software but for us if you remember that job like just one infrastructure, four hosts for one architecture and there are like two, three, four architectures more than that and what GitLab does is they just kill us, yeah and that stays there and that's a separate slide, I'm confusing the issues but basically we cannot tell GitLab okay we had infrastructure issue, can you restart? That's a big deal for us. So I mean they are more interesting things, we actually, that's from the beginning of January, we are producing 30 gigabytes of artifacts a day like kernel builds, opm builds, all kinds of stuff and if you use a shared GitLab instance like gitlab.com or whatever have you in redhead, we have a couple of those, people might not have the storage available so we would want to store it outside like in S3 if you build in the cloud you want to keep it there, you don't want to incur the transfer costs moving in and out which is not possible at the moment in GitLab so you can configure it per instance but not per project and it all goes on like these things so there are certain things where it doesn't really match very well with kernel testing and you try to work around it so it's possible but it gets more ugly so you can take a look at the code, it's on GitLab, don't blame us for however we did it. It's really hard to upgrade because if you have like a pipeline running for a day, if you do an upgrade to GitLab run of it, stop to accept jobs so then you don't get any builds, any tests for a day, you can work around it, upgrade different runners at different times stuff like that. What else do you have? Let me just skip down. Oh yeah, that's an interesting one. Most test systems don't expect the test to reboot so I don't know why but kernel tests actually reboot a couple of times, it's like something that kernel developers think is useful so you need to boot into your kernel but then you might also restart a couple of times in there. So you can't really have your test harness like the GitLab part have itself restarted so you need to have another in direction, just start another VM or have the GitLab part outside of your testing system in this case Beaker but otherwise you could just put it inside of your hot weather lab which you can't do at the moment. Yeah maybe we stop here and take some questions, otherwise I just complain about other bugs. Let's take questions. So the question is how much time do you actually gain from having CI? I think depending on who you ask there might be different answers so developers might most likely say like it doesn't help us at all in the beginning especially now where you might actually get infrastructure issues giving you false positives but I think we find a couple of issues a week. Actually four maybe, four a week where kernel developers were really sure that they got it right but they didn't. And that could be patches posted to the mail list or it could be something merged into stable for example in the stable Linux. The ultimate goal is actually to free resources inside of RHEL because upstream patches poke so we want to provide feedback outside of it that never actually goes through the whole pipeline. Now we only find out about it when it's already merged and built into an RPM. The ultimate goal is to have work done upstream which you most likely said. Okay. So you mentioned you were running Jenkins before you migrated. How was the clinical migration and what kind of things did you apply? Did you keep those instances running at the same time period? What did you just start to reach? We just wrote everything and then switched. Yeah the question was how painful the migration was. We took some of the tools that Jenkins was using and we used them in the new pipeline but we wrote everything that was in Jenkins there because you can't really run that in GitLab. So we had to rewrite the pick part and we had to rewrite the triggers and things like that. We replaced the separate tool which was controlled in Jenkins with those little triggers that I showed you. Any more questions? Yes? Did GitLab work on those issues? Yeah. Most some of them. The question was is GitLab working on those issues? So there is an issue that particularly pisses me off is that GitLab simply kills the runners. Yeah, this one. They simply kill the runners with sick kill. So for us it's a runner that's controlling that bigger resources like these, I don't know, 10 hosts that are running those tests for hours and we just forget about them because of that. Like, GitLab just forgets like, ah, whatever. And that host is occupied for these hours so we cannot clean up. And this bug was open for years, I think. And they're promising they will fix it soon so I hope they will. Okay, our time is up so catch us in the corridor.
 So we are going to talk about this thing.  And us is me, Nikolai Kondrashov, and Michael is sitting here.  He'll join me later.  Neither of us does actually look like that anymore.  So we are coming from Red Hat CKI team or CKI project, and we are a distributed team  which are doing the kernel CI at Red Hat.  So why do we need kernel CI?  Well, as most of you probably know, we do releases of distributions, and each release  has different kernel versions.  There's a lot of kernel versions.  And moreover, we are one of the major contributors to the kernel and certainly the biggest one  among distributions.  So this shows the comparison of unique email addresses from SUSE, Red Hat, and Canonical  contributed to the kernel.  And this is commits.  Each of those is per year, and the Red Hat is the blue one.  So we somehow got to make it consistent and reliable.  And if you look at this, this is a big cue of how code goes through the pipeline towards  Red Hat.  And until not so long ago, our tests were only there at the end.  So the developers would throw the commits over to the builds over to the QA, and they would  test it and then come back, and they would retest it.  That takes a long time.  And if you consider how long it takes for the whole code pipeline to execute one patch  to digest the patch through that and get to a release, it's a long way.  So what we want to do is we want to do this.  And we want to do it fast and provide as much as possible feedback so that ideally the bugs  are caught before even maintainers see them at the moment that the developers submit those.  And that is hard because there is just so much email.  I mean, look at this.  So we somehow are supposed to, I'm not having anything against this message that just shows  the complexity of these things.  So somehow we are supposed to put our web hooks in here.  This is patch number 62 out of 114.  And there is this amount of discussion going on.  So somehow we are supposed to test those and provide feedback to developers.  So this has been an ongoing discussion recently in the kernel circles.  And for example, the last Linux plumbers, there was one of those presentations that  are happening recently by Dmitry Bujkov, who did a very good take on those issues.  And I recommend you watch it if you're interested in the kernel development process.  He makes good points.  So what we've built is something like this.  This is simplified.  I actually lost a slightly more complex slide on the plane here because of how Slice.com  works.  But no mind, this is very simple.  And we'll go around.  So normally, if you wanted to check just the changes in the Git repo being committed there,  that's kind of easy.  So we have a bunch of repos with track.  And we check if there are new commits and we test those.  That's fairly trivial.  We actually have, inside of that, we have a bunch of Git repos for different releases  like RL7, RL6, RL5, RL8.  And we also track upstream repos, mainly stable at this moment.  But we track also a few others.  And we test those commits, of course.  And that's relatively easy.  We just pull the repo and run our test about which I'll tell you a little later.  Then there's the interesting part which I started with.  Turns out it's not that hard.  So it's hard.  It's hard.  Don't get me wrong.  But you can do it easier.  So there's a typical mail list, like Linux USB mail list.  There's a message from a series.  It looks like this.  Turns out there is a project called Patchwork, probably many of you know, which is used by  main trainers.  Well, most of all, they use it to track the patches as they've been processed, reviewed,  and tested.  And they check which patches were merged and which not.  So it looks like this.  If you go to patchwork.kernel.org, there is a bunch of projects.  And those projects can be mapped to a particular mail list or even a particular tag being  used, like tagging the subject being used in the mail list.  Well, at least the thread head is done this way.  I'm not sure if that's upstream actually.  So if you go to the same Linux USB mail list here, you can see there is again those patches.  But this time they're organized into series.  If you click on one of those links, you get to the particular patch series.  There are two patches in there.  We can go to specific patch and see what's going on, what's the patch, and we can download  the inbox there on the right or the whole series.  And the main thing that concerns us about this is that patchwork has REST API.  So we can go through those projects.  We can extract the patch series, the patches, and everything.  And we can track when they are peering.  This is, of course, sounds very simple, but the devil is in the details where you have  to kind of expect that not all messages come through at the same time.  When you go check and the series might not be complete, there are bugs, people sending  all kinds of messages in there.  Sometimes they're not picked up and things like that.  Patch series can have cover letter, don't have cover letter and things like that.  But you can make it work.  So the typical patchwork trigger would be tied to a particular patchwork instance and  a particular project there and associated with the corresponding Git repo.  Further on, we also have triggers from our package builds in Koji and the developers  builds in copper for Fedora and internally for L, although it's called a little differently.  Pardon.  Let's go back.  So Fedora build system looks like this.  There are a bunch of packages.  They are being built, prepared for releases and reviewed.  So we can look for kernel.  There's our kernel.  Let's take this one.  And here is information on the build of the kernel, like specific revision and everything  and all the packages that were built for all the architectures.  And copper looks like this.  Oh, wonderfully.  Finally connected.  So copper is more for developers.  You can have your package build and put into our RPM repo and picked up by your users or  by other developers.  We track those as well.  So you can look for the kernel there and find one of those and go in there and into builds  and see there's been a build and here's the packages.  But we don't talk through the web UI, of course.  There's the packages.  We talk through the, we listen to the Fedora message bus, which is used both by Codege  and copper and internally at Red Hat.  There is a message bus as well.  We just listen to the message.  There is a log from our trigger and checks like, okay, there's build completed.  There's a message coming through the bus.  We're not interested in this one, neither in this one.  But here is the kernel.  We pick it up and we trigger our pipeline.  So we also have to test our own CI.  So we have two special kinds of triggers for GitHub and GitLab for testing contributions  to our CI repo, so which we have many.  And as you can see, we have some on GitHub and some on GitLab because of historically.  So it looks simply like this.  You submit a PR or an MR and the bot comes and says like, hey, I'm a bot.  Send me a message and I'll test this for you.  So the developer says like a test please bot.  And bot says testing and then it triggers the pipelines through for the various, for the  various repos that we have.  Developer can go have lunch or two and then finishes and the bot says like passed or failed.  Well, as it happens, here's an example of my food request.  So the bot comes in and tells us like, here I am.  This is what you can do.  And it's a little different for GitLab and for various repos and can say like, oh, add  this keyword or that keyword and we'll test this and that and things like that.  And yeah, I am asking the bot to test.  So the bot says, yeah, I'm going.  Then it's supposed to resolve or failed, of course.  So that's how we do testing for our CI because yeah, many repos and because our CI is in  a separate pipeline and we have like, we can certainly have.  Two GitLab repos actually handling this bot this not repos GitLab instances.  That's a complication.  A fun one.  Yeah.  So further on that's and these are two major parts of our pipeline.  Of course, the test database and the tool which lets us pick which tests to run and organizes  everything into trees and other dependencies like where we can put we can run with this  build with that build on this architecture and that architecture and et cetera, whether  developers want to test this or that.  So basically the data flow is very simple.  They have Kpdb, which is a repo with the test information, which is currently private  because there is like all kinds of stuff for rail internal ones.  And we are still intending to open it up, but it's difficult to separate the upstream  tests from downstream tests in a complicated data structure and somehow merge it together.  So we had the database, which is basically YAML and we have a tool which takes it and  a bunch of parameters and then speeds out XML for bigger system, which actually runs  our tests.  And this system is installed internally, but it's open source, but you will have a very  hard time actually trying to install it, which we are working on right now very hard and  hopefully will be able to let you guys enjoy it.  But it's all open source.  It's out there, goes into documentation.  It's just nobody succeeded installing it by themselves.  So the databases can contain information about architectures again, of which there are five  right now.  Kind of host types and this describe what do we want the host to have, like do we want  to have this much RAM or this many CPUs or this much storage or even a particular PCI  card or a network card.  And we organize them into host types because that's easier.  They have three, so obviously particular repos or types of repos we want to test and those  effect which tests will run where like for example one test can run on REL7 but not on  REL5 and some on upstream and some tests are still internal.  Not many though.  Most of them are actually out there.  Components which describe what things the build contains, like upstream only contains  kernel image, but internal builds, they are built using RPMs and there could be debugging  for headers, the internal kernel headers and things like that or tools that some tests  need like for example some tests need debugging for things like that or some tests don't run  on debug build.  And then we organize tests into sets of course like for network tests, for file system memory,  etc., etc., virtual machines and of course the description of the tests with themselves  of which there are quite a big number, assume to be 100.  And these range from simple tests like just a shell script which just restarts the kernel  test something and it's done to very big ones like LTP, USEX and the top ones are listed  there I guess but those are not all which can contain like thousands of tests.  So an example of tests with data like a description where it is in the report is actually quite  outdated.  Anyway, the essence is there.  Where it is located and for example this one is in our test repository on GitHub where  most of our tests are, which host type it runs on, additional information of like I want  like this very specific host for this like and it could be down to a specific host name  on our in our bigger system like I want to run it like exactly here on this machine because  there's only this hardware there.  This description could look like this actually this says don't run on these ARM systems because  they don't work.  So information on maintainers and what's not well this is a discussion for upstream  that's not the depth for now so these are test maintainers who look after the test and  check that it's working and it's failing and they actually receive copies of failure reports  and they're supposed to take a look at as soon as something happens and tell the developer  okay sorry that's my bag that's a failure or say like this is your problem and they  are responsible for those tests which is going to upstream important because upstream developers  they don't see that much into our machines and things like that so they have a hard time  figuring out what actually happened which we are working on.  So there's a can there's the conditions for the test to run on like the sets which it  belongs to and this is also outdated my gosh.  This is also an interesting part that we specify which source files the particular test covers  more or less so that we can avoid running it when there is a patch that doesn't touch  those files and that's why we can kind of contain the runtime at least a little bit make  it shorter when we don't need it.  And this allows us to kind of describe when to run them for which code.  Architectures the test will run on and which trees it belongs to but there are no components  here because this is old and there could be like multiple cases of this RIP I wanted to  run it with this file system or the dot file system for example a file system test or additional  parameters and in walking the K-pad to normally people don't invoke it by hand but it runs  in the pipeline so you can say like okay generate me the XML for this run for this kernel  turbo for the upstream tree AR64 with this patch and highlight the output and it would  look something like this and it goes on and on and on and I'm not going to bore you with  those details.  So this is the input to beaker and saying how to run it and where to run it and in which  order.  So going to beaker it's a big system which maintains inventory for the hardware including  down to the components.  Let's you match that hardware has access control like particular groups have access to this  hardware those to this hardware and for example some NDA hardware could be there and protected  it also does the provisioning and puts up the machines, installs the operating system  from scratch using an account normally because we don't support running from images because  that's hard to do and we are distribution so we have to test the whole distribution  from install.  So it installs everything, it talks to the test harness, extracts test results and things  like that and looks after the machine so that if it does lock up it then it releases the  machine and erases everything.  So it could be like the system inventory we can find of those machines like these are  not very useful right now this is a titanium we still have those.  You can go into the machine and take a look at the details like this is just one tab about  the host information there's the CPU info storage peripherals things like that and this is an  example of some of our jobs running for stable repository of the Linux kernel.  This one job is just for one architecture and it has four hosts and here's an example  of one hosting executing those tests this is a bigger UI this is a bunch of tests there.  Further on now we are approaching the user visible stuff so we have a reporter which  watches over the pipelines and checks which stage they are on which job they are approaching  and sends the email reports to developers or whoever is interested and sometimes it  can send an early email saying like okay have we started this test and like watch out or  where we did the testing or something failed in the pipeline.  So there's an example of a successful report that was sent to a stable mail list here's  the it starts with the saying like we took this repo we took this commit and there's  the summary everything went fine and we were actually compiled and use those comments and  then we run them on these hosts like this architecture is ARC64, cross-host second  host, PPC64, two hosts, X86, X86 got more hosts, four hosts and we also have a notion  of waved tests, a test which you mark in that KPEB DB and saying this test is waved which  means run this test as normal maybe at the end of the run but ignore the result and don't  take it into account when giving a verdict whether it failed or not we say we ignore  it and we use this to test the test which were just introduced into the system or were  being fixed so that we can track like how are they performing like they doing okay and  the test maintainer can look after it until it stabilizes then remove the waved status  and this is done manually because tests are different you have to look after them.  And that's an example of report that was sent upstream our internal reports are a little  more elaborate you get to see actually links to the bigger results and explore the logs  and everything but those tests actually have yeah artifacts there is a blue link there  this contain the binaries, config files and logs things like that.  So and then finally we have the data warehouse it's a system which uses PostgreSQL and collects  all the information about our runs also similar to reporter and this kind of application of  the effort at this moment but we are working on that.  So it watches over all the jobs and collects the information like how it went and what's  the status and much tests run and there is a web UI which looks something like this  and provides statistics how much we failed how much we succeeded as pipelines and various  statuses this has been pulled like from GitLab using the GitLab API and there's a particular  pipeline and listing all the tests and all the hosts and you can go and see the results  in Beaker how it went.  And we maintain the test statistics how tests were failing or passing for exactly for the  purposes of deciding like when to waive the test if it's been doing bad and then send  it back to the maintainer and say like okay deal with it or we can actually take it out  of wave state if it's doing okay.  And same for hosts like if some host is behaving in Beaker and that's a problem because there  are just so many hosts that they break and you have to watch out and the host maintainer  like whoever maintains that host they don't have time to look after it so we look at those  and we say okay this host should be excluded from the runs and we add like don't run on  that host.  So finally the title of this talk no not yet this actually concerns Guillaume's talk  so there is this thing you probably seen it at the last slide.  It's the kernelci.org and there are lots of tests and they were recently approved as  accepted as a links foundation project to advance the state of kernelci and we joined  that effort and right now we are working on a database and the system used to aggregate  testing information from various CI systems so that at the ultimate goal so that there  is a single place to go and check kernelci results from whoever runs those tests and  so that the developers get on a single email with those results and not just five emails  from everyone.  Right now this is mostly the kernelci folks and us but others are joining hopefully soon  we start aggregating more data but we already have a tool.  You can take a look like how this looks I don't know if Guillaume showed this but this  is an example of how test reports like top level could look there.  So we took a Google BigQuery system for storing those results so they are more readily publicly  available and so that people can go and explore the data and see how kernel is doing and do  research if they need to.  So this is our repo with the code for that and it looks something like this when it's  pushing and it's the data and we are working on a dashboard to show this off and provide  the developers.  This is very resimentary at the moment.  So finally the interesting part I took a little while.  So we store our CI pipeline inside YAML but we store it in separate repos because of the  way we trigger those.  So to trigger GitLab we are actually doing commits to the repo and I'll show that in a  moment.  Basically these are two repos and one the repo on the left is only including pieces  from the other repo.  And this lets us to let the triggers do commits with the information about what we want to  test inside that repo.  And we need two repos so that these commits don't interfere with the development commits  we have.  So because there is like every time you want to test something there is a new commit and  that's an empty commit and it doesn't have any data in it so we use it just to identify  the particular pipeline and GitLab view.  So for example the baseline trigger like the Git repo trigger comes in and does and  sees that there are two changes in the like in one repo and another and does commits to  separate branches in that repo.  This trigger actually is retired now but it was quite kind of interesting.  So it also does and checks and finds something and does the commit.  Because the trigger that finds patches and does commit in that same branch and finally  the GitHub board comes and finds like okay there is a new merge request and I put it  in all the branches that we're interested in testing.  And it might look like this for example the stable branch has those commits all with the  pipelines running and this branch has its own commits here.  And the commits can look like this.  So there is data there but it's not for GitLab consumption only for us as debugging like this  says.  All the variables that we put in there all the descriptions like what we are triggering  on things like that this one is huge and this one is big as well it's abbreviated.  So we use a lot of GitLab extends property which lets us separate the general pipeline  into the pipeline into the menu of jobs and stages and into three specific information  like our pipeline specific information.  This is our shortest pipeline and it says okay pick those four stages from the pipeline  and we have ten maybe or more and this says okay pick this prepare step where we download  all the stuff for the execution all the dependencies and this says like this is the prepare and  this is stage prepare and this one we say okay again pick this create repo x8664 and  extend it and this is one of those same okay take this repo take this template of the job  and a bunch of variables and conditions and create a particular job for this specific  pipeline here.  And this time we are using the merge keys to merge those.  So we use an extend here extends here because this is a separate YAML file so we cannot  use merge keys.  And we use merge keys here it's the same one big as YAML file there and this is what  be create repo creating a RPM repository with build results for testing which are then installed  in biker.  Next stage is composed a little differently so we have a huge script which is split into  a few YAML objects and finally the last stage looks similar to that and so on so we have  pipelines which are much longer than that and more involved.  So why we took GitLab well we started out with Jenkins so we had a Python script controlling  Jenkins which had a Joe Britton Groovy which controlled another Python script which checked  out the kernel and build it and then fed it off to biker.  So that was not very reliable hard to debug, hard to understand, hard to maintain as a  contrast with GitLab we had relatively straightforward system you could keep everything in the Git  repo and keep changing it faster and it's more reliable than Jenkins for us.  And I hope Michael is able to say something, we have ten minutes left.  Now you will hear me complaining about GitLab but so I don't know how many people used GitLab  here and how many people have used GitHub.  So it's very familiar system so it's nicely documented, it has a huge API surface, people  are familiar with it so if I say GitLab people actually know what I'm talking about, if I  mention some other CI technology then people look at me like this, if I mention Jenkins  people just go away and say like no you don't want to be on your team, hopefully there's  nobody from Jenkins here, sorry.  But then we are actually testing kernel so that's quite similar to other software to  test in some aspects but for other aspects especially like testing there are some interesting  issues that you will see there.  The one is that GitLab, most of these general CI systems don't have any concept of a failed  pipeline because of infrastructure issues so if you look into what distributions do  for gating most of those actually have a test fail thing, maintain or fix it and then they  have something like oh our infrastructure failed or our test system and then it's actually  for somebody else to fix.  Now most of you might know that kernel maintainers don't react too well if you email them without  any good reasons or if you email the Linux kernel list with infrastructure issues they  get pissed quite easily so we really want to avoid that and that is not very easy so  you actually need to put stuff around GitLab to make that happen.  On the slide on the left you see what the test system actually gives you which is Beaker  in our case and then there's this one missing so a panic code in Beaker is actually the  hardware messed up, the kernel didn't boot but for whatever reason or distribution didn't  boot or there was some power surge or whatever or actually we messed up or our general infrastructure  or had the networking issues stuff like that and that's not in the system, might never  get in there because it's not something that you would normally have for your average.  Just a little bit about that, yeah, sorry.  So one consequence of that is that GitLab, yeah, okay, sorry, ah this is the line, sorry.  So the thing is GitLab has infrastructure issues, GitLab, GitLab CI has infrastructure  issues and you can select okay you can restart on this failure, on this failure, on this failure  but it actually doesn't matter to me which failure GitLab restarts on because it's GitLab's  thing, it can fail in various things but it doesn't allow me to tell like okay restart  on this issue or on this issue, I can only say test passed or failed and that probably  comes from where GitLab is intended to be used in, it's like test ran and nothing can  happen, it's just running tests on simple software but for us if you remember that job  like just one infrastructure, four hosts for one architecture and there are like two,  three, four architectures more than that and what GitLab does is they just kill us, yeah  and that stays there and that's a separate slide, I'm confusing the issues but basically  we cannot tell GitLab okay we had infrastructure issue, can you restart?  That's a big deal for us.  So I mean they are more interesting things, we actually, that's from the beginning of  January, we are producing 30 gigabytes of artifacts a day like kernel builds, opm builds,  all kinds of stuff and if you use a shared GitLab instance like gitlab.com or whatever  have you in redhead, we have a couple of those, people might not have the storage available  so we would want to store it outside like in S3 if you build in the cloud you want to  keep it there, you don't want to incur the transfer costs moving in and out which is  not possible at the moment in GitLab so you can configure it per instance but not per  project and it all goes on like these things so there are certain things where it doesn't  really match very well with kernel testing and you try to work around it so it's possible  but it gets more ugly so you can take a look at the code, it's on GitLab, don't blame us  for however we did it.  It's really hard to upgrade because if you have like a pipeline running for a day, if  you do an upgrade to GitLab run of it, stop to accept jobs so then you don't get any  builds, any tests for a day, you can work around it, upgrade different runners at different  times stuff like that.  What else do you have?  Let me just skip down.  Oh yeah, that's an interesting one.  Most test systems don't expect the test to reboot so I don't know why but kernel tests  actually reboot a couple of times, it's like something that kernel developers think is  useful so you need to boot into your kernel but then you might also restart a couple of  times in there.  So you can't really have your test harness like the GitLab part have itself restarted  so you need to have another in direction, just start another VM or have the GitLab  part outside of your testing system in this case Beaker but otherwise you could just put  it inside of your hot weather lab which you can't do at the moment.  Yeah maybe we stop here and take some questions, otherwise I just complain about other bugs.  Let's take questions.  So the question is how much time do you actually gain from having CI?  I think depending on who you ask there might be different answers so developers might most  likely say like it doesn't help us at all in the beginning especially now where you  might actually get infrastructure issues giving you false positives but I think we find a  couple of issues a week.  Actually four maybe, four a week where kernel developers were really sure that they got it  right but they didn't.  And that could be patches posted to the mail list or it could be something merged into  stable for example in the stable Linux.  The ultimate goal is actually to free resources inside of RHEL because upstream patches poke  so we want to provide feedback outside of it that never actually goes through the whole  pipeline.  Now we only find out about it when it's already merged and built into an RPM.  The ultimate goal is to have work done upstream which you most likely said.  Okay.  So you mentioned you were running Jenkins before you migrated.  How was the clinical migration and what kind of things did you apply?  Did you keep those instances running at the same time period?  What did you just start to reach?  We just wrote everything and then switched.  Yeah the question was how painful the migration was.  We took some of the tools that Jenkins was using and we used them in the new pipeline  but we wrote everything that was in Jenkins there because you can't really run that in  GitLab.  So we had to rewrite the pick part and we had to rewrite the triggers and things like  that.  We replaced the separate tool which was controlled in Jenkins with those little triggers that  I showed you.  Any more questions?  Yes?  Did GitLab work on those issues?  Yeah.  Most some of them.  The question was is GitLab working on those issues?  So there is an issue that particularly pisses me off is that GitLab simply kills the runners.  Yeah, this one.  They simply kill the runners with sick kill.  So for us it's a runner that's controlling that bigger resources like these, I don't  know, 10 hosts that are running those tests for hours and we just forget about them because  of that.  Like, GitLab just forgets like, ah, whatever.  And that host is occupied for these hours so we cannot clean up.  And this bug was open for years, I think.  And they're promising they will fix it soon so I hope they will.  Okay, our time is up so catch us in the corridor.
testing_abusing_gitlabci_test_kernel_0_167.png testing_abusing_gitlabci_test_kernel_168_299.png testing_abusing_gitlabci_test_kernel_300_539.png testing_abusing_gitlabci_test_kernel_540_767.png testing_abusing_gitlabci_test_kernel_768_1115.png testing_abusing_gitlabci_test_kernel_1116_1727.png testing_abusing_gitlabci_test_kernel_1728_1919.png testing_abusing_gitlabci_test_kernel_1920_4271.png testing_abusing_gitlabci_test_kernel_4272_4691.png testing_abusing_gitlabci_test_kernel_4692_4799.png testing_abusing_gitlabci_test_kernel_4800_6899.png testing_abusing_gitlabci_test_kernel_6900_7343.png testing_abusing_gitlabci_test_kernel_7344_7811.png testing_abusing_gitlabci_test_kernel_7812_7871.png testing_abusing_gitlabci_test_kernel_7872_9491.png testing_abusing_gitlabci_test_kernel_9492_10271.png testing_abusing_gitlabci_test_kernel_10272_10715.png testing_abusing_gitlabci_test_kernel_10716_10787.png testing_abusing_gitlabci_test_kernel_10788_10943.png testing_abusing_gitlabci_test_kernel_10944_11471.png testing_abusing_gitlabci_test_kernel_11472_11651.png testing_abusing_gitlabci_test_kernel_11652_11963.png testing_abusing_gitlabci_test_kernel_11964_12335.png testing_abusing_gitlabci_test_kernel_12336_13631.png testing_abusing_gitlabci_test_kernel_13632_14075.png testing_abusing_gitlabci_test_kernel_14076_19883.png testing_abusing_gitlabci_test_kernel_19884_23939.png testing_abusing_gitlabci_test_kernel_23940_26315.png testing_abusing_gitlabci_test_kernel_26316_26915.png testing_abusing_gitlabci_test_kernel_26916_27407.png testing_abusing_gitlabci_test_kernel_27408_31847.png testing_abusing_gitlabci_test_kernel_31848_33995.png testing_abusing_gitlabci_test_kernel_33996_35507.png testing_abusing_gitlabci_test_kernel_35508_36215.png testing_abusing_gitlabci_test_kernel_36216_36311.png testing_abusing_gitlabci_test_kernel_36312_36611.png testing_abusing_gitlabci_test_kernel_36612_39851.png testing_abusing_gitlabci_test_kernel_39852_40847.png testing_abusing_gitlabci_test_kernel_40848_40895.png testing_abusing_gitlabci_test_kernel_40896_41087.png testing_abusing_gitlabci_test_kernel_41088_41195.png testing_abusing_gitlabci_test_kernel_41196_41363.png testing_abusing_gitlabci_test_kernel_41364_41483.png testing_abusing_gitlabci_test_kernel_41484_41531.png testing_abusing_gitlabci_test_kernel_41532_41675.png testing_abusing_gitlabci_test_kernel_41676_41723.png testing_abusing_gitlabci_test_kernel_41724_41891.png testing_abusing_gitlabci_test_kernel_41892_42023.png testing_abusing_gitlabci_test_kernel_42024_42155.png testing_abusing_gitlabci_test_kernel_42156_42203.png testing_abusing_gitlabci_test_kernel_42204_42491.png testing_abusing_gitlabci_test_kernel_42492_42695.png testing_abusing_gitlabci_test_kernel_42696_42875.png testing_abusing_gitlabci_test_kernel_42876_43007.png testing_abusing_gitlabci_test_kernel_43008_57467.png testing_abusing_gitlabci_test_kernel_57468_58547.png testing_abusing_gitlabci_test_kernel_58548_58666.png