Welcome to KernelCI
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Tucker, Guillaume
KernelCI is a project dedicated to testing the upstream Linux kernel. Originally created by Linaro in 2014, it started a new chapter by becoming a Linux Foundation project in October 2019. Its future looks bright, with plenty of opportunities for new contributors to join.
 Hello everyone. Thanks for waking up early and attending this talk on Saturday morning. Can you hear me? Can you hear me at the back? Yes? Okay, cool. So there's no sound system. This is only for recording, in case you're wondering. So my name is Guillaume Taka. I work at Colabra and this talk is about kernel CI. So I had a talk last year already about kernel CI. This is kind of an update on what things have happened in the past year, but also about the future, really. So what is the purpose of kernel CI in case you're not familiar with the project and also what it wants to achieve in the longer term? So it's kind of climbing a big tall mountain, really, without trying to reach the point where kernels are released without bugs and have at least quality control. So we know which bugs are in the kernel. And extending test coverage and also having reusable tools. So the idea is to test the upstream kernel, but anyone using the kernel in their own downstream environment should be able to reuse these tools. We now have an official mission statement as written here. So that's basically what I'm trying to explain here, but this is like the official word we have now. People spend some time making this up to our thought. It deserves at least a slide here. So, yeah, last year I said maybe it will become a Linux Foundation project and this has happened. So in October, kernel CI has become a new project of the Linux Foundation with all the members listed here. So Colabra, BDB, Red Hat, Google, Microsoft, FandRissel, IO and Civil Infrastructure Platform, which is another Linux Foundation project. Now, we're still in the early days of we know what does that actually mean for the project. So we have a mission statement. We have a budget and some stickers, but there's a lot of other things coming soon, hopefully. There's a website also, part of the website explaining what that means for the project. So although it has become a Linux Foundation project, the aim is still to be really community-oriented. So of course there are things that are done by the members, for the members, but it's really to facilitate things to happen for the kernel community at large. So we're still focused on upstream completely. I mean, the tool again can be reused for other things. So if Red Hat wants to test a Fedora kernel, they can do that. Or maybe the project will help make it happen as an example. But it's still about testing the main one kernel, upstream kernel. And it's still about sending email reports to mailing lists and engaging more with subsystem maintainers and developers in general to basically help people add their own tests to kernel CI. A lot of, well, every subsystem have different workflows. And some maintainers have created their own small CI, which works perfectly well for them probably. But then they might be reinventing some wheel and also not have access to all the hardware that kernel CI provides. So anyway, that's the kind of consideration that is the main objective we're trying to achieve. So one kind of philosophy behind kernel CI is what we call the open testing philosophy. Because there is only one main kernel. So in the same way that people from many different origins, from many different reasons, contribute to the same code base that Linux kernel, when kernel 5.4 is released, that's what 5.4 is going to run on a supercomputer, on an Android phone or on anything. It's the same kernel, the same code base, although it does a lot of different things. So the reason why there are so many different test systems is because of these different use cases. But we could also have a common way of at least sharing the results, because some tests are relevant to all of these cases. If we want to test for memory leaks in the kernel, you can do that wherever the kernel is running. So everybody is doing their own product tests. But the idea is to have a test system that allows people to contribute their tests and contribute their hardware in the same way that they contribute the code. So like I said, there's a large number of contributors. So yeah, basically that's what I explained here. The image is trying to say that we should try to have the same surface of between tests and the code. So it's a bit like test coverage if you want, but like aligned the two things, like the whole idea of having open source development and having open source testing, open testing on top of it, basically. Now, how do you achieve that? So this is like an abstract pipeline explaining how kernel CI works. For all of these blocks, there is at least one implementation, but you could have different things for each of them, different instances. So let's start from the beginning. So you have a developer in the top left corner, so someone writing some code. So the code ends up as patches. Kernel CI doesn't test patches directly at the moment, but it can be done. Other systems do it. Or Git branches, then of course it gets built. The purple arrows are like when files and results are pushed to, then you have build artifacts. And after that, when you have a build, you can run tests and then process the results to detect when there was a regression or basically see what needs to be reported from the results. And then your results are stored and analyzed to be able to then provide a report to the people and the developers. And this is when the loop is closed. It's like an automation when you have, the objective is to have a working kernel. Like if you try to automate an arm, for example, like a mechanical arm, you want to know where, in which position it's going to be. If it's completely open and you just move something and you don't really look at it, you can guess where it's going. But if you have some sensors that will actually know where it is and you know the angle or everything, then you can have closed loop or automated system. And this is kind of what we're doing here except that it's doing this around developers and not around machines. But it's a similar kind of thing. Sorry, can you speak louder? Yeah, sorry. I'll speak louder. Thanks. There's a link here. So these slides are very little as PDF. There's a link on the page for this talk. And on this slide, seven, there's also a link to a shared document that has a more detailed description of the pipeline design, modular pipeline. So this is like the theory. Now in practice, what is that, how does that work? So right now we have, well, kernelCI.org, there is one database that's being used. That's been used for a long time. But we have a new one using Google BigQuery. And the idea is to use that as a prototyping database but with access to more testing systems. So all the tests currently, the test results currently available on kernelCI.org are also sent to this database. And now we have also Red Hat submitting test results to it. And we would like to see maybe other people. So typically, member companies would be the first ones to contribute to that. So we're kind of refining the database schema for like storing build results and test results. And the first outcome we would want to have is have a unified test email report. So based on all that collective data, we could then send an email, say for, you know, a stable branch release or something. Instead, I don't know if you've noticed on stable branch mailing lists, on stable mailing lists, you have a lot of test results from coming from different test systems. So that's one of the targets for having a common upstream kernel testing test system is to have only one report that's gathering all the data. So having this database is the first step. So if you look on the previous slide, that's basically the store. So yeah, store is artifact and database, but this is at least the database part. Yeah, and then we can build kernels in different places. So that's, you know, the block number two build, how do you build? If you have different people providing different, like, cloud compute and different ways of doing it, if you're modeling enough, then you can include them. So that makes the project easier to scale. And right now we're using Jenkins, but we've made it, we've changed it in a way so that it can run anywhere. So you can run all the kernel CI steps in the terminal. And some people have tried also using it in GitLab already. So Kevin, Kevin Hillman has done this in a, as a GitLab CI way. So he's using the kernel CI command line tools in his, in a branch, in a kernel branch. So automatically when there's, when your new changes are being pushed to the branch, it uses GitLab CI to build and test things like in other GitLab CI projects. But this could be used in other test systems as well. And also abstracting the lab API. So, you know, test labs are where hardware is available to test kernels. So by default we use lava, which is, you know, the linear project for automating tests. But it's not the, I mean, there are plenty of other things. So we've made an abstraction in the kernel CI core tools so that it can submit things to any lab. So right now it's only sending things to lava, but we can send it to any other lab that has a web API and can accept a test definition. So these are things we're working on. So yeah, the development goals, a bit more specific. So the command line tools are like KCI build to do, to reproduce a kernel CI build from scratch. KCI tests are to generate test definitions. So these are the things we're trying to improve for people so they become more portable. Also we want to make it easier, like I said a bit before, to make it easier for people to contribute tests. So if you're a maintainer or developer and you have a test or maybe you work on a part of the kernel and you make some tests for it, it should be easy for you to enable this in kernel CI. And right now this kind of a steep learning curve, if you want to do this at the moment, you need to be, you need to know too much about the internals of kernel CI to be able to add a test to the list. But that's one of the things we try to like lower the bar basically. And we need some good interaction with that. So we need some collaboration for the people who make the test as well. So we started with like a key people to try to do that. So maybe next year I'll come back and see. If you follow the project you'll see maybe things will become easier. But right now it is not very easy. It's still a good thing. If you have a test you really want it to be in kernel CI, you can send it to the mailing list or an IRC, kernel CI on pre-node, or kernel CI groups.io mailing list. Or just on the subsystem mailing list, people will know about kernel CI. Now, also, yeah, we're trying to improve the web dashboard. So I don't know if you've seen the current kernel CI dashboard. But here we're showing mostly builds and boot results. It's not really, there is a test tab, but it's not really good at showing test results. Because kernel CI was originally about billing kernels, like a variety of different ARM DevConfig. And then about boot testing. And it's still kind of stuck in that phase. So we're starting running tests, but the results are not shown properly on the web front end, on the web dashboard. So that's one thing we are fixing actually right now to unlock a lot of things. There's many tests that we kind of held back because maybe we could run them and then email reports have some limitations. And if the web dashboard doesn't show all the results, then the test kind of get hidden and that's a waste of results. But we still also need to improve the email reports. That's also a two-way process because it's hard to design something from the beginning. It's more like sending email reports and seeing how people react to it and whether the world is useful, what is not useful. And different subsystems or different mailing lists need different types of reports actually. So the key message I guess here is we need feedback. And if the email you get is not perfect, often people say if you send another one like that, I'll just put your filter to put all your KNLCI emails in spam or in junk. Well, that's maybe, yeah, if it keeps getting bad all the time. But normally we try to improve things as we go. So the key message is we can't get to perfect things from day one basically. And also a slightly more longer term project goal is to improve test bisections. So we have bisections for boot tests, for just plain boots at the moment. So like booting to a login, if you can log in there, it's a boot. And that's very easy to bisect because it's only one pass-fail result per test, per run. But as we're running more test suites that have a lot of tests, then it becomes harder to bisect things when you have different test cases, passing or failing at different rates. So that's, we have, I'll come back to this in another slide, but we have some ideas on how to do this as well. So reusable tools. Yes, so we have KCI build, KCI test, KCI root FS, and there's, to create root file systems. There's a wiki page that might be written in a not too distant future, but right now there's a wiki page kind of work in progress that gives you at least a starting point if you want to start, if you want to see what these tools do. We also have the interface to the BigQuery database. So there's a common line tool for submitting kernel CI results to the common kernel CI. So if you have a test system and you want any testing upstream kernel, you want to submit results, you can get in touch and use that tool and, you know, we can give you a token and then you submit your results. Then also Docker containers. I haven't worked on it myself recently, so I'm not an expert in the state of all of these, but I know that there's some good progress that has been done to have containers with a lava instance with Jenkins configured for kernel CI and the backend that's the backend currently used on kernel CI.org with MongoDB and front end, which is here on kernel CI.org. So there are containers to make it easy for someone to recreate the whole system in their own environment. And again, if you try to find some issues, it's all on GitHub on the kernel CI project on GitHub. But we need more people to use it, basically, to get some feedback on it. So about advanced by sections. So there's this tool I worked on called Skelphal, which is inspired by EasyBench, which was written by Martin Perez at Intel. So we've been talking because he made this tool for graphics testing. It's actually not really used in production EasyBench at the moment. But it had a much more advanced by section feature than the normal Git by-sec command you have, because this can deal with several test cases at the same time. So when you run one test, you can provide the results for many test cases. It can also request tests to be run several times if they are not stable. So the Skelphal project is basically extracting this and boiling it down to a more portable thing that can be used. And if you look at the project on this GitLab project, you can find there's a demo that will run, like create a Git history with dummy pass-fail things, and it will find a problem in it. And it's kind of ready to be used in kernel CI, but it will come a bit later, I think. But I hope that will really help for some things. So people say maybe we should be testing patches as people send them, because when you have one patch on the mailing list, you can test it on its own, and you know if it's good or bad before it gets merged. And yes, that's ideal, but it means there's a lot of patches. So adding, you know, supporting all of it and, you know, building, if you want to build all the kernel conflicts we build, or even a subset of that, say if you want to build 10 kernels for each patch that is sent and tested with 200 different platforms, and, you know, that doesn't scale very well, I think. So maybe in some cases, if we do it wisely, we can only test a subset based on which subsystem the patch was sent to, but it can take a lot of tweaking. So the idea of the whole advantage of having bisection is when you test Linux Next, for example, where all the new patches that have been applied are merged together, and if something breaks, if something breaks at this point, then you only need maybe 10 iterations to bisect from yesterday's Next to today's Next. So you build 10 kernels, basically, to find when you know that there is actually a problem. So it can be very efficient. So that's why bisection is such an important thing for kernel CI. Okay, so tests, wish list. So these are the things that have been identified so far, but again, maybe some people would think that network keeps failing all the time. Other things maybe are also important to test. But these are kind of obvious things that are directly to the ones at the top are the ones that kind of come with the kernel. And we're not really doing them too well right now, but that's, I think that's like an obvious list of things we should be doing. And in the Next Test Project, I know that LTP that was mentioned a bit earlier, that's at least some parts of it, because some parts may not be completely relevant to the kernel, maybe more like user space oriented, but there are definitely some things in there that we, that you would expect to be run on the kernel test system. And then some are more subsystem centric, like, you know, for media subsystem, you have visual to compliance and other things, and IGT and XFS tests. There's a lot of them, basically. And this is where more integration with, well, more communication with the subsystem maintainers and developers needs to happen. That's basically where we're trying to be, we're trying to go. There's a lot more that can be said. It's a very big project. It's kind of old because it has been around for five years, but it's new at the same time, because now we have a budget and we have some ideas. We have a lot of limitations have been removed since we've joined the Linux Foundation. So I'm hoping it's like a rebirth for the project. We need also some engagement from the people. We need people to realize that now, you know, this has been selected by the Linux kernel community, basically, as the main project for testing upstream, and you can reuse it for your own products as well. So does anybody have any questions? Yeah. Could you please just mention the technologies that are used for the test, like what kind of languages are the tests written out, what kind of technologies are used to be a stack? Okay. So the question is about the technologies used for testing in general. So there are many different things. So the tests that come with the kernel are normally written in C, like a self-test, or you have some shell scripts as well. And, well, the static analysis coxswinell is a separate language. And KUnit is also written in, I mean, KSAN, and these are like built-in things, so the kernel configs you enable, basically. And KUnit is a bit the same. It's part of the kernel, so it's all written in C. Now, you can have, there are also some test suites. A lot of, basically, the quick answer is a lot of it is written in C because it talks to the kernel, so it does a lot of system calls, and it makes sense to have a low level. So high-level languages that the tests are described in the declarative form or anything? Sorry to, so I can repeat. In other user space projects, we're testing using a high-level language that defines, declares the test, and then that is translated into native language. Yeah. So the question is whether there's a high-level language to define the tests and then run them. So, yeah, for, for example, the, the tests run in Lava, there's a YAML definition that explains what needs to be run. And then you'll have, you'll have some commands there that might be binaries or maybe shell scripts or maybe Python. So it depends. Each test suite has a different way of being run. Yeah. I think you were second. So for a single test run, what's the average run time and what is, what would be considered like a reasonable run time for a single run test, for test run? Okay. So the question is for one run, for one test run, what is the usual run time and what's the expected time for that, basically? What would be reasonable? Would be reasonable. Okay. So, of course, each, so the question can be answered in different ways. So if you want to test everything, then you can add, you know, time, how long it takes. Of course, some things run in parallel. So it's going to be like, until you get the whole result, it's going to be what, what takes the longest. That can also be done by splitting test suite into smaller things to run more in parallel. So there isn't a simple answer to that. But what is, what is desirable, basically, from a time duration? I think it depends on where in the process you are, where in the, can all workflow you are. So if you're sending a patch, for example, you want the result to be quick enough. If it's Linux next, because there's a version every day, you expect to have the results maybe within like six hours. That wouldn't be too bad. Or maybe 12 hours even if it's running a lot of things. If it takes more than that, then you have more kernel versions to test than at times it takes to actually run the test. If it's for the most stable release, maybe, maybe you can run more tests that will take two days and it will be valuable. So it depends on which version of the kernel at which stage you are in the development. So the context of the question is more like, if you would like to use the Docker image and put it on, say, Travis CI and have like three versions of the limit, there's like 15 minutes for a single Docker run. So basically, you have to design things to work within the limitation unless you want to go for the paid version, let's say. Okay. So the question was about integrating it in Travis CI to, as an example. Yeah. So, yeah. It's related to the question. Yeah. The idea is that if you want to run it in Travis CI as a Docker, then the limit is 50 minutes for the free version. So what you can do is for a single Docker. Yeah. So then you have to split it up in, let's say, if it's 500 minutes, but it's 10 Docker, if I didn't do 10 Docker, it's running parallel, let's say. Okay. Yeah. So it was a comment about if you wanted to integrate it in Travis CI with the limit to 50 minutes, basically you need to accommodate things around. You can also like build only one kernel version if you know, the DevConfig, the architecture you care about. And then, yeah. Does it even make sense to run to the Docker because you are not just running the kernel you built, right? I guess, so the question is about, does it make sense to run Docker? You could use Docker to build a kernel and run things that will. Yeah. Yeah. It's kind of a good point, right? No, you can use Docker to build a kernel at least. Yeah. You can build it. Anyway, yes. So how are tests being triggered? Do you need to submit them to kernel CI active? Actually, or do you just crave for requests of the mail in this? Because I guess that's the actual goal. So right now, there's a list of good branches that are monitored every hour. So the question is how can all CI tests triggered? And the answer is right now, there's a list of good branches that are being monitored. You can see them on, you know, on the website here you have trees and branches. So all these are monitored every hour. And the system knows the last version that was tested. And if there's a new version, then it triggers the whole thing and it builds a bunch of tests. So public requests to master and see maintainers trees? It's, I mean, even if one more, even if one more commit was pushed to the branch, that's a different revision at the head of the branch. And that's enough. Now, like I said, also we've proved that it can work in a GitLab CI environment where it's one, you know, a typical GitLab CI workflow and that can trigger things as well. So you can reuse the same tools in a different setting if you want to. I think maybe you were next. Yeah. And then, okay. No, we have five minutes. Five minutes. Okay. So two more questions. Yeah. One more question here. Okay. So if you want to move to testing like, how to control the main leaves, how are we going to define which test to run? Because we're obviously don't want to run them all. But what if there is a patch to like memory management breaking file system or JBL? So the question is if... Yeah, but how, right? You need to look at the patch, right? So the question is if we start testing patches, how do we determine which tests to run, basically? So first you know which mailing list it was sent to, that could be a clue. And then of course you can do some stats on the file that it's touching. And you can also... Right now we have some... There are already tools that are analyzing the patches and looking at the files that are changing. Yeah. I'm trying to figure stuff out. I guess that's what the redhead terminal testing stuff does. But yeah, so we have a YAML based configuration at the moment to... For each git branch which tests to run, we could have something like that for patches. So depending on which mailing list it was sent and maybe some people will care more about some things. Like some maintainers will say every time I push something or maybe maintainers don't send that many patches. But we could have any kind of arbitrary rules to... I think there was a question here. Yeah, I was just wondering if it's also across like arbitrary... If we're monitoring across... So we're monitoring a lot of different branches. So the question is can we... Are we monitoring any random branches? So we're monitoring the main ones of course like mainline and next and stable. And we're doing as much testing as we can on them. Then we're monitoring subsystem ones and we do a lot of testing but maybe a bit less. And then the further away you are from the mainline master branch, the less testing you get basically so it scales in that way. So you could have an individual branch from like a maintainer or individual developer. Maybe if you have like only one or two kernel builds and you do only a few tests on that, then it will... For one Linux next, if you do 200 builds, that's like 200 people doing one build basically. So that's the way it kind of scales. If the community think it's useful to have a branch, like when someone wants to have a branch, they send a request basically. If people think it's a good idea and if you have a build and test resources, then it can be added as well. The main rule is I guess that it needs to be upstream oriented. So if it's a downstream branch for your own product, that's not something that would be unknowingly normal. So the question was about generic way to talk to labs. So right now, so there's a command line tool called KCI test. So basically it's about the bit when we have a build and you go to the run box basically, when you start to test. So right now we're using... So this command line tool KCI test is in Python and there's like an abstract class with some methods like... There's one like to generate test definition and another one to submit to test definition and we could add other things like receive the results. So right now the results are sent directly to the back end. In some cases, we could like have the test labs send the result back to the place where KCI test would be used as well to receive the results and process it and store it in a database. So these are kind of the main primitive functions. And to really have it work in practice, we need to test it with more labs. Right now it's kind of only working with lava so it's still not very mature but that's basically where it's going. So I think we've run out of time. Yeah, thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you.
 Hello everyone. Thanks for waking up early and attending this talk on Saturday morning.  Can you hear me? Can you hear me at the back? Yes? Okay, cool. So there's no sound system.  This is only for recording, in case you're wondering. So my name is Guillaume Taka. I  work at Colabra and this talk is about kernel CI. So I had a talk last year already about  kernel CI. This is kind of an update on what things have happened in the past year, but  also about the future, really. So what is the purpose of kernel CI in case you're not  familiar with the project and also what it wants to achieve in the longer term? So it's  kind of climbing a big tall mountain, really, without trying to reach the point where kernels  are released without bugs and have at least quality control. So we know which bugs are  in the kernel. And extending test coverage and also having reusable tools. So the idea  is to test the upstream kernel, but anyone using the kernel in their own downstream environment  should be able to reuse these tools. We now have an official mission statement as written  here. So that's basically what I'm trying to explain here, but this is like the official  word we have now. People spend some time making this up to our thought. It deserves at least  a slide here. So, yeah, last year I said maybe it will become a Linux Foundation project  and this has happened. So in October, kernel CI has become a new project of the Linux Foundation  with all the members listed here. So Colabra, BDB, Red Hat, Google, Microsoft, FandRissel,  IO and Civil Infrastructure Platform, which is another Linux Foundation project. Now,  we're still in the early days of we know what does that actually mean for the project.  So we have a mission statement. We have a budget and some stickers, but there's a lot  of other things coming soon, hopefully. There's a website also, part of the website explaining  what that means for the project. So although it has become a Linux Foundation project,  the aim is still to be really community-oriented. So of course there are things that are done  by the members, for the members, but it's really to facilitate things to happen for the  kernel community at large. So we're still focused on upstream completely. I mean, the  tool again can be reused for other things. So if Red Hat wants to test a Fedora kernel,  they can do that. Or maybe the project will help make it happen as an example. But it's  still about testing the main one kernel, upstream kernel. And it's still about sending  email reports to mailing lists and engaging more with subsystem maintainers and developers  in general to basically help people add their own tests to kernel CI. A lot of, well, every  subsystem have different workflows. And some maintainers have created their own small CI,  which works perfectly well for them probably. But then they might be reinventing some wheel  and also not have access to all the hardware that kernel CI provides. So anyway, that's  the kind of consideration that is the main objective we're trying to achieve.  So one kind of philosophy behind kernel CI is what we call the open testing philosophy.  Because there is only one main kernel. So in the same way that people from many different  origins, from many different reasons, contribute to the same code base that Linux kernel, when  kernel 5.4 is released, that's what 5.4 is going to run on a supercomputer, on an Android  phone or on anything. It's the same kernel, the same code base, although it does a lot  of different things. So the reason why there are so many different test systems is because  of these different use cases. But we could also have a common way of at least sharing  the results, because some tests are relevant to all of these cases. If we want to test  for memory leaks in the kernel, you can do that wherever the kernel is running. So everybody  is doing their own product tests. But the idea is to have a test system that allows  people to contribute their tests and contribute their hardware in the same way that they contribute  the code. So like I said, there's a large number of contributors. So yeah, basically  that's what I explained here. The image is trying to say that we should try to have the  same surface of between tests and the code. So it's a bit like test coverage if you want,  but like aligned the two things, like the whole idea of having open source development  and having open source testing, open testing on top of it, basically. Now, how do you achieve  that? So this is like an abstract pipeline explaining how kernel CI works. For all of  these blocks, there is at least one implementation, but you could have different things for each  of them, different instances. So let's start from the beginning. So you have a developer  in the top left corner, so someone writing some code. So the code ends up as patches.  Kernel CI doesn't test patches directly at the moment, but it can be done. Other systems  do it. Or Git branches, then of course it gets built. The purple arrows are like when  files and results are pushed to, then you have build artifacts. And after that, when  you have a build, you can run tests and then process the results to detect when there was  a regression or basically see what needs to be reported from the results. And then your  results are stored and analyzed to be able to then provide a report to the people and  the developers. And this is when the loop is closed. It's like an automation when you  have, the objective is to have a working kernel. Like if you try to automate an arm,  for example, like a mechanical arm, you want to know where, in which position it's going  to be. If it's completely open and you just move something and you don't really look at  it, you can guess where it's going. But if you have some sensors that will actually know  where it is and you know the angle or everything, then you can have closed loop or automated  system. And this is kind of what we're doing here except that it's doing this around developers  and not around machines. But it's a similar kind of thing.  Sorry, can you speak louder?  Yeah, sorry. I'll speak louder. Thanks. There's a link here. So these slides are very little  as PDF. There's a link on the page for this talk. And on this slide, seven, there's also  a link to a shared document that has a more detailed description of the pipeline design,  modular pipeline. So this is like the theory. Now in practice,  what is that, how does that work? So right now we have, well, kernelCI.org, there is  one database that's being used. That's been used for a long time. But we have a new one  using Google BigQuery. And the idea is to use that as a prototyping database but with  access to more testing systems. So all the tests currently, the test results currently  available on kernelCI.org are also sent to this database. And now we have also Red Hat  submitting test results to it. And we would like to see maybe other people. So typically,  member companies would be the first ones to contribute to that. So we're kind of refining  the database schema for like storing build results and test results. And the first outcome  we would want to have is have a unified test email report. So based on all that collective  data, we could then send an email, say for, you know, a stable branch release or something.  Instead, I don't know if you've noticed on stable branch mailing lists, on stable mailing  lists, you have a lot of test results from coming from different test systems. So that's  one of the targets for having a common upstream kernel testing test system is to have only  one report that's gathering all the data. So having this database is the first step.  So if you look on the previous slide, that's basically the store. So yeah, store is artifact  and database, but this is at least the database part. Yeah, and then we can build kernels  in different places. So that's, you know, the block number two build, how do you build?  If you have different people providing different, like, cloud compute and different ways of  doing it, if you're modeling enough, then you can include them. So that makes the project  easier to scale. And right now we're using Jenkins, but we've made it, we've changed  it in a way so that it can run anywhere. So you can run all the kernel CI steps in the  terminal. And some people have tried also using it in GitLab already. So Kevin, Kevin  Hillman has done this in a, as a GitLab CI way. So he's using the kernel CI command line  tools in his, in a branch, in a kernel branch. So automatically when there's, when your  new changes are being pushed to the branch, it uses GitLab CI to build and test things  like in other GitLab CI projects. But this could be used in other test systems as well.  And also abstracting the lab API. So, you know, test labs are where hardware is available  to test kernels. So by default we use lava, which is, you know, the linear project for  automating tests. But it's not the, I mean, there are plenty of other things. So we've  made an abstraction in the kernel CI core tools so that it can submit things to any  lab. So right now it's only sending things to lava, but we can send it to any other lab  that has a web API and can accept a test definition. So these are things we're working on.  So yeah, the development goals, a bit more specific. So the command line tools are like  KCI build to do, to reproduce a kernel CI build from scratch. KCI tests are to generate  test definitions. So these are the things we're trying to improve for people so they  become more portable. Also we want to make it easier, like I said a bit before, to make  it easier for people to contribute tests. So if you're a maintainer or developer and  you have a test or maybe you work on a part of the kernel and you make some tests for  it, it should be easy for you to enable this in kernel CI. And right now this kind of a  steep learning curve, if you want to do this at the moment, you need to be, you need to  know too much about the internals of kernel CI to be able to add a test to the list. But  that's one of the things we try to like lower the bar basically. And we need some good interaction  with that. So we need some collaboration for the people who make the test as well. So we  started with like a key people to try to do that. So maybe next year I'll come back and  see. If you follow the project you'll see maybe things will become easier. But right  now it is not very easy. It's still a good thing. If you have a test you really want  it to be in kernel CI, you can send it to the mailing list or an IRC, kernel CI on  pre-node, or kernel CI groups.io mailing list. Or just on the subsystem mailing list, people  will know about kernel CI. Now, also, yeah, we're trying to improve the web dashboard.  So I don't know if you've seen the current kernel CI dashboard. But here we're showing  mostly builds and boot results. It's not really, there is a test tab, but it's not really good  at showing test results. Because kernel CI was originally about billing kernels, like  a variety of different ARM DevConfig. And then about boot testing. And it's still kind  of stuck in that phase. So we're starting running tests, but the results are not shown  properly on the web front end, on the web dashboard. So that's one thing we are fixing  actually right now to unlock a lot of things. There's many tests that we kind of held back  because maybe we could run them and then email reports have some limitations. And if the web  dashboard doesn't show all the results, then the test kind of get hidden and that's a waste  of results. But we still also need to improve the email reports. That's also a two-way process  because it's hard to design something from the beginning. It's more like sending email  reports and seeing how people react to it and whether the world is useful, what is not  useful. And different subsystems or different mailing lists need different types of reports  actually. So the key message I guess here is we need feedback. And if the email you get  is not perfect, often people say if you send another one like that, I'll just put your  filter to put all your KNLCI emails in spam or in junk. Well, that's maybe, yeah, if it  keeps getting bad all the time. But normally we try to improve things as we go. So the  key message is we can't get to perfect things from day one basically. And also a slightly  more longer term project goal is to improve test bisections. So we have bisections for  boot tests, for just plain boots at the moment. So like booting to a login, if you can log  in there, it's a boot. And that's very easy to bisect because it's only one pass-fail  result per test, per run. But as we're running more test suites that have a lot of tests,  then it becomes harder to bisect things when you have different test cases, passing or  failing at different rates. So that's, we have, I'll come back to this in another slide,  but we have some ideas on how to do this as well. So reusable tools. Yes, so we have  KCI build, KCI test, KCI root FS, and there's, to create root file systems. There's a wiki  page that might be written in a not too distant future, but right now there's a wiki page  kind of work in progress that gives you at least a starting point if you want to start,  if you want to see what these tools do. We also have the interface to the BigQuery database.  So there's a common line tool for submitting kernel CI results to the common kernel CI.  So if you have a test system and you want any testing upstream kernel, you want to submit  results, you can get in touch and use that tool and, you know, we can give you a token  and then you submit your results. Then also Docker containers. I haven't worked on it  myself recently, so I'm not an expert in the state of all of these, but I know that there's  some good progress that has been done to have containers with a lava instance with Jenkins  configured for kernel CI and the backend that's the backend currently used on kernel CI.org  with MongoDB and front end, which is here on kernel CI.org. So there are containers  to make it easy for someone to recreate the whole system in their own environment. And  again, if you try to find some issues, it's all on GitHub on the kernel CI project on  GitHub. But we need more people to use it, basically, to get some feedback on it. So  about advanced by sections. So there's this tool I worked on called Skelphal, which is  inspired by EasyBench, which was written by Martin Perez at Intel. So we've been talking  because he made this tool for graphics testing. It's actually not really used in production  EasyBench at the moment. But it had a much more advanced by section feature than the  normal Git by-sec command you have, because this can deal with several test cases at the  same time. So when you run one test, you can provide the results for many test cases. It  can also request tests to be run several times if they are not stable. So the Skelphal project  is basically extracting this and boiling it down to a more portable thing that can be  used. And if you look at the project on this GitLab project, you can find there's a demo  that will run, like create a Git history with dummy pass-fail things, and it will find a  problem in it. And it's kind of ready to be used in kernel CI, but it will come a bit  later, I think. But I hope that will really help for some things. So people say maybe  we should be testing patches as people send them, because when you have one patch on the  mailing list, you can test it on its own, and you know if it's good or bad before it  gets merged. And yes, that's ideal, but it means there's a lot of patches. So adding,  you know, supporting all of it and, you know, building, if you want to build all the kernel  conflicts we build, or even a subset of that, say if you want to build 10 kernels for each  patch that is sent and tested with 200 different platforms, and, you know, that doesn't scale  very well, I think. So maybe in some cases, if we do it wisely, we can only test a subset  based on which subsystem the patch was sent to, but it can take a lot of tweaking. So  the idea of the whole advantage of having bisection is when you test Linux Next, for  example, where all the new patches that have been applied are merged together, and if something  breaks, if something breaks at this point, then you only need maybe 10 iterations to bisect  from yesterday's Next to today's Next. So you build 10 kernels, basically, to find when  you know that there is actually a problem. So it can be very efficient. So that's why  bisection is such an important thing for kernel CI.  Okay, so tests, wish list. So these are the things that have been identified so far, but  again, maybe some people would think that network keeps failing all the time. Other  things maybe are also important to test. But these are kind of obvious things that are  directly to the ones at the top are the ones that kind of come with the kernel. And we're  not really doing them too well right now, but that's, I think that's like an obvious  list of things we should be doing. And in the Next Test Project, I know that LTP that  was mentioned a bit earlier, that's at least some parts of it, because some parts may not  be completely relevant to the kernel, maybe more like user space oriented, but there are  definitely some things in there that we, that you would expect to be run on the kernel  test system. And then some are more subsystem centric, like, you know, for media subsystem,  you have visual to compliance and other things, and IGT and XFS tests. There's a lot of them,  basically. And this is where more integration with, well, more communication with the subsystem  maintainers and developers needs to happen. That's basically where we're trying to be,  we're trying to go. There's a lot more that can be said. It's a very big project. It's  kind of old because it has been around for five years, but it's new at the same time,  because now we have a budget and we have some ideas. We have a lot of limitations have been  removed since we've joined the Linux Foundation. So I'm hoping it's like a rebirth for the  project. We need also some engagement from the people. We need people to realize that  now, you know, this has been selected by the Linux kernel community, basically, as the  main project for testing upstream, and you can reuse it for your own products as well.  So does anybody have any questions? Yeah.  Could you please just mention the technologies that are used for the test, like what kind  of languages are the tests written out, what kind of technologies are used to be a stack?  Okay. So the question is about the technologies used for testing in general. So there are  many different things. So the tests that come with the kernel are normally written in C,  like a self-test, or you have some shell scripts as well. And, well, the static analysis  coxswinell is a separate language. And KUnit is also written in, I mean, KSAN, and these  are like built-in things, so the kernel configs you enable, basically. And KUnit is a bit  the same. It's part of the kernel, so it's all written in C. Now, you can have, there  are also some test suites. A lot of, basically, the quick answer is a lot of it is written  in C because it talks to the kernel, so it does a lot of system calls, and it makes  sense to have a low level.  So high-level languages that the tests are described in the declarative form or anything?  Sorry to, so I can repeat.  In other user space projects, we're testing using a high-level language that defines,  declares the test, and then that is translated into native language.  Yeah. So the question is whether there's a high-level language to define the tests and  then run them. So, yeah, for, for example, the, the tests run in Lava, there's a YAML  definition that explains what needs to be run. And then you'll have, you'll have some  commands there that might be binaries or maybe shell scripts or maybe Python. So it depends.  Each test suite has a different way of being run. Yeah. I think you were second.  So for a single test run, what's the average run time and what is, what would be considered  like a reasonable run time for a single run test, for test run?  Okay. So the question is for one run, for one test run, what is the usual run time and  what's the expected time for that, basically?  What would be reasonable?  Would be reasonable.  Okay. So, of course, each, so the question can be answered in different ways. So if you  want to test everything, then you can add, you know, time, how long it takes. Of course,  some things run in parallel. So it's going to be like, until you get the whole result,  it's going to be what, what takes the longest. That can also be done by splitting test suite  into smaller things to run more in parallel. So there isn't a simple answer to that. But  what is, what is desirable, basically, from a time duration? I think it depends on where  in the process you are, where in the, can all workflow you are. So if you're sending a patch,  for example, you want the result to be quick enough. If it's Linux next, because there's  a version every day, you expect to have the results maybe within like six hours. That  wouldn't be too bad. Or maybe 12 hours even if it's running a lot of things. If it takes  more than that, then you have more kernel versions to test than at times it takes to  actually run the test. If it's for the most stable release, maybe, maybe you can run more  tests that will take two days and it will be valuable. So it depends on which version  of the kernel at which stage you are in the development.  So the context of the question is more like, if you would like to use the Docker image  and put it on, say, Travis CI and have like three versions of the limit, there's like  15 minutes for a single Docker run. So basically, you have to design things to work within  the limitation unless you want to go for the paid version, let's say.  Okay. So the question was about integrating it in Travis CI to, as an example.  Yeah. So, yeah. It's related to the question. Yeah. The idea is that if you want to run  it in Travis CI as a Docker, then the limit is 50 minutes for the free version. So what  you can do is for a single Docker. Yeah. So then you have to split it up in, let's say,  if it's 500 minutes, but it's 10 Docker, if I didn't do 10 Docker, it's running parallel,  let's say. Okay. Yeah. So it was a comment about if you wanted to integrate it in Travis  CI with the limit to 50 minutes, basically you need to accommodate things around. You  can also like build only one kernel version if you know, the DevConfig, the architecture  you care about. And then, yeah.  Does it even make sense to run to the Docker because you are not just running the kernel  you built, right?  I guess, so the question is about, does it make sense to run Docker? You could use Docker  to build a kernel and run things that will.  Yeah.  Yeah.  It's kind of a good point, right?  No, you can use Docker to build a kernel at least.  Yeah.  You can build it. Anyway, yes.  So how are tests being triggered? Do you need to submit them to kernel CI active?  Actually, or do you just crave for requests of the mail in this?  Because I guess that's the actual goal.  So right now, there's a list of good branches that are monitored every hour. So the question  is how can all CI tests triggered? And the answer is right now, there's a list of good  branches that are being monitored. You can see them on, you know, on the website here  you have trees and branches. So all these are monitored every hour. And the system knows  the last version that was tested. And if there's a new version, then it triggers the whole  thing and it builds a bunch of tests.  So public requests to master and see maintainers trees?  It's, I mean, even if one more, even if one more commit was pushed to the branch, that's  a different revision at the head of the branch. And that's enough. Now, like I said, also  we've proved that it can work in a GitLab CI environment where it's one, you know, a  typical GitLab CI workflow and that can trigger things as well. So you can reuse the same  tools in a different setting if you want to. I think maybe you were next.  Yeah.  And then, okay.  No, we have five minutes.  Five minutes.  Okay. So two more questions.  Yeah.  One more question here. Okay. So if you want to move to testing like, how to control the  main leaves, how are we going to define which test to run? Because we're obviously don't  want to run them all. But what if there is a patch to like memory management breaking  file system or JBL?  So the question is if...  Yeah, but how, right? You need to look at the patch, right? So the question is if we  start testing patches, how do we determine which tests to run, basically? So first you  know which mailing list it was sent to, that could be a clue. And then of course you can  do some stats on the file that it's touching. And you can also... Right now we have some...  There are already tools that are analyzing the patches and looking at the files that are  changing.  Yeah.  I'm trying to figure stuff out. I guess that's what the redhead terminal testing stuff does.  But yeah, so we have a YAML based configuration at the moment to... For each git branch which  tests to run, we could have something like that for patches. So depending on which mailing  list it was sent and maybe some people will care more about some things. Like some maintainers  will say every time I push something or maybe maintainers don't send that many patches.  But we could have any kind of arbitrary rules to... I think there was a question here.  Yeah, I was just wondering if it's also across like arbitrary...  If we're monitoring across...  So we're monitoring a lot of different branches. So the question is can we... Are we monitoring  any random branches? So we're monitoring the main ones of course like mainline and next  and stable. And we're doing as much testing as we can on them. Then we're monitoring subsystem  ones and we do a lot of testing but maybe a bit less. And then the further away you  are from the mainline master branch, the less testing you get basically so it scales in  that way. So you could have an individual branch from like a maintainer or individual developer.  Maybe if you have like only one or two kernel builds and you do only a few tests on that,  then it will... For one Linux next, if you do 200 builds, that's like 200 people doing  one build basically. So that's the way it kind of scales. If the community think it's  useful to have a branch, like when someone wants to have a branch, they send a request  basically. If people think it's a good idea and if you have a build and test resources,  then it can be added as well. The main rule is I guess that it needs to be upstream oriented.  So if it's a downstream branch for your own product, that's not something that would be  unknowingly normal. So the question was about generic way to talk to labs. So right now,  so there's a command line tool called KCI test. So basically it's about the bit when  we have a build and you go to the run box basically, when you start to test. So right  now we're using... So this command line tool KCI test is in Python and there's like an  abstract class with some methods like... There's one like to generate test definition and  another one to submit to test definition and we could add other things like receive the  results. So right now the results are sent directly to the back end. In some cases, we  could like have the test labs send the result back to the place where KCI test would be  used as well to receive the results and process it and store it in a database. So these are  kind of the main primitive functions. And to really have it work in practice, we need  to test it with more labs. Right now it's kind of only working with lava so it's still  not very mature but that's basically where it's going. So I think we've run out of time.  Yeah, thank you very much.  Thank you very much.  Thank you very much.  Thank you very much.  Thank you very much.  Thank you very much.  Thank you very much.  Thank you.
testing_welcome_kernelci_0_167.png testing_welcome_kernelci_168_875.png testing_welcome_kernelci_876_1823.png testing_welcome_kernelci_1824_2231.png testing_welcome_kernelci_2232_3347.png testing_welcome_kernelci_3348_5051.png testing_welcome_kernelci_5052_7439.png testing_welcome_kernelci_7440_10571.png testing_welcome_kernelci_10572_12635.png testing_welcome_kernelci_12636_12875.png testing_welcome_kernelci_12876_13067.png testing_welcome_kernelci_13068_13127.png testing_welcome_kernelci_13128_15467.png testing_welcome_kernelci_15468_17951.png testing_welcome_kernelci_17952_17999.png testing_welcome_kernelci_18000_18131.png testing_welcome_kernelci_18132_18179.png testing_welcome_kernelci_18180_18515.png testing_welcome_kernelci_18516_18563.png testing_welcome_kernelci_18564_21503.png testing_welcome_kernelci_21504_23735.png testing_welcome_kernelci_23736_27467.png testing_welcome_kernelci_27468_29303.png testing_welcome_kernelci_29304_31295.png testing_welcome_kernelci_31296_38411.png testing_welcome_kernelci_38412_38459.png testing_welcome_kernelci_38460_38507.png testing_welcome_kernelci_38508_38567.png testing_welcome_kernelci_38568_38675.png testing_welcome_kernelci_38676_44411.png testing_welcome_kernelci_44412_44495.png testing_welcome_kernelci_44496_44555.png testing_welcome_kernelci_44556_46079.png testing_welcome_kernelci_46080_46203.png