libliftoff status update
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Ser, Simon
This talk will explain some basics about KMS, introduce libliftoff, describe the library's current status and outline the next steps. Many DRM drivers have been exposing overlay planes for quite some time. Overlay planes can improve battery consumption by scanning out directly client buffers, skipping composition. While Wayland compositors and the X server usually take advantage of the cursor plane (and sometimes are able to use the primary plane to directly scan out a client's buffer), overlay planes are under-used. The exception is Weston, which tries to use overlay planes. Other compositors ignore them. The main challenge is to figure out how to assign buffers coming from clients to hardware planes. The only API exposed by KMS is atomic test commits, so user-space needs to try different combinations. It would be nice to have a common library shared between compositors to de-duplicate the work. During the XDC 2019 conference we discussed about libliftoff, an attempt at designing such a library. Feedback was positive from both compositor writers and driver developers. We discussed about the API, the potential pitfalls and future goals. The scope of the library has been expanded: libliftoff could also provide some feedback to clients so that they allocate buffers suitable for hardware planes. Additionally, because the KMS API makes it tricky to find the best way to make use of hardware planes, libliftoff could grow some vendor-specific plugins.
 Okay. Nice. So, hi. I'm Simon Sir, also known as Immersion, and I'm going to talk about LibLift-Tuff. So, yeah. LibLift-Tuff's goal is to take advantage of KMS planes. So, I'll first explain what is a hardware plane. Then I explain what is LibLift-Tuff and what's the current status of the library. And then we'll see what the next steps are. So, this talk will be designed for mere mortals to understand what the LibLift-Tuff is. So, if there are experts in the room, maybe it will be a little boring at the beginning. But, yeah. So first, yeah. What's a hardware plane? So, first, it has nothing to do with actual planes. So, it's hardware features in GPUs. So, before getting into hardware planes, let's just see how we get a frame and screen. How do we show something on screen? So, basically, there's a user space program that wants to show something on screen. So it has a frame. So, here's a typical screen with a terminal and a few windows. So, it talks to the kernel. It's a kernel interface called KMS. So, it submits a frame to KMS. KMS then talks to the driver. So, you have a different driver depending on the GPU vendor. So, for Intel, it's i9-15. For AMD, it's AMD GPU. And each driver will program the hardware to display the frame and screen. So, one important thing is that since, I don't know, few years, we have a new API to do this, which is called the Atomic API. So, submitting a frame is now Atomic, so you don't have a partial frame on screen. You don't have Turing and things like this. So, it's much better than before. With the legacy API, you could have corrupted frames that way. So, the client is typically called a compositor because it will take a few windows. So, a terminal window, a calculator window, a document window here, and it will draw all of these windows into a large buffer. So, it will actually perform a copy and then it will submit the final frame buffer to KMS. So, nowadays, this is performed using OpenGL. So, planes basically allow the compositor not to copy and not to use OpenGL at all, but to submit directly the client buffers to KMS. So, here, the compositor will submit three buffers in one step, in one atomic curve. With some metadata, so for instance, it says terminal window is on the top left, the calculator window is on the bottom right. So, just submit this whole state to KMS and the hardware will perform the composition directly in the scan out engine. So, you don't copy anymore. And you have an API to set properties and all of these windows, like the position. So, why do we want to use planes? So, it's zero copy as I've said. So, sometimes it's very important because some hardware needs a lot of time to perform copies, depending on the location of buffers and everything. For instance, ARM GPUs, it makes a big difference. You also get lower latency if you don't do a copy because you don't need to wait until the copy is over to submit the frame to KMS. And it also improves power consumption because when you use planes, the render engine can go to sleep. So, the part of the GPU that is used for OpenGL is not used anymore and is not using battery, only the scan out engine which is sending frames to the screen is still awake. But planes come with some downsides too. They are very hard, so pretty hard to set up and use, especially when you don't control the buffers. Especially buffers come from clients. So, right now, compositors don't really support planes. There's one exception, Western supports planes pretty well. But apart from that, everybody just always composites the whole image. So one little exception is the cursor plane. So when I move my cursor here, most compositors are able to put it in a plane, a special plane called the cursor plane. But that's the only thing compositors are able to use except Western. So one of the issues with planes is that I said planes are hard to use and that's because they come with some constraints. So for instance, here are the planes I have on my Intel machine. I have three planes. The first plane here is able to display buffers with only a few formats. So for instance, it can do CA8, RGB565 and all of this in the list. The second plane can do a different set of formats with YUV formats. And the last plane is only able to do ARGB 888. So you can see that I only have a limited number of planes with a limited number of formats supported on each plane. So for instance, let's take an example with these three windows and the three planes I have. So for instance, if this window is using ARGB, this window is XRGB, this window is XRGB, then I can put everything into planes and everything is fine. But if the first window is using XRGB, then I can do that and I can't use planes. So planes come with a large number of constraints. So we said number of planes, formats, but you can also some constraints on the buffer size, some constraints on the Z position. So which plane is over which other plane? So for instance, on my machine, the first plane would be under the second plane, which is under the last plane. And I can change that. Some hardware is able to change the Z position. There are also some bandwidth constraints. So if I use windows that are too large, I can't put them into planes. And a lot of real stuff, for instance, on my YUV plane, if I want to, on my second plane, if I want to display YUV buffer, the position must be even, I think. And if the position is odd, the position coordinates are odd, then it doesn't work. So that's Intel specific, of course. And every vendor has some vendor-specific constraints like this. So yeah, that's a mess. And the only way we know whether some combination will work on it is to perform what we call an atomic test-only commit. So we basically say, hey, I want to display this window on this plane, this other window on this other plane. And we ask the kernel, we ask KMS, will this work? And KMS says, yes or no. But we don't know why. So yeah, that's how we use planes, basically. So now we can dive into what's Leiblifthof. So the goal of Leiblifthof is to make it easier to use hardware planes. One goal is also to not abstract too much, to be as thin as possible, a thin layer of abstraction, to get into the way of the compositor. So if the compositor wants to use special hardware features, so we want to let the compositor customize a lot how it uses planes. So we had a workshop at XZC last year. And so this presentation is basically what we've done so far and what are the next steps. So the basic idea behind Leiblifthof is to expose some layers. So layers are virtual planes. They are the same as planes. You can set the position, you can set the buffer, you can set a bunch of properties in them. But they don't have any constraints. So you have as many layers as you want, and you can set any property in them. It's fine. So basically, compositors can use layers just like they would use planes. And then Leiblifthof will see which layers the compositor has set up and then try to map them into planes. So Leiblifthof performs a layer-to-plane mapping. So let's see a very basic example. So it's a pretty simple API. So for each GPU, you can create a Leiblifthof device. And then for each device, you have a bunch of outputs. You can create an output, a Leiblifthof output. And then you can create as many layers as you want. Here I create a layer. I set the frame buffer ID. I can set the position. I can set the scaling method and a bunch of stuff like this. So these are just plain, standard plane properties. Yeah, probably a real-world example will set up more layers like this. And then the compositor can just call this function, LeifthofOutputApply. And this will fill an atomic commit. We will hold the plane state. And the compositor can just perform the atomic commit. So send all the properties to the kernel and to display a new frame. And that's all. So all the compositor has to do is set up a bunch of layers. And then this is like before. Like this doesn't change. The compositor was already doing that before. So one small problem with this is that sometimes some window cannot make it into a plane. So we've seen before that if this uses XRGB, then I need to fall back to composition. But I don't need to do that for... So in this example, if this can't be put into a plane, we must fall back to composition to copy them into a large frame buffer like before with OpenGL. And if this makes it into a plane, then we don't need to copy it. So this is a mixed mode where we use planes for some windows, but not for all of them. So the way it works is that... So yeah, one important thing before we can manage this situation is that libList doesn't perform any composition. So the compositor is still responsible for using OpenGL and copy window buffers into the larger buffer. So basically, compositors will have this larger frame buffer, they need to mark it to tell libList off. This is the composition layer. So please, if you need to fall back to composition for some windows, please use this layer to indicate I need to copy... To perform some copies and this. So there's a function for this and then compositors need to... After performing the... Calling the function that performs the mapping, libList.output.apply, compositors need to check each layer. And if the layer couldn't be put into a plane, then the compositor needs to copy it with OpenGL. So the current status is that... So the layer to plane mapping works. We have some support for collision detection. So for instance, if you have two windows that don't collide, then you don't care whether a plane is on top of the other or not. You can just put the two windows into planes and you don't care that the relative position. So we have support for basic incremental updates. So if you only update the buffer property and you don't update any other property, we can reuse the previous mapping we had. We don't need to re-compute the whole thing. And one important item that we have some unit tests. I think I feel these unit tests are pretty important because it's very easy to change a bit of the algorithm and then get it wrong and it's very hard to debug. So we have a mock libdarm library which fakes some hardware planes and checks that libdifter does the right thing. So I also focused on doing some field testing. So making sure that in the real world it makes sense to do what libdifter does. So the first thing I've done is starting Glider which is an experimental compositor using libdifter. This is just to prove that the API is fine and that we've real clients, we've real windows. It works. So the goal, I'm working on WLWoods which is the Waylon compositor library and I'm working on making it ready for libdifter. So we need to add a bunch of new APIs and refactor some things. So Valve is working on a compositor called Gamescope for the SteamOS distribution and it will use libdifter as well. So that's pretty cool to have some other people trying it out. So with Glider I've performed some very, very early benchmarks. So we can see here that the compositor is the first item. This test was done by using a very simple client that just was a gradient in a 250 pixels buffer. So a pretty small buffer. And we can see that here the power estimate is like, yeah, it's pretty good. We used less power than before. So that's a good sign. So but yeah, again, we need to take a step back when reading these numbers because when falling back to when using composition there was no optimization like damage tracking or anything like this. So when doing composition the whole screen buffer was copied each time and when using planes there was absolutely no copy whatsoever. So yeah, that's pretty good. But maybe real world scenarios would be not as optimistic as this and clients will probably use a lot more power, of course. So yeah, what's next? So I have a bunch of short term goals. The first one is to perform more benchmarks. One issue is that when we leave lift off as a bunch of layers and need to try to map as many layers as possible to planes, the algorithm takes quite a long time because we need to perform a lot of atomic commits. So we need to take a layer, put it into a plane and ask the canal, will this work? If it does work, take the second layer, try to put it into a plane, ask the canal again, is this fine? So this takes quite a long time depending on the hardware configuration. So I have some ideas to try to find the best layer to plane mapping a little faster. We need to test more hardware and see how it behaves. But one important thing is that the incremental update saves a lot of time. So most of the time you only change the buffer. You don't really change the position of the windows a lot. So you just update the contents of the window. So incremental updates are very important to mitigate this also. So the goal here is to not miss a frame. We basically have a budget. If the screen refresh rate is 60 hertz, then we have 16 milliseconds to do everything, to do a new frame. So yeah, we need to be fast. We need also to better support. Oh yeah, so layer priority is an important thing. So if a window updates a lot, so for instance if you have a video player, each frame will be a new one and you will never basically reuse the previous frame. Then you really want to put it into a plane to avoid having to copy the thing. So you want to try to see which layers are updating more often than the others and put them into planes in priority. I'd also like to have better multi-output supports because right now the first output that comes will take all the planes it can take. And if you have another output, then it won't have as many planes. So we need to be fair when splitting planes across outputs. And also we need a way to migrate planes. So for instance, if I use 10 planes and one output and a new output comes in, I need to migrate planes from the first one to the other one. And it's a little bit tricky because sometimes the output and that we flashed at the same time. So for instance, if both outputs have different refresh rates, yeah, there are some synchronization issues here. I have also a bunch of long-term goals. So the first one is to have a feedback loop. So basically the idea is that right now clients allocate buffers sending to the compositor and the compositor needs to do the copy. And so the client decides what the buffer format is. So if here the client decided XRGB. But then the compositor cannot put it into a plane. So the idea of the feedback loop is to not just be sad and say, okay, I can't put the window into plane, life is terrible. So the idea is just to add a way for the compositor to say, okay, to say to the client, I can't put this format into a plane. But if you allocate using these different formats, ARGB, then life would be better. And so the client can do it. So we need a little protocol. So I'm working in Wayland. So I'm working in Wayland protocol to do this. Some kind of feedback loop to say, okay, use RGB. And the client says, okay, I'm using ARGB now. So yeah, this is the working progress. It's called the DMA buff hints. One of our long term goal is to have driver specific plugins. So as I said right now, we have only the atomic test and API to know if a plane combination will work. So the idea will be to have some driver specific plugins inside LibLift itself. So we could have logic to say, okay, I know that this won't work. So I won't even try it and things like this. And I know that the bandwidth limitation is this limit. So I won't try to go over it. So this would allow us to be more clever when allocating, when doing the plane mapping. And the last item, but yeah, it's for the future is exotic configuration. Sometimes you have planes which are under the composition layer. So you need to draw a hole into the composition plane to be able to show some planes under it. So it's very, very tricky. We'll see if it's worth it. So there are a bunch of references here if you're interested. And thank you for attention. Yeah, feel free to ask questions. Yeah, go ahead, David. So how do you know which windows to put into LibLift? Is there everything in there or just the ones that you care about, like the games and video games? So can you repeat the question? How do you know which surfaces you should put into LibLift? Do you put in every window that we have or just the ones that have a high frequency? Okay, so I'll repeat the question for people on the stream. So the question is, so this is a question from the compositor point of view. So as a compositor, do I put every window and create a LibLift layer or do I do something else? So yeah, the answer is basically you take every client window, you create a layer, and then if it works, if it's put into a plane, LibLift will tell you, okay, you can put it into a plane, and if it doesn't, then you need to fall back to composition as you were doing before. Yeah. I was wondering, is it typical for neurographic cards to have only three layers or something more? Okay, so what's the base? So the question is basically how many planes do we have generally? So my Sunday Bridge laptop is pretty old and only has three planes, so that's not a lot. I know that newer Intel hardware has a lot more planes, I think like seven planes or something, and some ARM GPUs basically don't have a limit, don't have a maximum number of planes. You can have as many planes as you want, but you have some bandwidth limits. So they expose 10 planes, and you can try to use them, but at some point it won't work anymore. So yeah, the trend is basically we'll get more on our planes as we keep going, so that's pretty good news. Yep. So the heuristics being the kernel, we are talking about doing the device plugins in user space, but in my mind it would rather be in the kernel, right? Yeah, so we've talked also about LibLift of IOCTL. So basically you set up your layers and send all your layers to the kernel, and then as the kernel will this work and do I need to composite anything on it? Yeah, there's been some talk about this. I think it's important here just to start with this, and then if it works, then we can discuss about maybe putting it into the kernel, or maybe having device-specific plugins. We'll discuss with the kernel developers what makes most sense, but for user space they could just keep using this API and it won't change, like, it's basically an implementation detail. So we can change how it works behind the scenes, and it won't matter a lot for user space. Yeah, the idea here was just to use the current API as it was designed, and then depending on the result of the experiment, will we continue using this API or not? We'll see. Yep. What about traditional Xorg? Does it use any kind of feature? So Xorg, yeah, I don't think Xorg is using hardware planes. Is it? It's old style, so all the X-video stuff. All the X-video stuff from the 90s to the 2000s. So it was implemented YUP and all the using all the things there as well. So it was able to... It did differently here. Yeah, but so right now if I have Xorg long session, will you use plane or what? Yeah, yeah, okay. So, yeah, okay, so legacy, so there's some legacy, and it was able to use planes at some point, but the new stuff is not able to. And the number of planes on all winners is 74 for the old winners. All winner devices, 74. 74, yeah. 74. Are you kidding me? So each pipeline. For each pipeline, okay, all right. I want Wibbintel starting from Ice Lake, you can have... So Ice Lake, eight planes. So eight planes per pipe, right? So per output, okay. So 42 planes total distributed across... Okay, okay, okay. Yeah, so for like, I think for most desktop applications, like eight planes is pretty good already. Not needed a lot more. So we can do like 10 more minutes of question time because Eric is a bit late. Feel free to discuss. Are you planning on handling transforms properly? So some planes sometimes can and can't scale, they can and can't rotate, they can only rotate in specific formats. Yeah. So the way you do scaling is basically setting some properties on the planes. And layers, you can set any property on it. So basically you can set the scaling properties and it will just work. It will just try the... So on each plane to use it and maybe it was, maybe not. If it doesn't work, it will ask you to fall back to composition. So it's everything... So you can set any property. So all plane features should just work, basically. But would you optimistically set every property and then it could fail or would you like roll back? So you can set a property that doesn't exist and then it will fail always. But then you don't have to be talking about which property to roll back, right? And which... So like if you have a conservative set of properties like X and Y position and Z order, for example, which you expect to work, but then you want to go more like transform, which is less likely to work and you have maybe 10 properties of which four of them think they're going to work and six of them, some subset of them might work. So basically the idea is that when you want to do a transform, if you can't do the transform, in any case, you'll fall back to composition, right? So what I'm saying is that following that decomposition, it would be a self-optimal. There's a more optimal solution that doesn't include transformers. Like maybe only one of your windows is rotated. So what's the situation of like, can you describe a like this stuff? Yeah, so you have terminal here and the calculator here and the calculator is 90 degrees or it's like a picture or something. So they can make sense to rotate. So it's rotated. You have one rotated client and one non-rotated. And ideally you can put them all in planes and then let the CPU rotate that one client for you. So you might specify with the lib lift up layer, I have this one that I want to rotate and this one I don't want to rotate. Yeah, sure. So you can extrapolate this for another five properties that are maybe some kind of exotic like this one's Y and D and this one's not. And it fails because the GPU doesn't support rotation. Okay. So the compositor expect to just peel off properties one at a time and keep trying to take off the rotation and try to get out of the L. So if you want to, if you take out the rotation, then your final image won't be the same, right? Right, but then the compositor could just rotate it in software and still use the plane for the next one. So yeah, yeah, so basically that's what happens. So if so what I said client can't be put into a plane. Because of the composition. So it will ask you to composite it, but the other client can be put into a plane and that's fine. Are there plans for multiple composition layers? So like you can composite onto a plane and then composite onto the background? Yeah, so that's more complicated. It's planned, but we'll see for the future. I think this will already be big wins. So we'll see. Yeah, having multiple composition layers. Yeah. All right. More questions? Yeah, go on please. More curiosity, but so since this is set, the criteria for allocating the layers for planes are very dependent on a lot of things as I understand. Yep. I expected it might very well be that you have to frequently reschedule things. If you move windows and for example one coordinate that was even becomes odd. Yeah. And I don't know what happens. So you probably have to change your allocation. And maybe at some points you didn't expect, at some point you could avoid composition and immediately you have to start using composition. Yeah. I expected this could be a problem if your composition flow is cold caches. It's not being used recently. It could be quite slower the first time you use it. Because you need to like boot up the engine. Yeah, I don't know if it takes time to, we need to do more experiments. Yeah, I'm not sure. I need to ask the drive device, the cannot driver's case to see if that would be an issue. Yeah. You basically don't want to use discrete GPUs for composition anyway. Because if you unplug it, then you don't have anything. You don't have to be external with this. Yeah. There's a lot of complicated issues with the layer to plane mapping algorithm. It's pretty annoying too. Yeah. Can I ask you how it will be like this in guessing in advance? Like if the client, the user moves their mask or sort of over the panel bar, it looks like they're getting ready to drag. You might want to boot up that render. Yeah. Yeah. Maybe the compositor could just fall back to the full composition whenever something is moving. Yeah, so in any case, if something took Harry, he's going on, then it will fall back to composition anyway. So yeah. Yeah, I'm not a fan of like letting the compositor say, okay, the guy is moving the mouse and other things. So I'm going to do some heuristics too. Yeah. We need to do some experiments to see if it's well fit or not. Yeah, sure. If there are any kind of mechanisms, can you show like how is it exposing the API regarding either partial updates like when the window doesn't need to update a lot of it or when it's not clear or things like that? So partial updates and windows being created behind other windows. So partial updates should. So for planes, you don't really need to say to the GPU whether only a region are actually there is a property for this. Yes. Some of them have own buffers on the panel itself. Yeah. Part of the transfers. Actually, you can set up a layer and say, I only updated this region of the layer. And then some hardware will be able to make use of this. So the compositor without liblystuff already does damage tracking. So it keeps track of which parts of the screen have been updated since last frame. And then only repaint the parts that need to be repainted. So it could just like set the property and the layer and it would be fine. And about so right now I don't have any logic to not show layers that are completely occluded behind other layers. But it's a little difficult sometimes because you if you have transparency, for instance, you still want to like do as a layer even if it's completely occluded behind. So the composite so the compositor will have more hints. Some clients can say my window is completely opaque. So it's fine if you don't do anything under it. So compositors are more knowledge to do this and should already do this for OpenGL. I'll see if, yeah, I don't know if it's worth it to add this to the blifter. Maybe in the future we'll see. Depending the buffer format, if it's XRGB, for instance, we know it won't have an alpha channel. So we can say it's opaque and we won't do anything under it. Yeah, maybe we'll see. Okay, yeah, sure.
 Okay.  Nice.  So, hi.  I'm Simon Sir, also known as Immersion, and I'm going to talk about LibLift-Tuff.  So, yeah.  LibLift-Tuff's goal is to take advantage of KMS planes.  So, I'll first explain what is a hardware plane.  Then I explain what is LibLift-Tuff and what's the current status of the library.  And then we'll see what the next steps are.  So, this talk will be designed for mere mortals to understand what the LibLift-Tuff is.  So, if there are experts in the room, maybe it will be a little boring at the beginning.  But, yeah.  So first, yeah.  What's a hardware plane?  So, first, it has nothing to do with actual planes.  So, it's hardware features in GPUs.  So, before getting into hardware planes, let's just see how we get a frame and screen.  How do we show something on screen?  So, basically, there's a user space program that wants to show something on screen.  So it has a frame.  So, here's a typical screen with a terminal and a few windows.  So, it talks to the kernel.  It's a kernel interface called KMS.  So, it submits a frame to KMS.  KMS then talks to the driver.  So, you have a different driver depending on the GPU vendor.  So, for Intel, it's i9-15.  For AMD, it's AMD GPU.  And each driver will program the hardware to display the frame and screen.  So, one important thing is that since, I don't know, few years, we have a new API to do this,  which is called the Atomic API.  So, submitting a frame is now Atomic, so you don't have a partial frame on screen.  You don't have Turing and things like this.  So, it's much better than before.  With the legacy API, you could have corrupted frames that way.  So, the client is typically called a compositor because it will take a few windows.  So, a terminal window, a calculator window, a document window here,  and it will draw all of these windows into a large buffer.  So, it will actually perform a copy and then it will submit the final frame buffer to KMS.  So, nowadays, this is performed using OpenGL.  So, planes basically allow the compositor not to copy and not to use OpenGL at all,  but to submit directly the client buffers to KMS.  So, here, the compositor will submit three buffers in one step, in one atomic curve.  With some metadata, so for instance, it says terminal window is on the top left,  the calculator window is on the bottom right.  So, just submit this whole state to KMS and the hardware will perform the composition  directly in the scan out engine.  So, you don't copy anymore.  And you have an API to set properties and all of these windows, like the position.  So, why do we want to use planes?  So, it's zero copy as I've said.  So, sometimes it's very important because some hardware needs a lot of time to perform copies,  depending on the location of buffers and everything.  For instance, ARM GPUs, it makes a big difference.  You also get lower latency if you don't do a copy because you don't need to wait until  the copy is over to submit the frame to KMS.  And it also improves power consumption because when you use planes, the render engine can  go to sleep.  So, the part of the GPU that is used for OpenGL is not used anymore and is not using  battery, only the scan out engine which is sending frames to the screen is still awake.  But planes come with some downsides too.  They are very hard, so pretty hard to set up and use, especially when you don't control  the buffers.  Especially buffers come from clients.  So, right now, compositors don't really support planes.  There's one exception, Western supports planes pretty well.  But apart from that, everybody just always composites the whole image.  So one little exception is the cursor plane.  So when I move my cursor here, most compositors are able to put it in a plane, a special plane  called the cursor plane.  But that's the only thing compositors are able to use except Western.  So one of the issues with planes is that I said planes are hard to use and that's because  they come with some constraints.  So for instance, here are the planes I have on my Intel machine.  I have three planes.  The first plane here is able to display buffers with only a few formats.  So for instance, it can do CA8, RGB565 and all of this in the list.  The second plane can do a different set of formats with YUV formats.  And the last plane is only able to do ARGB 888.  So you can see that I only have a limited number of planes with a limited number of formats  supported on each plane.  So for instance, let's take an example with these three windows and the three planes I  have.  So for instance, if this window is using ARGB, this window is XRGB, this window is XRGB,  then I can put everything into planes and everything is fine.  But if the first window is using XRGB, then I can do that and I can't use planes.  So planes come with a large number of constraints.  So we said number of planes, formats, but you can also some constraints on the buffer  size, some constraints on the Z position.  So which plane is over which other plane?  So for instance, on my machine, the first plane would be under the second plane, which  is under the last plane.  And I can change that.  Some hardware is able to change the Z position.  There are also some bandwidth constraints.  So if I use windows that are too large, I can't put them into planes.  And a lot of real stuff, for instance, on my YUV plane, if I want to, on my second plane,  if I want to display YUV buffer, the position must be even, I think.  And if the position is odd, the position coordinates are odd, then it doesn't work.  So that's Intel specific, of course.  And every vendor has some vendor-specific constraints like this.  So yeah, that's a mess.  And the only way we know whether some combination will work on it is to perform what we call  an atomic test-only commit.  So we basically say, hey, I want to display this window on this plane, this other window  on this other plane.  And we ask the kernel, we ask KMS, will this work?  And KMS says, yes or no.  But we don't know why.  So yeah, that's how we use planes, basically.  So now we can dive into what's Leiblifthof.  So the goal of Leiblifthof is to make it easier to use hardware planes.  One goal is also to not abstract too much, to be as thin as possible, a thin layer of  abstraction, to get into the way of the compositor.  So if the compositor wants to use special hardware features, so we want to let the compositor  customize a lot how it uses planes.  So we had a workshop at XZC last year.  And so this presentation is basically what we've done so far and what are the next steps.  So the basic idea behind Leiblifthof is to expose some layers.  So layers are virtual planes.  They are the same as planes.  You can set the position, you can set the buffer, you can set a bunch of properties  in them.  But they don't have any constraints.  So you have as many layers as you want, and you can set any property in them.  It's fine.  So basically, compositors can use layers just like they would use planes.  And then Leiblifthof will see which layers the compositor has set up and then try to map  them into planes.  So Leiblifthof performs a layer-to-plane mapping.  So let's see a very basic example.  So it's a pretty simple API.  So for each GPU, you can create a Leiblifthof device.  And then for each device, you have a bunch of outputs.  You can create an output, a Leiblifthof output.  And then you can create as many layers as you want.  Here I create a layer.  I set the frame buffer ID.  I can set the position.  I can set the scaling method and a bunch of stuff like this.  So these are just plain, standard plane properties.  Yeah, probably a real-world example will set up more layers like this.  And then the compositor can just call this function, LeifthofOutputApply.  And this will fill an atomic commit.  We will hold the plane state.  And the compositor can just perform the atomic commit.  So send all the properties to the kernel and to display a new frame.  And that's all.  So all the compositor has to do is set up a bunch of layers.  And then this is like before.  Like this doesn't change.  The compositor was already doing that before.  So one small problem with this is that sometimes some window cannot make it into a plane.  So we've seen before that if this uses XRGB, then I need to fall back to composition.  But I don't need to do that for...  So in this example, if this can't be put into a plane, we must fall back to composition  to copy them into a large frame buffer like before with OpenGL.  And if this makes it into a plane, then we don't need to copy it.  So this is a mixed mode where we use planes for some windows, but not for all of them.  So the way it works is that...  So yeah, one important thing before we can manage this situation is that libList doesn't  perform any composition.  So the compositor is still responsible for using OpenGL and copy window buffers into  the larger buffer.  So basically, compositors will have this larger frame buffer, they need to mark it to tell  libList off.  This is the composition layer.  So please, if you need to fall back to composition for some windows, please use this layer to  indicate I need to copy...  To perform some copies and this.  So there's a function for this and then compositors need to...  After performing the...  Calling the function that performs the mapping, libList.output.apply, compositors need to check  each layer.  And if the layer couldn't be put into a plane, then the compositor needs to copy it with  OpenGL.  So the current status is that...  So the layer to plane mapping works.  We have some support for collision detection.  So for instance, if you have two windows that don't collide, then you don't care whether  a plane is on top of the other or not.  You can just put the two windows into planes and you don't care that the relative position.  So we have support for basic incremental updates.  So if you only update the buffer property and you don't update any other property, we  can reuse the previous mapping we had.  We don't need to re-compute the whole thing.  And one important item that we have some unit tests.  I think I feel these unit tests are pretty important because it's very easy to change  a bit of the algorithm and then get it wrong and it's very hard to debug.  So we have a mock libdarm library which fakes some hardware planes and checks that libdifter  does the right thing.  So I also focused on doing some field testing.  So making sure that in the real world it makes sense to do what libdifter does.  So the first thing I've done is starting Glider which is an experimental compositor using  libdifter.  This is just to prove that the API is fine and that we've real clients, we've real windows.  It works.  So the goal, I'm working on WLWoods which is the Waylon compositor library and I'm working  on making it ready for libdifter.  So we need to add a bunch of new APIs and refactor some things.  So Valve is working on a compositor called Gamescope for the SteamOS distribution and  it will use libdifter as well.  So that's pretty cool to have some other people trying it out.  So with Glider I've performed some very, very early benchmarks.  So we can see here that the compositor is the first item.  This test was done by using a very simple client that just was a gradient in a 250  pixels buffer.  So a pretty small buffer.  And we can see that here the power estimate is like, yeah, it's pretty good.  We used less power than before.  So that's a good sign.  So but yeah, again, we need to take a step back when reading these numbers because when  falling back to when using composition there was no optimization like damage tracking or  anything like this.  So when doing composition the whole screen buffer was copied each time and when using  planes there was absolutely no copy whatsoever.  So yeah, that's pretty good.  But maybe real world scenarios would be not as optimistic as this and clients will probably  use a lot more power, of course.  So yeah, what's next?  So I have a bunch of short term goals.  The first one is to perform more benchmarks.  One issue is that when we leave lift off as a bunch of layers and need to try to map as  many layers as possible to planes, the algorithm takes quite a long time because we need to  perform a lot of atomic commits.  So we need to take a layer, put it into a plane and ask the canal, will this work?  If it does work, take the second layer, try to put it into a plane, ask the canal again,  is this fine?  So this takes quite a long time depending on the hardware configuration.  So I have some ideas to try to find the best layer to plane mapping a little faster.  We need to test more hardware and see how it behaves.  But one important thing is that the incremental update saves a lot of time.  So most of the time you only change the buffer.  You don't really change the position of the windows a lot.  So you just update the contents of the window.  So incremental updates are very important to mitigate this also.  So the goal here is to not miss a frame.  We basically have a budget.  If the screen refresh rate is 60 hertz, then we have 16 milliseconds to do everything,  to do a new frame.  So yeah, we need to be fast.  We need also to better support.  Oh yeah, so layer priority is an important thing.  So if a window updates a lot, so for instance if you have a video player, each frame will  be a new one and you will never basically reuse the previous frame.  Then you really want to put it into a plane to avoid having to copy the thing.  So you want to try to see which layers are updating more often than the others and put  them into planes in priority.  I'd also like to have better multi-output supports because right now the first output  that comes will take all the planes it can take.  And if you have another output, then it won't have as many planes.  So we need to be fair when splitting planes across outputs.  And also we need a way to migrate planes.  So for instance, if I use 10 planes and one output and a new output comes in, I need to  migrate planes from the first one to the other one.  And it's a little bit tricky because sometimes the output and that we flashed at the same  time.  So for instance, if both outputs have different refresh rates, yeah, there are some synchronization  issues here.  I have also a bunch of long-term goals.  So the first one is to have a feedback loop.  So basically the idea is that right now clients allocate buffers sending to the compositor  and the compositor needs to do the copy.  And so the client decides what the buffer format is.  So if here the client decided XRGB.  But then the compositor cannot put it into a plane.  So the idea of the feedback loop is to not just be sad and say, okay, I can't put the  window into plane, life is terrible.  So the idea is just to add a way for the compositor to say, okay, to say to the client, I can't  put this format into a plane.  But if you allocate using these different formats, ARGB, then life would be better.  And so the client can do it.  So we need a little protocol.  So I'm working in Wayland.  So I'm working in Wayland protocol to do this.  Some kind of feedback loop to say, okay, use RGB.  And the client says, okay, I'm using ARGB now.  So yeah, this is the working progress.  It's called the DMA buff hints.  One of our long term goal is to have driver specific plugins.  So as I said right now, we have only the atomic test and API to know if a plane combination  will work.  So the idea will be to have some driver specific plugins inside LibLift itself.  So we could have logic to say, okay, I know that this won't work.  So I won't even try it and things like this.  And I know that the bandwidth limitation is this limit.  So I won't try to go over it.  So this would allow us to be more clever when allocating, when doing the plane mapping.  And the last item, but yeah, it's for the future is exotic configuration.  Sometimes you have planes which are under the composition layer.  So you need to draw a hole into the composition plane to be able to show some planes under  it.  So it's very, very tricky.  We'll see if it's worth it.  So there are a bunch of references here if you're interested.  And thank you for attention.  Yeah, feel free to ask questions.  Yeah, go ahead, David.  So how do you know which windows to put into LibLift?  Is there everything in there or just the ones that you care about, like the games and  video games?  So can you repeat the question?  How do you know which surfaces you should put into LibLift?  Do you put in every window that we have or just the ones that have a high frequency?  Okay, so I'll repeat the question for people on the stream.  So the question is, so this is a question from the compositor point of view.  So as a compositor, do I put every window and create a LibLift layer or do I do something  else?  So yeah, the answer is basically you take every client window, you create a layer, and  then if it works, if it's put into a plane, LibLift will tell you, okay, you can put it  into a plane, and if it doesn't, then you need to fall back to composition as you were  doing before.  Yeah.  I was wondering, is it typical for neurographic cards to have only three layers or something  more?  Okay, so what's the base?  So the question is basically how many planes do we have generally?  So my Sunday Bridge laptop is pretty old and only has three planes, so that's not a lot.  I know that newer Intel hardware has a lot more planes, I think like seven planes or  something, and some ARM GPUs basically don't have a limit, don't have a maximum number  of planes.  You can have as many planes as you want, but you have some bandwidth limits.  So they expose 10 planes, and you can try to use them, but at some point it won't work  anymore.  So yeah, the trend is basically we'll get more on our planes as we keep going, so that's  pretty good news.  Yep.  So the heuristics being the kernel, we are talking about doing the device plugins in  user space, but in my mind it would rather be in the kernel, right?  Yeah, so we've talked also about LibLift of IOCTL.  So basically you set up your layers and send all your layers to the kernel, and then as  the kernel will this work and do I need to composite anything on it?  Yeah, there's been some talk about this.  I think it's important here just to start with this, and then if it works, then we can  discuss about maybe putting it into the kernel, or maybe having device-specific plugins.  We'll discuss with the kernel developers what makes most sense, but for user space they  could just keep using this API and it won't change, like, it's basically an implementation  detail.  So we can change how it works behind the scenes, and it won't matter a lot for user space.  Yeah, the idea here was just to use the current API as it was designed, and then depending  on the result of the experiment, will we continue using this API or not?  We'll see.  Yep.  What about traditional Xorg?  Does it use any kind of feature?  So Xorg, yeah, I don't think Xorg is using hardware planes.  Is it?  It's old style, so all the X-video stuff.  All the X-video stuff from the 90s to the 2000s.  So it was implemented YUP and all the using all the things there as well.  So it was able to...  It did differently here.  Yeah, but so right now if I have Xorg long session, will you use plane or what?  Yeah, yeah, okay.  So, yeah, okay, so legacy, so there's some legacy, and it was able to use planes at some  point, but the new stuff is not able to.  And the number of planes on all winners is 74 for the old winners.  All winner devices, 74.  74, yeah.  74.  Are you kidding me?  So each pipeline.  For each pipeline, okay, all right.  I want Wibbintel starting from Ice Lake, you can have...  So Ice Lake, eight planes.  So eight planes per pipe, right?  So per output, okay.  So 42 planes total distributed across...  Okay, okay, okay.  Yeah, so for like, I think for most desktop applications, like eight planes is pretty  good already.  Not needed a lot more.  So we can do like 10 more minutes of question time because Eric is a bit late.  Feel free to discuss.  Are you planning on handling transforms properly?  So some planes sometimes can and can't scale, they can and can't rotate, they can only rotate  in specific formats.  Yeah.  So the way you do scaling is basically setting some properties on the planes.  And layers, you can set any property on it.  So basically you can set the scaling properties and it will just work.  It will just try the...  So on each plane to use it and maybe it was, maybe not.  If it doesn't work, it will ask you to fall back to composition.  So it's everything...  So you can set any property.  So all plane features should just work, basically.  But would you optimistically set every property and then it could fail or would you like roll  back?  So you can set a property that doesn't exist and then it will fail always.  But then you don't have to be talking about which property to roll back, right?  And which...  So like if you have a conservative set of properties like X and Y position and Z order,  for example, which you expect to work, but then you want to go more like transform, which  is less likely to work and you have maybe 10 properties of which four of them think  they're going to work and six of them, some subset of them might work.  So basically the idea is that when you want to do a transform, if you can't do the transform,  in any case, you'll fall back to composition, right?  So what I'm saying is that following that decomposition, it would be a self-optimal.  There's a more optimal solution that doesn't include transformers.  Like maybe only one of your windows is rotated.  So what's the situation of like, can you describe a like this stuff?  Yeah, so you have terminal here and the calculator here and the calculator is 90 degrees or  it's like a picture or something.  So they can make sense to rotate.  So it's rotated.  You have one rotated client and one non-rotated.  And ideally you can put them all in planes and then let the CPU rotate that one client  for you.  So you might specify with the lib lift up layer, I have this one that I want to rotate and this  one I don't want to rotate.  Yeah, sure.  So you can extrapolate this for another five properties that are maybe some kind of exotic  like this one's Y and D and this one's not.  And it fails because the GPU doesn't support rotation.  Okay.  So the compositor expect to just peel off properties one at a time and keep trying to  take off the rotation and try to get out of the L.  So if you want to, if you take out the rotation, then your final image won't be the same, right?  Right, but then the compositor could just rotate it in software and still use the plane for  the next one.  So yeah, yeah, so basically that's what happens.  So if so what I said client can't be put into a plane.  Because of the composition.  So it will ask you to composite it, but the other client can be put into a plane and that's  fine.  Are there plans for multiple composition layers?  So like you can composite onto a plane and then composite onto the background?  Yeah, so that's more complicated.  It's planned, but we'll see for the future.  I think this will already be big wins.  So we'll see.  Yeah, having multiple composition layers.  Yeah.  All right.  More questions?  Yeah, go on please.  More curiosity, but so since this is set, the criteria for allocating the layers for planes  are very dependent on a lot of things as I understand.  Yep.  I expected it might very well be that you have to frequently reschedule things.  If you move windows and for example one coordinate that was even becomes odd.  Yeah.  And I don't know what happens.  So you probably have to change your allocation.  And maybe at some points you didn't expect, at some point you could avoid composition  and immediately you have to start using composition.  Yeah.  I expected this could be a problem if your composition flow is cold caches.  It's not being used recently.  It could be quite slower the first time you use it.  Because you need to like boot up the engine.  Yeah, I don't know if it takes time to, we need to do more experiments.  Yeah, I'm not sure.  I need to ask the drive device, the cannot driver's case to see if that would be an issue.  Yeah.  You basically don't want to use discrete GPUs for composition anyway.  Because if you unplug it, then you don't have anything.  You don't have to be external with this.  Yeah.  There's a lot of complicated issues with the layer to plane mapping algorithm.  It's pretty annoying too.  Yeah.  Can I ask you how it will be like this in guessing in advance?  Like if the client, the user moves their mask or sort of over the panel bar, it looks like  they're getting ready to drag.  You might want to boot up that render.  Yeah.  Yeah.  Maybe the compositor could just fall back to the full composition whenever something is moving.  Yeah, so in any case, if something took Harry, he's going on, then it will fall back to composition anyway.  So yeah.  Yeah, I'm not a fan of like letting the compositor say, okay, the guy is moving the mouse and  other things.  So I'm going to do some heuristics too.  Yeah.  We need to do some experiments to see if it's well fit or not.  Yeah, sure.  If there are any kind of mechanisms, can you show like how is it exposing the API regarding  either partial updates like when the window doesn't need to update a lot of it or when it's  not clear or things like that?  So partial updates and windows being created behind other windows.  So partial updates should.  So for planes, you don't really need to say to the GPU whether only a region are actually  there is a property for this.  Yes.  Some of them have own buffers on the panel itself.  Yeah.  Part of the transfers.  Actually, you can set up a layer and say, I only updated this region of the layer.  And then some hardware will be able to make use of this.  So the compositor without liblystuff already does damage tracking.  So it keeps track of which parts of the screen have been updated since last frame.  And then only repaint the parts that need to be repainted.  So it could just like set the property and the layer and it would be fine.  And about so right now I don't have any logic to not show layers that are completely occluded  behind other layers.  But it's a little difficult sometimes because you if you have transparency, for instance,  you still want to like do as a layer even if it's completely occluded behind.  So the composite so the compositor will have more hints.  Some clients can say my window is completely opaque.  So it's fine if you don't do anything under it.  So compositors are more knowledge to do this and should already do this for OpenGL.  I'll see if, yeah, I don't know if it's worth it to add this to the blifter.  Maybe in the future we'll see.  Depending the buffer format, if it's XRGB, for instance, we know it won't have an alpha channel.  So we can say it's opaque and we won't do anything under it.  Yeah, maybe we'll see.  Okay, yeah, sure.
kms_planes_0_155.png kms_planes_156_1511.png kms_planes_1512_2051.png kms_planes_2052_4103.png kms_planes_4104_4979.png kms_planes_4980_6419.png kms_planes_6420_8075.png kms_planes_8076_9095.png kms_planes_9096_9143.png kms_planes_9144_9371.png kms_planes_9372_9563.png kms_planes_9564_10823.png kms_planes_10824_11519.png kms_planes_11520_13331.png kms_planes_13332_14159.png kms_planes_14160_15467.png kms_planes_15468_16895.png kms_planes_16896_19223.png kms_planes_19224_20507.png kms_planes_20508_22475.png kms_planes_22476_24395.png kms_planes_24396_26351.png kms_planes_26352_28415.png kms_planes_28416_28559.png kms_planes_28560_28751.png kms_planes_28752_33911.png kms_planes_33912_33959.png kms_planes_33960_34151.png kms_planes_34152_34511.png kms_planes_34512_35315.png kms_planes_35316_35963.png kms_planes_35964_37127.png kms_planes_37128_37727.png kms_planes_37728_37775.png kms_planes_37776_37823.png kms_planes_37824_37895.png kms_planes_37896_37955.png kms_planes_37956_38675.png kms_planes_38676_39143.png kms_planes_39144_39359.png kms_planes_39360_43019.png kms_planes_43020_44915.png kms_planes_44916_45323.png kms_planes_45324_46187.png kms_planes_46188_49403.png kms_planes_49404_59039.png kms_planes_59040_59123.png kms_planes_59124_59291.png kms_planes_59292_59351.png kms_planes_59352_59399.png kms_planes_59400_59517.png