WPE, The WebKit port for Embedded platforms
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Normand, Philippe
WPEWebKit is a WebKit flavour (also known as port) specially crafted for embedded platforms and use-cases. During this talk I would present WPEWebKit's architecture with a special emphasis on its multimedia backend based on GStreamer. I would also demonstrate various use-cases for WPE, spanning from Kiosk apps and Set-top-box user-interfaces to advanced scenarios such as Web overlays for live TV broadcasting. WPEWebKit is designed for simplicity and performance. It allows application developers to easily deploy hardware-accelerated fullscreen (or not) browsers with multimedia support, small (both in memory usage and disk space) and light as possible, and implementing the most relevant HTML specifications. Traditionally WebKit ports are associated with a specific widget toolkit library (GTK, Qt, Cocoa,...) but WPEWebKit breaks with this monolithic design and thus enables a new range of use-cases. By delegating the final web page rendering to dedicated view-backends, WPEWebKit allows flexible and tight integration in a wide range of hardware platforms. We also provide a Qt5 QML plugin that can easily replace the deprecated QtWebKit-based module. WPEWebKit leverages GStreamer for its multiple multimedia backends, ensuring your WPEWebKit-based browser supports zero-copy hardware video decoding on the most common embedded platforms such as the Raspberry Pi, i.MX6 and i.MX8M SoCS. WPEWebKit can also be used in pure GStreamer applications! Thanks to the GstWPE plugin, web-pages can be "injected" in GStreamer pipelines as audio and video streams. This new plugin thus enables use-cases such has HTML overlays. WPEWebKit is an open source project with a growing community, and it is developed within the ecosystem of the WebKit project, which powers many open source and proprietary web browsers.
 All right. So I guess started them. Yeah. Good afternoon, everybody. I have this presentation about WPE, which is a web engine based on WebKit and designed for embedded platforms. So during the talk, I give a short introduction about WebKit, a very short overview. And then I will more focus on applications. How you can use WPE in your application, what are the use cases for it. And probably in five time, I'll give a short demo about something I wrote. All right. So I'm Philippe. I'm WebKit reviewer, a digital committer. I work for a company based in Spain called Igalia. We are working on a cooperative. We are currently about 90 IP Igalians across the world. So it's based on WebKit, which probably, as you know, was a project started by Apple as a fork of KHTML around 2004. And the main goal for them at the time was to build Safari. And since then, other people joined the projects, other companies. The community added new ports. I will talk about that later. So I briefly introduced the WebKit to multi-process architecture. On that diagram, you can only see two processes, but nowadays there are more. There's one about network nowadays, one about storage. And this recently introduced GPU process as well. So the application is called the UI process in that architecture. And it communicates with other processes using IPC. And all the DOM, passing all the things internal to the web engine are done in those processes, like JavaScript and all those things. So WP is a port aimed for embedded platforms. Traditionally, WebKit ports are built on top of a toolkit widget. But this port is a bit special in the sense that it decouples rendering to dedicated what we call rendering backends. That means that it gives more flexibility to the application developers. And there's more openness about the platforms that you can support using that model. So that means that the rendering is deferred to another part of the port that is loaded at runtime, basically. And the input events are also under using that model. So those rendering backends usually require a thing called Weleon EGL, which is a way for applications to share graphics resources between processes. And they have EGL extensions dedicated to that. So there are quite a few backends nowadays. I will just focus on two. The first one is the FDO backend, which we'll talk about in a minute. And the other one is the RDK backend, which is developed by companies such as Comcast. They have deployed it in a wide range of set of boxes, actually, in the world. So the FDO backend, it depends on any GL. So that means usually if you have a GPU driver that's using MSA, or probably something that provides EGL, you can have binary drivers. So that backend provides a high-level API for applications to be able to get EGL images from the web engine so that they can do whatever they want with that. And then that's also what we recommend for the WP community. It's the backend used on the build boards, and it's the backend we use the most in the community. Then obviously you need a browser or some kind of application. So at EGL, we are working on a minimalistic browser called Cog. It depends on the WP backend FDO, and it's minimalistic. There's actually no window decresion or nothing. And single-webview, although we are working on multi-webview support, and it also can be controlled using D-Bus. So in your application, you can remote control the browser using other applications. So as a basic, you need to have a Wayland compositor to be able to run Cog. But since recently, we wrote a new backend that leverages the DRM architecture, and thus we don't really require a running Wayland compositor. So that's quite cool because it reduces the dependencies that you need for your application, and if you need a full-screen application, you can use that backend. Like kiosks, set-top boxes, UI, any full-screen display. And then the rendering is done with DRM. So the Wayland buffers are imported as GB and buffer objects, and then rendered using DRM. I have a small showcase for that. It's basically a thing I did in my free time. I was a bit tired of using Codiatom, so I wanted to have a minimalistic media center self-contained. So I built a web extension for WP. And that can be loaded at runtime by the web process. And in that web extension, I use a UPNP library called libg-upnp. So I use that library to discover the media servers on the network. And then for each server, I modify the DOM and add these elements to the HTML page rendered. So then I make a little build-up web page based on what's on the network, basically. And then I can do the video playback with the normal video element that's stand-down nowadays in WebKit. So there's a demo. It's really subsequent. If you want to look at it on your phone, you can scan that code. Otherwise, I can just move on. But yeah, it's an interesting showcase of what you can do with Cog. And it gives an overview of really what you can do. Another thing I wanted to discuss about is acute applications. As you probably know, acute is used widely used in the industry nowadays. And if you have a web engine in your application such as Qt WebKit or Qt Web Engine nowadays, you probably would like to have an alternative because Qt WebKit is not maintained anymore. It has a lot of security issues. And some people don't like Qt Web Engine because it's too big. Yeah. So this is an alternative to those two options you have. So what we did in upstream WebKit, we've built a QML module that can be swapped in instead of the Qt WebKit-based QML module. So if you have an application using Qt WebKit, you can directly use that module. And it will use internal AWP instead of Qt WebKit. So what you gain is maintain web engine with security releases. And so in that sense, it's quite a good advantage. There are some drawbacks, though. It works only on Linux. So if your application needs to run on Windows, you're a bit screwed. And then there's dependency on Web and EGL that we can't really work around. It works currently in the EGLFS, QPA, and WLAN EGL as well. So that means you can work on desktop and on embedded platforms. So to enable it, you just need to download Qt WebKit. WP WebKit, sorry. And turn on that CMake option called enable WPQTAPI. And then around time, you just need to make sure that the SO file is in your import path. And then I have a small code snippet, QML snippet there that shows how to basically use that WP view in your application. So the changes compared to Qt WebKit is that you just need to change the import line. And the module name is WP view. But in the Qt WebKit world, I think it's a different name. Otherwise, the API is similar. So on title change here that is already available in Qt WebKit can be used as it is. All right. And then the last thing I wanted to talk about is how to use WP in multimedia application using Gstreamer. That's a bit of an over way. Think about streaming browsers or HTML overlays in live TV, for instance. So that could be a use case for WP. Just a brief overview of Gstreamer. Who hasn't heard about Gstreamer? Nobody? Yes? Okay. So basically it's a multimedia framework based on graphs. So in your application, you build a graph and the data flow will flow from the source to the sync and there will be data processing like decoding and rendering. There's many, many plugins. I won't go into details. But what I did is write a new plugin that uses WP and I built a source element for that so that the video, the web view can be outputted as a video stream. And it's quite cool because we have zero copy from WebKit world into Gstreamer world using the ChargedGL context. So there's no memory copies in there. The use cases as I said, HTML overlays and streaming browsers. I started, so that's a demo I wrote. On the right side, you can see some HTML and on the left side, the preview of it with a live video behind. And then you can stream that. I can show you, show it later. And when you update the HTML, it's updated in the video, of course. So that's for the TV broadcasting world. It could be useful, for instance. Some ongoing work I've been doing lately on that plugin, adding audio support. I've reported type already. I need to upstream it. And then navigation events in Gstreamer are not really, have been designed many years ago and it's to be modernized a bit. So I started working on that as well. Maybe adding mouse core support to achievements and improving keyboard support as well. So the code is in WPWadkid.org. We have a Yokto layer. The support is upstream in ViewTort as well. This has been working quite well on AMX6, AMX8M platform, Raspberry Pi 3 and 4. And it's deployed in the wild already. So you can use it now. I guess I have time for a demo. Okay. If it works. Okay. Okay, let's try this again. Okay, it doesn't work. But you can install it with a flat pack available and you can install it quite easily on your desktop. I can provide the flat pack ref later on if needed. But I don't have it installed here. Sorry. No, it's not working. Anyway, so yeah, that's it. If you have any questions, I will be happy to answer. Yes, a question? So how's platform support? I've seen some support for IONX6, IONX8, right? Anything new and notable in that? Sorry? Anything new and notable from platforms supported by WPE? Yeah, so I can say that I've been working on, most especially on IONX8m lately, on making sure that WPE works with the AdNative driver, Graphics driver, instead of the Viviente. And apart from that, I'm not sure what you're asking about. Yeah? I think there's one more. Thank you. Sorry, quick question. Sorry if it's basic. Just wanted to know what the license for WPE is. Yeah, so the license is BSD and LGPL, too. Okay, thank you.
 All right.  So I guess started them.  Yeah.  Good afternoon, everybody.  I have this presentation about WPE, which is a web engine based on WebKit and designed  for embedded platforms.  So during the talk, I give a short introduction about WebKit, a very short overview.  And then I will more focus on applications.  How you can use WPE in your application, what are the use cases for it.  And probably in five time, I'll give a short demo about something I wrote.  All right.  So I'm Philippe.  I'm WebKit reviewer, a digital committer.  I work for a company based in Spain called Igalia.  We are working on a cooperative.  We are currently about 90 IP Igalians across the world.  So it's based on WebKit, which probably, as you know, was a project started by Apple  as a fork of KHTML around 2004.  And the main goal for them at the time was to build Safari.  And since then, other people joined the projects, other companies.  The community added new ports.  I will talk about that later.  So I briefly introduced the WebKit to multi-process architecture.  On that diagram, you can only see two processes, but nowadays there are more.  There's one about network nowadays, one about storage.  And this recently introduced GPU process as well.  So the application is called the UI process in that architecture.  And it communicates with other processes using IPC.  And all the DOM, passing all the things internal to the web engine are done in those processes,  like JavaScript and all those things.  So WP is a port aimed for embedded platforms.  Traditionally, WebKit ports are built on top of a toolkit widget.  But this port is a bit special in the sense that it decouples rendering to dedicated what  we call rendering backends.  That means that it gives more flexibility to the application developers.  And there's more openness about the platforms that you can support using that model.  So that means that the rendering is deferred to another part of the port that is loaded  at runtime, basically.  And the input events are also under using that model.  So those rendering backends usually require a thing called Weleon EGL, which is a way  for applications to share graphics resources between processes.  And they have EGL extensions dedicated to that.  So there are quite a few backends nowadays.  I will just focus on two.  The first one is the FDO backend, which we'll talk about in a minute.  And the other one is the RDK backend, which is developed by companies such as Comcast.  They have deployed it in a wide range of set of boxes, actually, in the world.  So the FDO backend, it depends on any GL.  So that means usually if you have a GPU driver that's using MSA, or probably something that  provides EGL, you can have binary drivers.  So that backend provides a high-level API for applications to be able to get EGL images  from the web engine so that they can do whatever they want with that.  And then that's also what we recommend for the WP community.  It's the backend used on the build boards, and it's the backend we use the most in the  community.  Then obviously you need a browser or some kind of application.  So at EGL, we are working on a minimalistic browser called Cog.  It depends on the WP backend FDO, and it's minimalistic.  There's actually no window decresion or nothing.  And single-webview, although we are working on multi-webview support, and it also can  be controlled using D-Bus.  So in your application, you can remote control the browser using other applications.  So as a basic, you need to have a Wayland compositor to be able to run Cog.  But since recently, we wrote a new backend that leverages the DRM architecture, and thus  we don't really require a running Wayland compositor.  So that's quite cool because it reduces the dependencies that you need for your application,  and if you need a full-screen application, you can use that backend.  Like kiosks, set-top boxes, UI, any full-screen display.  And then the rendering is done with DRM.  So the Wayland buffers are imported as GB and buffer objects, and then rendered using  DRM.  I have a small showcase for that.  It's basically a thing I did in my free time.  I was a bit tired of using Codiatom, so I wanted to have a minimalistic media center  self-contained.  So I built a web extension for WP.  And that can be loaded at runtime by the web process.  And in that web extension, I use a UPNP library called libg-upnp.  So I use that library to discover the media servers on the network.  And then for each server, I modify the DOM and add these elements to the HTML page rendered.  So then I make a little build-up web page based on what's on the network, basically.  And then I can do the video playback with the normal video element that's stand-down  nowadays in WebKit.  So there's a demo.  It's really subsequent.  If you want to look at it on your phone, you can scan that code.  Otherwise, I can just move on.  But yeah, it's an interesting showcase of what you can do with Cog.  And it gives an overview of really what you can do.  Another thing I wanted to discuss about is acute applications.  As you probably know, acute is used widely used in the industry nowadays.  And if you have a web engine in your application such as Qt WebKit or Qt Web Engine nowadays,  you probably would like to have an alternative because Qt WebKit is not maintained anymore.  It has a lot of security issues.  And some people don't like Qt Web Engine because it's too big.  Yeah.  So this is an alternative to those two options you have.  So what we did in upstream WebKit, we've built a QML module that can be swapped in instead  of the Qt WebKit-based QML module.  So if you have an application using Qt WebKit, you can directly use that module.  And it will use internal AWP instead of Qt WebKit.  So what you gain is maintain web engine with security releases.  And so in that sense, it's quite a good advantage.  There are some drawbacks, though.  It works only on Linux.  So if your application needs to run on Windows, you're a bit screwed.  And then there's dependency on Web and EGL that we can't really work around.  It works currently in the EGLFS, QPA, and WLAN EGL as well.  So that means you can work on desktop and on embedded platforms.  So to enable it, you just need to download Qt WebKit.  WP WebKit, sorry.  And turn on that CMake option called enable WPQTAPI.  And then around time, you just need to make sure that the SO file is in your import path.  And then I have a small code snippet, QML snippet there that shows how to basically  use that WP view in your application.  So the changes compared to Qt WebKit is that you just need to change the import line.  And the module name is WP view.  But in the Qt WebKit world, I think it's a different name.  Otherwise, the API is similar.  So on title change here that is already available in Qt WebKit can be used as it is.  All right.  And then the last thing I wanted to talk about is how to use WP in multimedia application  using Gstreamer.  That's a bit of an over way.  Think about streaming browsers or HTML overlays in live TV, for instance.  So that could be a use case for WP.  Just a brief overview of Gstreamer.  Who hasn't heard about Gstreamer?  Nobody?  Yes?  Okay.  So basically it's a multimedia framework based on graphs.  So in your application, you build a graph and the data flow will flow from the source  to the sync and there will be data processing like decoding and rendering.  There's many, many plugins.  I won't go into details.  But what I did is write a new plugin that uses WP and I built a source element for that  so that the video, the web view can be outputted as a video stream.  And it's quite cool because we have zero copy from WebKit world into Gstreamer world using  the ChargedGL context.  So there's no memory copies in there.  The use cases as I said, HTML overlays and streaming browsers.  I started, so that's a demo I wrote.  On the right side, you can see some HTML and on the left side, the preview of it with a  live video behind.  And then you can stream that.  I can show you, show it later.  And when you update the HTML, it's updated in the video, of course.  So that's for the TV broadcasting world.  It could be useful, for instance.  Some ongoing work I've been doing lately on that plugin, adding audio support.  I've reported type already.  I need to upstream it.  And then navigation events in Gstreamer are not really, have been designed many years  ago and it's to be modernized a bit.  So I started working on that as well.  Maybe adding mouse core support to achievements and improving keyboard support as well.  So the code is in WPWadkid.org.  We have a Yokto layer.  The support is upstream in ViewTort as well.  This has been working quite well on AMX6, AMX8M platform, Raspberry Pi 3 and 4.  And it's deployed in the wild already.  So you can use it now.  I guess I have time for a demo.  Okay.  If it works.  Okay.  Okay, let's try this again.  Okay, it doesn't work.  But you can install it with a flat pack available and you can install it quite easily on your  desktop.  I can provide the flat pack ref later on if needed.  But I don't have it installed here.  Sorry.  No, it's not working.  Anyway, so yeah, that's it.  If you have any questions, I will be happy to answer.  Yes, a question?  So how's platform support?  I've seen some support for IONX6, IONX8, right?  Anything new and notable in that?  Sorry?  Anything new and notable from platforms supported by WPE?  Yeah, so I can say that I've been working on, most especially on IONX8m lately, on  making sure that WPE works with the AdNative driver, Graphics driver, instead of the Viviente.  And apart from that, I'm not sure what you're asking about.  Yeah?  I think there's one more.  Thank you.  Sorry, quick question.  Sorry if it's basic.  Just wanted to know what the license for WPE is.  Yeah, so the license is BSD and LGPL, too.  Okay, thank you.
ema_wpe_0_179.png ema_wpe_180_863.png ema_wpe_864_1415.png ema_wpe_1416_1931.png ema_wpe_1932_1991.png ema_wpe_1992_2039.png ema_wpe_2040_2735.png ema_wpe_2736_3743.png ema_wpe_3744_5099.png ema_wpe_5100_6383.png ema_wpe_6384_7475.png ema_wpe_7476_8483.png ema_wpe_8484_9911.png ema_wpe_9912_11615.png ema_wpe_11616_12251.png ema_wpe_12252_13187.png ema_wpe_13188_14051.png ema_wpe_14052_14723.png ema_wpe_14724_15191.png ema_wpe_15192_16079.png ema_wpe_16080_16763.png ema_wpe_16764_17615.png ema_wpe_17616_18383.png ema_wpe_18384_19103.png ema_wpe_19104_19811.png ema_wpe_19812_19859.png ema_wpe_19860_20519.png ema_wpe_20520_21863.png ema_wpe_21864_22343.png ema_wpe_22344_22703.png ema_wpe_22704_23783.png ema_wpe_23784_25271.png ema_wpe_25272_25396.png