XDP and page_pool allocator
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Apalodimas, Ilias
Bianconi, Lorenzo
XDP support is an increasing trend on the network devices. XDP main goal is processing packets at the lowest point in the software stack avoiding overheads. Memory recycling of received buffers achieved through the in kernel pagepool API plays a fundamental role in the increased performance. Adding XDP support on a driver can be non-trivial. In this talk we'll demonstrate how porting a standard ethernet driver (mvneta/netsec) to XDP and the pagepool API can boost performance. Part of the page_pool evolution involves adding the recycling support in the kernel's SKB stack and leverage the increased performance attributes of the API.
 Hi, we're going to present XDP and the PagePool allocator and how you can easily add a driver or convert an existing Linux driver into using XDP by using an internal API. The reason we decided to talk is that I'm Amelias, I'm the technical lead for Linaro Edge and Fog Networking Department and I'm serving as a co-maintenor for the PagePool API at the moment. I've added XDP support on a kernel driver and Lorenzo is a software engineer for RedHat, he's maintaining a wireless driver, it's MT76, right? And he's added XDP support on the Espresso bin board which uses the Marvel MVnet driver. So do you know what XDP is? Anyone does? All right, good. Let's go a bit faster on this one. It's a software offload path for the kernel. We, on the driver level, we add some hooks on the RX path and by using the PagePool API for the memory allocation, we don't have to keep reallocating and freeing memory when we process the package, we just have to sync to the correct direction, the DMA direction for the CPU and the network interface to pick up the packets. It was initially designed as a, to operate on layer two and layer three, while the Linux kernel operates on layer two to say layer seven, but it's mostly optimized around layer four, around layer four. Because two reasons we get better performance on XDP, the first one is that we, on most of the cases, we managed to recycle the memory we're using and we skip all the kernel paths that we don't really want, like IP tables or the TC hook or the root lookups and stuff like that. It's important to keep in mind that this is not a kernel bypass. One of the functionalities is a kernel bypass and you can dump packets directly to user space, but it's a kernel, it's an internal fast path and we'll elaborate on this a bit more. It uses existing kernel APIs and existing kernel functionality and you can program the number of packets and the type of the packets you want to process through BPF. It's currently being used by Facebook and CloudFer on load balancers, on DDoS protection. So this is pretty much how the architecture looks like. The XDP you see on the driver is actually the BPF program doing the decision making. So if it's an XDP pass, which is one of the actions you have on XDP, you send back the packet to the Linux networking stuff. If it's a TX, an XDP TX, what you do is that you send the packet back out of the interface it came from by changing the header or the source and MAC address or the IP or anything you want to change on the packet. The XDP redirect currently sends the packet over to user space or a remote CPU or anything you decide on that one. And there's the NDO XDP XMIT which you can pick up the packet the moment it arrives on your network interface and offload it to a different network art without having to go through the kernel network stack. Now the reason we created the PagePool API is that the memory model for the whole approach is a bit weird. We require packets to be in contiguous physical memory and this is not a requirement from us it comes from the BPF direct access for validating the packet and the correctness. So you can't have one packet split across multiple physical pages at the moment. So you're limited to non-jumbo frames and we don't mean 1,000, 5,000 byte packets. It's just anything below a page size can be accommodated in an XDP frame. So the problem with that is that you cannot allocate the memory you want with whatever we have in the kernel of functions like NAPIA lockfrag which allocates fragments for your data and it's faster because we cast things in there. You really have to allocate the page, you have to account for the headroom and the tail size we need on BPF and for whatever you need on the SKB. Now what we discussed is that the buffers must be recycling in order to get the speed. So the PagePool allocator we have, it's optimized for one packet per page. We have used cases of people splitting the page and fitting it multiple packets in it but you can't recycle based on the PagePool allocator recycling functions. You have to recycle on your own in that case. On the native packet recycling we do it in the NAPI context mostly so this is really fast because you don't have any extra locking overhead, you're already protected by the NAPI context. And the API also offers DMA managing capabilities that means that it can map your buffers, it can sync your buffers correctly and there's improvements from Lorenzo that speed up this even more. Now this is not all perfect. If you switch from NAPIA lock SKB that the kernel is doing to an XDP, your network stack, your normal network stack in the kernel is going to slow down because allocating a page compared to allocating fragments is substantially slower. But if you use it for XDP then you get all the native performance improvements we have with recycling packets. The memory footprint is bigger because instead of allocating the amount of size you want for the packet you allocate the page and you fit the packet wherever you want in that page. And we do have some off three patches to get some performance back from that penalty. So we have patches and we manage to recycle buffers even if they address the normal network stack. So if it's an SKB we eventually recycle that buffer as well. So actually Ilias has gone through some XDP requirement and some more general information about XDP and I will give some more details about how implement XDP in an Ethernet driver. Actually I use the MWVneta Marvel 1 gigabit drivers as reference since for example the Intel or the Marvel Melonox implementation are much more complex. We need to take into account that in order to be accepted in the Linux kernel our driver needs to implement all the possible XDP verdicts that are XDP drop, XDP TX, XDP pass and XDP redirect. Here I reported some hardware specification of the Marvel Espresso bin that is the development board I used to add XDP support to the MWVneta driver. And we can see that the Marvel Espresso bin runs a Cortex A53 and for networking we have two gigabit Ethernet LAN ports, one Ethernet 1 port, all of them connected together through an Ethernet DSA hardware switch. This diagram I client the life cycle of a buffer using a page pool allocator and we can see that the page pool allocator is usually created opening an interface since it is actually associated to a given RxQ in order to avoid locking penalties. From here we can see that it's possible to rely on the page pool API in order to for the MA mapping and the MA syncing using this flex actually. And what is important to notice in this slide is that when the NAPI pole runs it actually runs an EVPF program that is attached to our network interface and the EVPF program will return an XDP verdict, let's say a result, then the buffer will be recycled according to this result. Actually the page pool allocator will have two caches, one in interrupt caches that is used for when the driver is running in interrupt context and we have a single reference to the buffer or a pointer ring cache that is used when we have a single reference to the buffer. Whenever our driver needs to refill the DMA engine with the new buffer running, for example in this case the M1.net RxRefill, we can access to these caches instead of going through this lower page allocator. Here I reported the M1.net XDP architecture and we can see that whenever the M1.net Apollo runs it allocates an XDP buffer that is the counterpart of an SKB for XDP and the M1.net RxDP will actually run the EVPF program in the EVPF sandbox on our XDP buffer and will retire one of this XDP verdict, XDP pass, XDP drop, TX on the red, or redirect and the buffer will be managed accordingly. It's important to notice here that the XDP buffer, the struct XDP buffer is allocated on the stack and not through a KMM cache as is done for a classic SKB. Now let's go through each possible XDP verdict and let's consider XDP drop. XDP drop is returned by our EVPF program when it wants to drop the packet as fast as it can and the typical use case for XDP drop is an anti-DDoS application. We can see here that whenever the packet returns XDP drop, the packet will be recycled in the in interrupt cache using page pool recycled redirect. Moreover here I reported the comparison between a simple program that just run XDP drop and the same functionality implemented with a DC filter and DC action. We can see that with the XDP we can almost reach 600 kilopack per second drop while with DC we can just roughly drop 180 kilopack per second. Here we see the XDP TX, how is the XDP TX works in the Mevunet driver and XDP TX is used to transmit the packet back to the interface where we receive the packet. We can see that now, typical application for example is a load balancer in this case. We can see that now that running the Mevunet XDP X-MIT back, it's not the, I mean the Mevunet XDP X-MIT back will reinsert the packet in the hardware the MATX ring and it's not important in this case to demerit map the buffer since it has been already mapped by the page pool API. We just need to flush the CPU caches in this case because the device is not current. Here we have XDP redirect that XDP redirect is used to transmit the packet to for example a remote interface or if or to remote CPU or to even a socket using for example AFXDP and the typical use case is like for example the year two forwarding. It's important to notice here that in order to redirect it to a remote interface for example the remote device should implement the NDO redirect X-MIT function pointer and here we have the implementation done for the Mevunet. We notice here that in this case is necessary to demerit map the buffer since it is being received actually by from remote device. Last verdict is XDP pass. XDP pass is used to send the packet to the standard Linux network stack and in the Mevunet implementation we can see that we can rely on the build SKB. So there is no need to reallocate the buffer for the payload of the packet but we need to take into account when we allocate the packet using the page pool API that we need to take into account even the size of the SKB shared info. What we notice moreover in this slide is that in this particular case we are not able to recycle the buffer yet since whenever we need to refill the DMA engine with new buffers we need to go through the standard page allocator but as Silvia said this feature is under development. So in conclusion we saw some XDP requirements and some basic information about XDP like XDP memory model. We saw some basic about the page pool allocator and how to implement each XDP verdict using this API and we saw the Mevunet implementation as reference. The future works are definitely the adding support for SKB recycling for the XDP pass functionally for XDP pass verdict and for example regarding Mevunet we need to add support for the XDP support for the other buffer manager that is available on some device like Solid Drunclear Fog, native support for AFXDP and some interesting bits that are currently on the XDP roadmap. Questions? For me or for him? I'm not sure actually one of my questions is AFXDP, with AFXDP the user space needs to allocate the memory and one of the rest functions we had previously was that you could use a huge page when you needed to map memory from the user space. You can do it. All right. Okay. But it's not optimized internally in structure. The answer is that you can do it, but it's not internally optimized at the moment for the time being so. Can we take this? Yes. I've recently been experimenting with the XUB on VATH interfaces, which is slow. I'm wondering if there's any whole pointy transfer that could become fast in the future. Which interfaces? The virtual VATH interfaces. Like virtualized interfaces. The first video interfaces? No, no. VH. VH. VH. This depends on the card you're on. I think it's software. Software implementation. Yeah. I've never tried it, actually. We don't have any intention of working on it at the moment, so. Thank you. Thank you. Thank you.
 Hi, we're going to present XDP and the PagePool allocator and how you can easily add a driver  or convert an existing Linux driver into using XDP by using an internal API.  The reason we decided to talk is that I'm Amelias, I'm the technical lead for Linaro  Edge and Fog Networking Department and I'm serving as a co-maintenor for the PagePool  API at the moment.  I've added XDP support on a kernel driver and Lorenzo is a software engineer for RedHat,  he's maintaining a wireless driver, it's MT76, right?  And he's added XDP support on the Espresso bin board which uses the Marvel MVnet driver.  So do you know what XDP is?  Anyone does?  All right, good.  Let's go a bit faster on this one.  It's a software offload path for the kernel.  We, on the driver level, we add some hooks on the RX path and by using the PagePool API  for the memory allocation, we don't have to keep reallocating and freeing memory when  we process the package, we just have to sync to the correct direction, the DMA direction  for the CPU and the network interface to pick up the packets.  It was initially designed as a, to operate on layer two and layer three, while the Linux  kernel operates on layer two to say layer seven, but it's mostly optimized around layer  four, around layer four.  Because two reasons we get better performance on XDP, the first one is that we, on most  of the cases, we managed to recycle the memory we're using and we skip all the kernel paths  that we don't really want, like IP tables or the TC hook or the root lookups and stuff  like that.  It's important to keep in mind that this is not a kernel bypass.  One of the functionalities is a kernel bypass and you can dump packets directly to user  space, but it's a kernel, it's an internal fast path and we'll elaborate on this a bit  more.  It uses existing kernel APIs and existing kernel functionality and you can program the  number of packets and the type of the packets you want to process through BPF.  It's currently being used by Facebook and CloudFer on load balancers, on DDoS protection.  So this is pretty much how the architecture looks like.  The XDP you see on the driver is actually the BPF program doing the decision making.  So if it's an XDP pass, which is one of the actions you have on XDP, you send back the  packet to the Linux networking stuff.  If it's a TX, an XDP TX, what you do is that you send the packet back out of the interface  it came from by changing the header or the source and MAC address or the IP or anything  you want to change on the packet.  The XDP redirect currently sends the packet over to user space or a remote CPU or anything  you decide on that one.  And there's the NDO XDP XMIT which you can pick up the packet the moment it arrives on  your network interface and offload it to a different network art without having to go  through the kernel network stack.  Now the reason we created the PagePool API is that the memory model for the whole approach  is a bit weird.  We require packets to be in contiguous physical memory and this is not a requirement from  us it comes from the BPF direct access for validating the packet and the correctness.  So you can't have one packet split across multiple physical pages at the moment.  So you're limited to non-jumbo frames and we don't mean 1,000, 5,000 byte packets.  It's just anything below a page size can be accommodated in an XDP frame.  So the problem with that is that you cannot allocate the memory you want with whatever  we have in the kernel of functions like NAPIA lockfrag which allocates fragments for your  data and it's faster because we cast things in there.  You really have to allocate the page, you have to account for the headroom and the tail  size we need on BPF and for whatever you need on the SKB.  Now what we discussed is that the buffers must be recycling in order to get the speed.  So the PagePool allocator we have, it's optimized for one packet per page.  We have used cases of people splitting the page and fitting it multiple packets in it  but you can't recycle based on the PagePool allocator recycling functions.  You have to recycle on your own in that case.  On the native packet recycling we do it in the NAPI context mostly so this is really  fast because you don't have any extra locking overhead, you're already protected by the  NAPI context.  And the API also offers DMA managing capabilities that means that it can map your buffers, it  can sync your buffers correctly and there's improvements from Lorenzo that speed up this  even more.  Now this is not all perfect.  If you switch from NAPIA lock SKB that the kernel is doing to an XDP, your network stack,  your normal network stack in the kernel is going to slow down because allocating a page  compared to allocating fragments is substantially slower.  But if you use it for XDP then you get all the native performance improvements we have  with recycling packets.  The memory footprint is bigger because instead of allocating the amount of size you want  for the packet you allocate the page and you fit the packet wherever you want in that page.  And we do have some off three patches to get some performance back from that penalty.  So we have patches and we manage to recycle buffers even if they address the normal network  stack.  So if it's an SKB we eventually recycle that buffer as well.  So actually Ilias has gone through some XDP requirement and some more general information  about XDP and I will give some more details about how implement XDP in an Ethernet driver.  Actually I use the MWVneta Marvel 1 gigabit drivers as reference since for example the  Intel or the Marvel Melonox implementation are much more complex.  We need to take into account that in order to be accepted in the Linux kernel our driver  needs to implement all the possible XDP verdicts that are XDP drop, XDP TX, XDP pass and XDP  redirect.  Here I reported some hardware specification of the Marvel Espresso bin that is the development  board I used to add XDP support to the MWVneta driver.  And we can see that the Marvel Espresso bin runs a Cortex A53 and for networking we have  two gigabit Ethernet LAN ports, one Ethernet 1 port, all of them connected together through  an Ethernet DSA hardware switch.  This diagram I client the life cycle of a buffer using a page pool allocator and we  can see that the page pool allocator is usually created opening an interface since it is actually  associated to a given RxQ in order to avoid locking penalties.  From here we can see that it's possible to rely on the page pool API in order to for  the MA mapping and the MA syncing using this flex actually.  And what is important to notice in this slide is that when the NAPI pole runs it actually  runs an EVPF program that is attached to our network interface and the EVPF program will  return an XDP verdict, let's say a result, then the buffer will be recycled according  to this result.  Actually the page pool allocator will have two caches, one in interrupt caches that is  used for when the driver is running in interrupt context and we have a single reference to  the buffer or a pointer ring cache that is used when we have a single reference to the  buffer.  Whenever our driver needs to refill the DMA engine with the new buffer running, for example  in this case the M1.net RxRefill, we can access to these caches instead of going through  this lower page allocator.  Here I reported the M1.net XDP architecture and we can see that whenever the M1.net Apollo  runs it allocates an XDP buffer that is the counterpart of an SKB for XDP and the M1.net  RxDP will actually run the EVPF program in the EVPF sandbox on our XDP buffer and will  retire one of this XDP verdict, XDP pass, XDP drop, TX on the red, or redirect and the  buffer will be managed accordingly.  It's important to notice here that the XDP buffer, the struct XDP buffer is allocated  on the stack and not through a KMM cache as is done for a classic SKB.  Now let's go through each possible XDP verdict and let's consider XDP drop.  XDP drop is returned by our EVPF program when it wants to drop the packet as fast as it  can and the typical use case for XDP drop is an anti-DDoS application.  We can see here that whenever the packet returns XDP drop, the packet will be recycled in the  in interrupt cache using page pool recycled redirect.  Moreover here I reported the comparison between a simple program that just run XDP drop and  the same functionality implemented with a DC filter and DC action.  We can see that with the XDP we can almost reach 600 kilopack per second drop while with  DC we can just roughly drop 180 kilopack per second.  Here we see the XDP TX, how is the XDP TX works in the Mevunet driver and XDP TX is  used to transmit the packet back to the interface where we receive the packet.  We can see that now, typical application for example is a load balancer in this case.  We can see that now that running the Mevunet XDP X-MIT back, it's not the, I mean the  Mevunet XDP X-MIT back will reinsert the packet in the hardware the MATX ring and it's not  important in this case to demerit map the buffer since it has been already mapped by  the page pool API.  We just need to flush the CPU caches in this case because the device is not current.  Here we have XDP redirect that XDP redirect is used to transmit the packet to for example  a remote interface or if or to remote CPU or to even a socket using for example AFXDP  and the typical use case is like for example the year two forwarding.  It's important to notice here that in order to redirect it to a remote interface for example  the remote device should implement the NDO redirect X-MIT function pointer and here we  have the implementation done for the Mevunet.  We notice here that in this case is necessary to demerit map the buffer since it is being  received actually by from remote device.  Last verdict is XDP pass.  XDP pass is used to send the packet to the standard Linux network stack and in the Mevunet  implementation we can see that we can rely on the build SKB.  So there is no need to reallocate the buffer for the payload of the packet but we need to  take into account when we allocate the packet using the page pool API that we need to take  into account even the size of the SKB shared info.  What we notice moreover in this slide is that in this particular case we are not able to  recycle the buffer yet since whenever we need to refill the DMA engine with new buffers  we need to go through the standard page allocator but as Silvia said this feature is under development.  So in conclusion we saw some XDP requirements and some basic information about XDP like  XDP memory model.  We saw some basic about the page pool allocator and how to implement each XDP verdict using  this API and we saw the Mevunet implementation as reference.  The future works are definitely the adding support for SKB recycling for the XDP pass  functionally for XDP pass verdict and for example regarding Mevunet we need to add support  for the XDP support for the other buffer manager that is available on some device like Solid  Drunclear Fog, native support for AFXDP and some interesting bits that are currently on  the XDP roadmap.  Questions?  For me or for him?  I'm not sure actually one of my questions is AFXDP, with AFXDP the user space needs  to allocate the memory and one of the rest functions we had previously was that you  could use a huge page when you needed to map memory from the user space.  You can do it.  All right.  Okay.  But it's not optimized internally in structure.  The answer is that you can do it, but it's not internally optimized at the moment for  the time being so.  Can we take this?  Yes.  I've recently been experimenting with the XUB on VATH interfaces,  which is slow.  I'm wondering if there's any whole pointy transfer that  could become fast in the future.  Which interfaces?  The virtual VATH interfaces.  Like virtualized interfaces.  The first video interfaces?  No, no.  VH.  VH.  VH.  This depends on the card you're on.  I think it's software.  Software implementation.  Yeah.  I've never tried it, actually.  We don't have any intention of working on it at the moment,  so.  Thank you.  Thank you.  Thank you.
xdp_and_page_pool_api_0_167.png xdp_and_page_pool_api_168_395.png xdp_and_page_pool_api_396_1043.png xdp_and_page_pool_api_1044_3467.png xdp_and_page_pool_api_3468_3635.png xdp_and_page_pool_api_3636_3995.png xdp_and_page_pool_api_3996_4091.png xdp_and_page_pool_api_4092_4151.png xdp_and_page_pool_api_4152_4247.png xdp_and_page_pool_api_4248_4343.png xdp_and_page_pool_api_4344_4547.png xdp_and_page_pool_api_4548_4607.png xdp_and_page_pool_api_4608_6287.png xdp_and_page_pool_api_6288_7163.png xdp_and_page_pool_api_7164_8699.png xdp_and_page_pool_api_8700_10007.png xdp_and_page_pool_api_10008_10847.png xdp_and_page_pool_api_10848_13415.png xdp_and_page_pool_api_13416_14723.png xdp_and_page_pool_api_14724_16331.png xdp_and_page_pool_api_16332_17579.png xdp_and_page_pool_api_17580_18875.png xdp_and_page_pool_api_18876_20243.png xdp_and_page_pool_api_20244_21683.png xdp_and_page_pool_api_21684_24983.png xdp_and_page_pool_api_24984_25103.png xdp_and_page_pool_api_25104_25217.png