IP Forwarding Fastpath
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Momiyama, Nanako
User-space packet forwarding has demonstrated impressive performance. However, doing so in production has introduced a lot of fundamental and engineering problems, such as isolation, scalability to the number of VMs and API/ CLI compatibility. We therefore explore a software packet switching architecture to address these problems, basing our solution on the kernel packet forwarding plane. At EuroBSDcon 2016 we demonstrated that applying VALE/mSwitch for packet I/O and DXR for IPv4 lookup accelerates the FreeBSD packet forwarding rate by a factor of 1.7, reaching to 2.4 Mpps on a single CPU core. In this talk we present new methods which increase this rate to 5.6 Mpps. Despite of the superior performance, commercial interests have exposed problems with using user-space software switches and routers; examples include lack of compatibility with existing CLIs and APIs, complexity to (securely) serve VMs, scalability to the number of ports or VMs, energy consumption (e.g., due to polling on a NIC) and lack of extensive protocol support. In this work we therefore do not take a path of user-space networking; we instead explore a kernel forwarding plane architecture that achieves similar performance to user-space networking. We start from improving packet I/O and IP lookup, because they are known to be bottlenecks. We used current FreeBSD for baseline and applied VALE/mSwitch for packet I/O and DXR for IP lookup. Default FreeBSD forwards packets at 1.4 Mpps and applying VALE/mSwitch and DXR increases this rate to 2.4 Mpps. Since this rate is still far away from the 10 Gbps line rate at which mSwitch/VALE can forward, we carefully analyse bottlenecks in our system. We find that L2 and L3 protocol processing is expensive at both input and output paths. We therefore implemented new methods to conditionally bypass expensive code paths: quick protocol identification for input and destination MAC address caching within DXR FIB for output. As a result, we increase the packet forwarding rate to 5.6 Mpps.
... have video, say something... Hello? Say something, girl. Okay. So, what I said before, so, so, so I just explained about the Excel and, and so, the Excel creates the very compact art data structure to, based on the, the OS defaults about X3, so they reuse the data and they create a compact data and the data structures is to fit our into the CPU cache hybalonarchies, so it can accelerate the virtual cap speed and so, and this time our, I ported our DXR patch, our is created for our, the free VSE 8, it's a little bit old, so I ported the original one to the, to fit into the free VSE 12 and the DXR feed lookup function is called instead IP found boot and this is our experimental set up. There's the two machines are, and the one machine is installed on my modified kernel and the OS, the base OS is free VSE 12 and the CPU and the network interface card is very common one, our Intel CPU and engineering, it's Tangier Nick, so there's the two machine and connected to the back to back with Tangier Nick and the other machines are generated a packet using package gem and measure the packet rates provided by the router, it's very, very simple and the measurement metric are is, the package size is 60 bytes, are short packets exclude CRC and traffic has their same destination so it become the maximum cash FB capacity and routing size is minimized, are less than 10 entries and this is the result of the performance measurement, so when we, the defaults are previously for a packet 1.43 million packet per second and when we replace the two parts, this line, and the XR, are this performance up to the 2.43 million packet per second but the M switch itself has a potential to, has a potential to forwarding the packets are over 10 million packet per second because it doesn't go up the kernel so we can see there's a huge gap, still huge gap between our, when we forward the packet to the kernel or not, so, so to find the ways there are bottleneck, we measure the, the latency more details, okay, so this is the other measurement to know the bottleneck and the measurement methodology is a little bit tricky are using M switch and the M switch is are the hard-coded to forward the packet to an intended output so it means that when the packet returns are to the M switch layer, it is anyway if the packet corrupt or not, or it is anyway are send to the other nick, so, so the received packets are on the packet generator machine are and measure the, measure the packets that traffic late and then we can measure the time from the packet alive at the nick and the, the returning point, so the table are summarize the results and the point is even if, so here's the, the which function takes the time and the first is L2 input and the second is L3 input, the third one is L3 output and the last one is L2 output, so some of these results are the point is even if our mutual gap are cost is minimized, L2 input and output of routine takes about over 200, 215 nanoseconds and make, make slow down the performance, okay, so our measurement confirms that the packet IO and the mutual gap are not extensive anymore and anymore, so the point is that input and L2 input and output part, okay, so from these are the first design and the measurement we can say the three things and the packet IO and the mutual gap is not, is not a bottleneck anymore and L2 input and output part are expensive, so for further speeding up or we have to bypass the expensive this part, okay, so from this takeaway we found the, we placed the packet IO and mutual gap are not enough to speed up, so then we renewed the design, so this is the additional, this is the renewed design and implementation, so the basic idea is our co-editionary bypass expensive code are at the both input and output path, okay, so this time we focus on IPv4 forwarding and the proposal new methods to bypass the, the other features, so for avoiding our input part of the Ethernet processing we implement a filtering feature to allow the packet which matches the rule of this time IPv4 and can shortcut the Ethernet input and the directory goes to the L3 input routine, so this red arrow indicates the path path and the black one is the forward path, so this is input, input bypass part, so the other is output bypass part, so to improve the performance of our Ethernet output routine, reducing the frequency to our access to Mac address table is important, so L2 cap has already optimized by Qo internal casting mechanisms, but our target speed is, requires nanoseconds a level saving and the cost of L2 cap is not trivial according to our measurement, so, so we extended the DxR feed to cast Ethernet header information and to Ethernet information for the next hop, okay, and so, and the first packets are needed to have to go to the slow, block, block a low path, but if the second, subsequent packet has the same destination, it can use the cache data in the DxR feed, so if the packet has the same destination, it can bypass the IPh output routine and directly back to the M-suit layer, and this cache information is varied within the batch M-suit process, so this is the new design implementation, and the next is the evaluation, the machines, the experiment is about as almost same as the previous measurement, and but this time the packet size is varies from the short packet to the large packet, and the other is the same, okay, so this is the result of various packet size, so the graph describes the throughput of an IP forwarding of different modules with a six variety of packet size, so we compare previously default forwarding plan to our implementation, so X-axis is the packet size and and to, and to know the, to know the which improvements effects are the performance, we create and run the full version of our implementation, so the all of our implementation are equipped the M-suit and DxR, and the difference between our Ds are four modules is the level of the bypass, okay, so no bypass, the below the no bypass indicates this implementation has the M-suit and DxR, but no input or output bypass, and the green one has the input bypass and output bypass means it has only the output bypass, and the pink one is three bypassing the input and output part, so the native free VSD and and the most most right part gray bar means the line rate with the packet size, okay, so the native free VSD and the no bypass records the line rate at the large packet size, or 1k or something, but our implementation reach the line rate with our 252, and also in a short packet, our three input plus output bypass can reach over 5 million packet per second, it is which is a factor of 3.8 improvement of warding the warding speed, so the other rough reports are the relation between the throughput and the batch size in M-switch, the batch size varies from 1 to 1,024, we need two different packet size, and so we investigate the batch size effect on throughput, because M-switch paper shows are the throughput of M-switch affected by the batch size, and so we expected that our implementation is also affected by the batch size, okay, because not only the M-switch itself can be affected by the batch size, but we are setting the cache aligning time with the batch size, okay, so the throughput increases with growing the batch size, and it reach its peak to 256 or 512, and not our change after that, so this shape is almost the same as in the M-switch paper, so M-switch paper, so we need more investigation, and analyze and investigate, but the cost of our cache could be, it's not our high-impact to the performance, okay, so from these results, we can say that our new implementation reduced over the 80% of L2 input and also L2 output routine, and our implementation accelerates the default 3BSD IP4 in late by a factor of 3.8, and the reach over 5 million packet per second with minimal size of packet, and the default is 1.45, so the cache creation cost appears relatively cheap, but this part is needed, the further analysis, okay, in a future work, about future work, so this measurement is basically on the single core performance, so I need to our more measurement with multi-core scalability, and also need to our scalability to the loading table entry, and also I want to do some contribution to the 3BSD itself, so here's the conclusion, so my research started to the problem that the user space networking has many problems in practice, so that my approach is the fast kernel-based packet loading, and in this approach we take three measures of design, and replace the packet IO and the L2GAP part, and then on the bypass expensive routine at the both input and output part, so this approach reaches 3.8 times speed up to the default 3BSD, and they increase the packet loading rate to the 5 million packet per second, so this results demonstrated the possibility of kernel-based packet loading, okay, so I will show the demo, so the first demo is very simple performance test of my L2GAP, and the second one is to show the compatibility to our normal routing sockets, so in the L2GAP, my modified kernel, so running in the application are the BGP demo, top of my L2GAP, and the shorter this BGP demo are Vox, so the first demo, this is the packet generate, can you see it? Okay, this machine is our packet generator machine, and with using with the packet gen, this is the process of sending the packets, so this process creates the packet are in 1488 million packet per second, so sending a packet to the routing machine right now, and the other, so in the diatonic, here's the receiving process, and the receiving packet with 1.6 million packet per second, and so we attach the router machine, in the router machine we attach the NIC to the M switch, and then sending machine also creates the 10-year line-length traffic, and this time we attach the NIC to the M switch, so the router machine can forward over 5 million packet per second this time, okay. And the other demo is our, using the BGP demo are in top of this router, so in the router machine, the BGP demo is running, and the other side of the machine, here's the emulator too are creating the BGP demo as a neighbor, so this emulator creates the BGP demo and sending the routing information to the to my router, so in this our machine can receive the BGP information, like this, okay, good update, any, any, okay, thanks for everything, any questions when you comment? I wonder if you did a better job than you think you did, so have you met, you mentioned, you keep mentioning actual 10-year line-length, what's the, did you measure our previous PIX driver and how fast the single key can receive, and receive up to the point with your switch runs, and just drop it as an interest to switch? So you're measuring everything from the above the driver, but have you measured how long it takes the driver to allocate an Mbup and to receive the packet and to dispatch the packet up, so for example, have you measured your input traffic coming in and then you just freeing the packet at the very beginning of your path? Actually, in my implementation, I don't use a device driver because it's replaced by Netmap. So you're using Netmap? Yes, yes, yes, yes, yes. Okay. Yeah. Other question? About IPv6, and this implementation, our user DXR, so DXR is only for the IPv4, so packet IO part can accelerate the rate, even if the IPv6, but the loading or lookup part is not efficient to the IPv6. How many rows do you have in the loading table for the test? This measurement is less than 10, so the loading table size is minimized less than 10. So that's why I need to, in the future, I need to, the loading table scalability and the relation to the performance. Okay. Thank you.
...  have video,  say something...  Hello? Say something, girl.  Okay. So, what I said before, so, so, so I just explained about the Excel and, and  so, the Excel creates the very compact art data structure to, based on the,  the OS defaults about X3, so they reuse the data and they create a compact data  and the data structures is to fit our into the CPU cache hybalonarchies, so it  can accelerate the virtual cap speed and so, and this time our, I ported our DXR  patch, our is created for our, the free VSE 8, it's a little bit old, so I ported the  original one to the, to fit into the free VSE 12 and the DXR feed lookup  function is called instead IP found boot and this is our experimental set up.  There's the two machines are, and the one machine is installed on my modified  kernel and the OS, the base OS is free VSE 12 and the CPU and the network  interface card is very common one, our Intel CPU and engineering, it's  Tangier Nick, so there's the two machine and connected to the back to back with  Tangier Nick and the other machines are generated a packet using package gem and  measure the packet rates provided by the router, it's very, very simple and the  measurement metric are is, the package size is 60 bytes, are short packets  exclude CRC and traffic has their same destination so it become the  maximum cash FB capacity and routing size is minimized, are less than 10 entries  and this is the result of the performance measurement, so when we, the  defaults are previously for a packet 1.43 million packet per second and when we  replace the two parts, this line, and the XR, are this performance  up to the 2.43 million packet per second but the M switch itself has a  potential to, has a potential to forwarding the packets are over 10 million  packet per second because it doesn't go up the kernel so we can see there's a  huge gap, still huge gap between our, when we forward the packet to the kernel or not,  so, so to find the ways there are bottleneck, we measure the, the latency  more details, okay, so this is the other measurement to know the bottleneck and  the measurement methodology is a little bit tricky are using M switch and the M  switch is are the hard-coded to forward the packet to an intended output so it  means that when the packet returns are to the M switch layer, it is anyway if the  packet corrupt or not, or it is anyway are send to the other nick, so, so the  received packets are on the packet generator machine are and measure the,  measure the packets that traffic late and then we can measure the time from the  packet alive at the nick and the, the returning point, so the table are summarize  the results and the point is even if, so here's the, the which function takes the  time and the first is L2 input and the second is L3 input, the third one is  L3 output and the last one is L2 output, so some of these results are the point is  even if our mutual gap are cost is minimized, L2 input and output of routine  takes about over 200, 215 nanoseconds and make, make slow down the performance, okay,  so our measurement confirms that the packet IO and the mutual gap are not  extensive anymore and anymore, so the point is that input and L2 input and  output part, okay, so from these are the first design and the measurement we can  say the three things and the packet IO and the mutual gap is not, is not a  bottleneck anymore and L2 input and output part are expensive, so for further  speeding up or we have to bypass the expensive this part, okay, so from this  takeaway we found the, we placed the packet IO and mutual gap are not enough to speed up, so then we renewed the design, so this is the additional, this is the  renewed design and implementation, so the basic idea is our co-editionary bypass  expensive code are at the both input and output path, okay, so this time we focus  on IPv4 forwarding and the proposal new methods to bypass the, the other  features, so for avoiding our input part of the Ethernet processing we  implement a filtering feature to allow the packet which matches the rule of this  time IPv4 and can shortcut the Ethernet input and the directory goes to the L3  input routine, so this red arrow indicates the path path and the black one is the  forward path, so this is input, input bypass part, so the other is output bypass part, so to  improve the performance of our Ethernet output routine, reducing the  frequency to our access to Mac address table is important, so L2 cap has already  optimized by Qo internal casting mechanisms, but our target speed is,  requires nanoseconds a level saving and the cost of L2 cap is not trivial  according to our measurement, so, so we extended the DxR feed to cast Ethernet  header information and to Ethernet information for the next hop, okay, and  so, and the first packets are needed to have to go to the slow, block, block a low  path, but if the second, subsequent packet has the same destination, it can use the  cache data in the DxR feed, so if the packet has the same destination, it can  bypass the IPh output routine and directly back to the M-suit layer, and this cache  information is varied within the batch M-suit process, so this is the new design  implementation, and the next is the evaluation, the machines, the experiment  is about as almost same as the previous measurement, and but this time the packet  size is varies from the short packet to the large packet, and the other is the same, okay,  so this is the result of various packet size, so the graph describes the  throughput of an IP forwarding of different modules with a six variety of  packet size, so we compare previously default forwarding plan to our  implementation, so X-axis is the packet size and  and to, and to know the, to know the which improvements effects are the  performance, we create and run the full version of our implementation, so the  all of our implementation are equipped the M-suit and DxR, and the difference  between our Ds are four modules is the level of the bypass, okay, so no bypass,  the below the no bypass indicates this implementation has the M-suit and DxR,  but no input or output bypass, and the green one has the input bypass and  output bypass means it has only the output bypass, and the pink one is three  bypassing the input and output part, so the native free VSD and and the most  most right part gray bar means the line rate with the packet size, okay, so the  native free VSD and the no bypass records the line rate at the large packet  size, or 1k or something, but our implementation reach the line rate with  our 252, and also in a short packet, our three input plus output bypass can reach  over 5 million packet per second, it is which is a factor of 3.8  improvement of warding the warding speed, so the other rough reports are the  relation between the throughput and the batch size in M-switch, the batch size  varies from 1 to 1,024, we need two different packet size, and so we  investigate the batch size effect on throughput, because M-switch paper shows  are the throughput of M-switch affected by the batch size, and so we expected  that our implementation is also affected by the batch size, okay, because not  only the M-switch itself can be affected by the batch size, but we are setting the  cache aligning time with the batch size, okay, so the throughput increases with  growing the batch size, and it reach its peak to 256 or 512, and not our change  after that, so this shape is almost the same as in the M-switch paper, so  M-switch paper, so we need more investigation, and analyze and  investigate, but the cost of our cache could be, it's not our high-impact  to the performance, okay, so from these results, we can say that our new  implementation reduced over the 80% of L2 input and also L2 output routine, and  our implementation accelerates the default 3BSD IP4 in late by a factor of 3.8,  and the reach over 5 million packet per second with minimal size of packet, and  the default is 1.45, so the cache creation cost appears relatively cheap, but this  part is needed, the further analysis, okay, in a future work, about future work, so  this measurement is basically on the single core performance, so I need to  our more measurement with multi-core scalability, and also need to our  scalability to the loading table entry, and also I want to do some  contribution to the 3BSD itself, so here's the conclusion, so my research  started to the problem that the user space networking has many  problems in practice, so that my approach is the fast kernel-based packet  loading, and in this approach we take three measures of design, and replace the  packet IO and the L2GAP part, and then on the bypass expensive routine at the both  input and output part, so this approach reaches 3.8  times speed up to the default 3BSD, and they increase the  packet loading rate to the 5 million packet per second, so this results  demonstrated the possibility of kernel-based packet loading, okay, so I  will show the demo, so the first demo is very simple performance test of my  L2GAP, and the second one is to show the compatibility to our normal  routing sockets, so in the L2GAP, my modified kernel, so running in  the application are the BGP demo, top of my L2GAP, and the shorter this BGP  demo are Vox, so the first demo, this is the packet generate, can you see it?  Okay, this machine is our packet generator machine, and with using with the  packet gen, this is the process of sending the packets, so this process  creates the packet are in 1488 million packet per second, so sending a packet  to the routing machine right now, and the other, so in the diatonic, here's the  receiving process, and the receiving packet with 1.6 million packet per second,  and so we attach the router machine, in the router machine we attach the  NIC to the M switch, and then sending machine also creates the 10-year  line-length traffic, and this time we attach the NIC to the M switch, so the  router machine can forward over 5 million packet per second this time, okay.  And the other demo is our, using the BGP demo are in top of this router, so  in the router machine, the BGP demo is running, and the other side of the machine,  here's the emulator too are creating the BGP demo as a neighbor, so this emulator  creates the BGP demo and sending the routing information to the to my router,  so in this our machine can receive the BGP information, like this,  okay, good update, any, any, okay, thanks for everything, any questions when you comment?  I wonder if you did a better job than you think you did, so have you met, you mentioned, you keep mentioning actual 10-year line-length, what's the,  did you measure our previous PIX driver and how fast the single key can receive, and receive up to the point with your switch runs, and just drop it as an interest to switch?  So you're measuring everything from the above the driver, but have you measured how long it takes the driver to allocate an Mbup and to receive the packet and to dispatch the packet up, so for example, have you measured your input traffic coming in and then you just freeing the packet at the very beginning of your path?  Actually, in my implementation, I don't use a device driver because it's replaced by Netmap.  So you're using Netmap?  Yes, yes, yes, yes, yes.  Okay.  Yeah.  Other question?  About IPv6, and this implementation, our user DXR, so DXR is only for the IPv4, so packet IO part can accelerate the rate, even if the IPv6, but the loading or lookup part is not efficient to the IPv6.  How many rows do you have in the loading table for the test?  This measurement is less than 10, so the loading table size is minimized less than 10.  So that's why I need to, in the future, I need to, the loading table scalability and the relation to the performance.  Okay.  Thank you.
NanakoMomiyama__IPForwardingFastpathBSDCan2017_0_131.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_132_2675.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_2676_4331.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_4332_7619.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_7620_9203.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_9204_9887.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_9888_10235.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_10236_12011.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_12012_14651.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_14652_14819.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_14820_23135.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23136_23255.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23256_23303.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23304_23459.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23460_23507.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23508_23939.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_23940_25559.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_25560_26495.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_26496_26651.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_26652_27191.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_27192_28247.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_28248_28343.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_28344_28547.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_28548_28811.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_28812_28883.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_28884_29099.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_29100_29579.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_29580_31355.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_31356_32639.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_32640_32747.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_32748_32831.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_32832_35771.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_35772_35855.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_35856_35903.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_35904_36479.png NanakoMomiyama__IPForwardingFastpathBSDCan2017_36480_36696.png