<story><title>Serving Netflix Video at 400Gb/s on FreeBSD [pdf]</title><url>https://people.freebsd.org/~gallatin/talks/euro2021.pdf</url></story><parent_chain><item><author>dragontamer</author><text>&amp;gt; We&amp;#x27;re free to choose whatever egress NIC we want&lt;p&gt;Wait, I got lost again... You say you can &amp;quot;output on any egress NIC&amp;quot;. So all four egress NICs have access to the TLS encryption keys and are cooperating through the FreeBSD kernel to get this information?&lt;p&gt;Is there some kind of load-balancing you&amp;#x27;re doing on the machine? Trying to see which NIC has the least amount of traffic and routing to the least utilized NIC?</text></item><item><author>drewg123</author><text>Thanks for the feeback.  I was hoping that the other words on the slide (LACP and bonding) would give enough context.&lt;p&gt;I&amp;#x27;m afraid that my presentation didn&amp;#x27;t really have room to dive much into LACP.  I briefly said something like the following when giving that slide:&lt;p&gt;Basically, the LACP link partner (router) hashes traffic consistently across the multiple links in the LACP bundle, using a hash of its choosing (typically an N-tuple, involving IP address and TCP port).  Once it has selected a link for that connection, that connection will always land on that link on ingress (unless the LACP bundle changes in terms of links coming and going).  We&amp;#x27;re free to choose whatever egress NIC we want (it does not need to be the same NIC the connection entered on).  The issue is that there is no way for us to tell the router to move the TCP connection from one NIC to another (well, there is in theory, but our routers can&amp;#x27;t do it)&lt;p&gt;I hope that helps</text></item><item><author>dragontamer</author><text>I eventually figured it out. But I would suggest maybe giving a brief 1-slide thingy on &amp;quot;lagg&amp;quot;, and link aggregation? (Maybe it was clear from the presentation, but I only see the slides here soooooo...)&lt;p&gt;I&amp;#x27;m not the best at network infrastructure, though I&amp;#x27;m more familiar with NUMA stuff. So I was trying to figure out how you only got 1-IP address on each box despite 4-ports across 2-nics.&lt;p&gt;I assume some Linux &amp;#x2F; Windows devops people are just not as familiar with FreeBSD tools like that!&lt;p&gt;EDIT: Now that I think of it: maybe a few slides on how link-aggregation across NICs &amp;#x2F; NUMA could be elaborated upon further? I&amp;#x27;m frankly not sure if my personal understanding is correct. I&amp;#x27;m imagining how TCP-connections are fragmented into IP-packets, and how those packets may traverse your network, and how they get to which NUMA node... and it seems really more complex to me than your slides indicate? Maybe this subject will take more than just one slide?</text></item><item><author>drewg123</author><text>These are the slides from my EuroBSDCon presentation. AMA</text></item></parent_chain><comment><author>myrandomcomment</author><text>LACP (which is a control protocol) is a standard feature supported forever on switches. To put it simply both the server and switch see a single port (fake) that has physical ethernet members and a hashing algorithm puts the traffic on that physical ethernet link based on the selected hash. The underlying sw&amp;#x2F;hw picks which physical link to put the packet on. The input to the hash in picking the link can be src&amp;#x2F;dst (for all items listed here) MAC, port, IP addr. LACP handles the negotiation of what that hash is between the ends and also hands a link failure &amp;quot;hey man, something broke, we have one less link now). For any given single flow it will hash to the same link. So a for example in a 4x10G lag (also called a port-channel in networking speak) max bandwidth for a single flow would be 10g the max of a single member. In an ideal world the hashing would be perfectly balanced however it is possible to have a set of flows all hash to the same link. Hope that helps.</text></comment>
<story><title>Serving Netflix Video at 400Gb/s on FreeBSD [pdf]</title><url>https://people.freebsd.org/~gallatin/talks/euro2021.pdf</url></story><parent_chain><item><author>dragontamer</author><text>&amp;gt; We&amp;#x27;re free to choose whatever egress NIC we want&lt;p&gt;Wait, I got lost again... You say you can &amp;quot;output on any egress NIC&amp;quot;. So all four egress NICs have access to the TLS encryption keys and are cooperating through the FreeBSD kernel to get this information?&lt;p&gt;Is there some kind of load-balancing you&amp;#x27;re doing on the machine? Trying to see which NIC has the least amount of traffic and routing to the least utilized NIC?</text></item><item><author>drewg123</author><text>Thanks for the feeback.  I was hoping that the other words on the slide (LACP and bonding) would give enough context.&lt;p&gt;I&amp;#x27;m afraid that my presentation didn&amp;#x27;t really have room to dive much into LACP.  I briefly said something like the following when giving that slide:&lt;p&gt;Basically, the LACP link partner (router) hashes traffic consistently across the multiple links in the LACP bundle, using a hash of its choosing (typically an N-tuple, involving IP address and TCP port).  Once it has selected a link for that connection, that connection will always land on that link on ingress (unless the LACP bundle changes in terms of links coming and going).  We&amp;#x27;re free to choose whatever egress NIC we want (it does not need to be the same NIC the connection entered on).  The issue is that there is no way for us to tell the router to move the TCP connection from one NIC to another (well, there is in theory, but our routers can&amp;#x27;t do it)&lt;p&gt;I hope that helps</text></item><item><author>dragontamer</author><text>I eventually figured it out. But I would suggest maybe giving a brief 1-slide thingy on &amp;quot;lagg&amp;quot;, and link aggregation? (Maybe it was clear from the presentation, but I only see the slides here soooooo...)&lt;p&gt;I&amp;#x27;m not the best at network infrastructure, though I&amp;#x27;m more familiar with NUMA stuff. So I was trying to figure out how you only got 1-IP address on each box despite 4-ports across 2-nics.&lt;p&gt;I assume some Linux &amp;#x2F; Windows devops people are just not as familiar with FreeBSD tools like that!&lt;p&gt;EDIT: Now that I think of it: maybe a few slides on how link-aggregation across NICs &amp;#x2F; NUMA could be elaborated upon further? I&amp;#x27;m frankly not sure if my personal understanding is correct. I&amp;#x27;m imagining how TCP-connections are fragmented into IP-packets, and how those packets may traverse your network, and how they get to which NUMA node... and it seems really more complex to me than your slides indicate? Maybe this subject will take more than just one slide?</text></item><item><author>drewg123</author><text>These are the slides from my EuroBSDCon presentation. AMA</text></item></parent_chain><comment><author>martyvis</author><text>The whole paper is discussing bottlenecks, path optimisation between resources, and impacts of those on overall throughout. It&amp;#x27;s not a simple loadbalancing question being answered</text></comment>