<commit_before>---
authors: <NAME> <<EMAIL>>
state: draft
discussion: https://github.com/joyent/rfd/issues?q=%22RFD+118%22
---

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright (c) 2018 Joyent, Inc.
-->

# RFD 118 MAC Checksum Offload Extensions

When sending and receiving networking traffic, there are many different
checksums that need to be calculated and verified in a packet. These
checksums are often included in the L2, L3, and L4 headers. An example
of an L2 checksum is the Ethernet CRC. An example of an L3 header
checksum is the IPv4 header checksum. An example of an L4 header
checksum is the TCP header checksum.

Specifically, we're concerned with L3 and L4 checksums. The L2 Ethernet
checksum is usually taken care of by hardware. Not all protocols have
checksums. Consider ARP or LLDP. Both of these are L3 protocols, but
neither have checksums. Similarly, while IPv4 has an L3 header checksum,
IPv6 does not!

We're specifically concerned about the following L3 protocols with
checksums:

* IPv4

We're specifically concerned about the following L4 protocols with
checksums:

* TCP
* UDP
* ICMP
* SCTP

Traditionally, these checksums were all calculated by the networking
stack. However, hardware has offered support to offload the calculation
of these checksums. This RFD covers some additional checksum extensions
offered by modern hardware that we'd like to take advantage of.

## Checksums and the GLDv3

The GLDv3 is the framework that networking device drivers implement. The
GLDv3 has a notion of a capabilities, one of which, `MAC_CAPAB_HCKSUM`
is used to identify whether or not hardware supports checksum offload
features. A driver indicates what features it supports for transmit.
When the networking stack is assembling packets, it will use this
information to determine whether a checksum will be calculated in
software or if it will be calculated in hardware.

Drivers can currently indicate the following flags. These flags are
documented in [`mac(9E)`](http://illumos.org/man/9e/mac): 

* `HCKSUM_IPHDRCKSUM`: This indicates that the calculation of a checksum
of an IPv4 header can be offloaded

* `HCKSUM_INET_FULL_V4`: This indicates that the calculation of a
checksum of an IPv4 packet's L4 header can be offloaded.

* `HCKSUM_INET_FULL_V6`: This indicates that the calculation of a
checksum of an IPv6 packet's L4 header can be offloaded.

* `HCKSUM_INET_PARTIAL`: The hardware requires that a psuedo-header is
calculated for it when performing L4 offloads.

Later, when a driver transmits a packet it calls the
[`mac_hcksum_get(9F)`](http://illumos.org/man/9f/mac_hcksum_set) function
to retrieve the set of operations it must perform on the packet. Based
on these options, it generally sets bit in the packet's descriptor
header.

When a driver receives data, it looks at the header and indicates which
checksum features were triggered in a packet's descriptor header.
Depending on the type of packet, nothing may have been done or it's
possible that checksums were validated and errors were determined. Once
the driver understands what has occurred, it calls the
[`mac_hcksum_set(9F)`](http://illumos.org/man/9f/mac_hcksum_get)
function and sets checksum information as documented in the manual page.

Aside from this, everything else is handled internally in the OS and by
consumers such as DLS, IP, viona, etc.

### Limitations

Today there are two major issues that we want to bring up.

1. Support for VXLAN encapsulation checksum offload
2. Clarifying the `HCKSUM_INET_FULL_*` values.

The first issue is that we want to have the ability to leverage hardware
checksum offload for encapsulation protocols. Modern hardware has the
ability to peer into and understand VXLAN, GRE, and Geneve encapsulation
headers and calculate the inner and outer L3 and L4 header checksums.
This can be used on transmit to calculate the checksum in hardware and
on receive to verify that the checksum is correct in hardware.

This is complicated by the fact that in the VXLAN spec, it is considered
optional to set a checksum in the UDP header. However, if one does, then
the checksum must be validated.

The second issue is somewhat related. Today, when we have the
`HCKSUM_INET_FULL_V4` and `HCKSUM_INET_FULL_V6` flags we do not indicate
the set of L4 protocols that are considered by these. The issue comes to
the forefront when hardware supports different sets from what the
operating system expects.

The four L4 checksums that come up most often are the ones we mentioned
earlier: TCP, UDP, ICMP, and SCTP. Of these, the OS implies that those
values mean TCP, UDP, and ICMP. However, while most hardware supports
TCP and UDP checksum offload, the same is not true for ICMP. For
example, drivers like `qede` do not support ICMPv6 checksum offload.

## Proposals

### Dealing with `HCKSUM_INET_FULL_*`

To deal with the issue around what does `HCKSUM_INET_FULL_V4` and
`HCKSUM_INET_FULL_V6` mean, I propose that we clarify things in the
manual page to specifically say that these only refer to the TCP and UDP
protocols.

This does mean that there are some drivers that currently provide ICMP
checksum support that will be missing out. However, the percentage of
traffic that is ICMP and the corresponding cost for such drivers will be
minimal. In addition, we should add two new flags to cover ICMP.

* `HCKSUM_INET_FULL_ICMPv4`

This indicates that hardware supports checksum offload for IPv4 ICMP
packets.

* `HCKSUM_INET_FULL_ICMPv6`

This indicates that hardware supports checksum offload for IPv6 ICMP
packets.

These could also be extended to cover SCTP in a similar fashion by
adding `HCKSUM_INET_FULL_SCTPv4` and `HCKSUM_INET_FULL_SCTPv6`. Or if
some future IPv4 protocol has hardware checksum offload support, we can
indicate it in a similar way.

One nice side effect of this is that it does not impact the set of flags
that we need to offer to use with `mac_hcksum_get(9F)` or
`mac_hcksum_set(9F)`.


### Encapsulation Checksum Offload

There are a few more challenges with checksum offload features. First,
the following hardware table of features is useful to share. This table
table surveys devices that are currently on the market and what they
support for VXLAN encapsulation. Inner L4 only refers to TCP and UDP.

| Driver | Inner L3 | Inner L4 | Outer L3 | Outer L4 | Notes |
|--------|----------|----------|----------|----------|-------|
| bnxt | yes | yes | yes | yes | no illumos driver |
| cxgbe | yes* | yes* | yes* | yes* | Pending verification |<commit_msg>add text clarifying HCKSUM_INET_PARTIAL handling<commit_after>---
authors: <NAME> <<EMAIL>>
state: draft
discussion: https://github.com/joyent/rfd/issues?q=%22RFD+118%22
---

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright (c) 2018 Joyent, Inc.
-->

# RFD 118 MAC Checksum Offload Extensions

When sending and receiving networking traffic, there are many different
checksums that need to be calculated and verified in a packet. These
checksums are often included in the L2, L3, and L4 headers. An example
of an L2 checksum is the Ethernet CRC. An example of an L3 header
checksum is the IPv4 header checksum. An example of an L4 header
checksum is the TCP header checksum.

Specifically, we're concerned with L3 and L4 checksums. The L2 Ethernet
checksum is usually taken care of by hardware. Not all protocols have
checksums. Consider ARP or LLDP. Both of these are L3 protocols, but
neither have checksums. Similarly, while IPv4 has an L3 header checksum,
IPv6 does not!

We're specifically concerned about the following L3 protocols with
checksums:

* IPv4

We're specifically concerned about the following L4 protocols with
checksums:

* TCP
* UDP
* ICMP
* SCTP

Traditionally, these checksums were all calculated by the networking
stack. However, hardware has offered support to offload the calculation
of these checksums. This RFD covers some additional checksum extensions
offered by modern hardware that we'd like to take advantage of.

## Checksums and the GLDv3

The GLDv3 is the framework that networking device drivers implement. The
GLDv3 has a notion of a capabilities, one of which, `MAC_CAPAB_HCKSUM`
is used to identify whether or not hardware supports checksum offload
features. A driver indicates what features it supports for transmit.
When the networking stack is assembling packets, it will use this
information to determine whether a checksum will be calculated in
software or if it will be calculated in hardware.

Drivers can currently indicate the following flags. These flags are
documented in [`mac(9E)`](http://illumos.org/man/9e/mac):

* `HCKSUM_IPHDRCKSUM`: This indicates that the calculation of a checksum
of an IPv4 header can be offloaded

* `HCKSUM_INET_FULL_V4`: This indicates that the calculation of a
checksum of an IPv4 packet's L4 header can be offloaded.

* `HCKSUM_INET_FULL_V6`: This indicates that the calculation of a
checksum of an IPv6 packet's L4 header can be offloaded.

* `HCKSUM_INET_PARTIAL`: The hardware requires that a psuedo-header is
calculated for it when performing L4 offloads.

Later, when a driver transmits a packet it calls the
[`mac_hcksum_get(9F)`](http://illumos.org/man/9f/mac_hcksum_set) function
to retrieve the set of operations it must perform on the packet. Based
on these options, it generally sets bit in the packet's descriptor
header.

When a driver receives data, it looks at the header and indicates which
checksum features were triggered in a packet's descriptor header.
Depending on the type of packet, nothing may have been done or it's
possible that checksums were validated and errors were determined. Once
the driver understands what has occurred, it calls the
[`mac_hcksum_set(9F)`](http://illumos.org/man/9f/mac_hcksum_get)
function and sets checksum information as documented in the manual page.

Aside from this, everything else is handled internally in the OS and by
consumers such as DLS, IP, viona, etc.

### Limitations

Today there are two major issues that we want to bring up.

1. Support for VXLAN encapsulation checksum offload
2. Clarifying the `HCKSUM_INET_FULL_V*` and `HCKSUM_INET_PARTIAL` values.

The first issue is that we want to have the ability to leverage hardware
checksum offload for encapsulation protocols. Modern hardware has the
ability to peer into and understand VXLAN, GRE, and Geneve encapsulation
headers and calculate the inner and outer L3 and L4 header checksums.
This can be used on transmit to calculate the checksum in hardware and
on receive to verify that the checksum is correct in hardware.

This is complicated by the fact that in the VXLAN spec, it is considered
optional to set a checksum in the UDP header. However, if one does, then
the checksum must be validated.

The second issue is somewhat related. Today, when we have the
`HCKSUM_INET_FULL_V4`, `HCKSUM_INET_FULL_V6` and `HCKSUM_INET_PARTIAL` flags
we do not indicate the set of L4 protocols that are considered by these.
The issue comes to the forefront when hardware supports different sets from
what the operating system expects.

The four L4 checksums that come up most often are the ones we mentioned
earlier: TCP, UDP, ICMP, and SCTP. Of these, the OS implies that those
values mean TCP, UDP, and ICMP. However, while most hardware supports
TCP and UDP checksum offload, the same is not true for ICMP. For
example, the `qede` driver does not support ICMPv6 checksum offload and
the `i40e` driver does not support partial checksum calculation for ICMP.

## Proposals

### Dealing with `HCKSUM_INET_FULL_V*` and `HCKSUM_INET_PARTIAL`

To deal with the question around what does `HCKSUM_INET_FULL_V4`,
`HCKSUM_INET_FULL_V6` and `HCKSUM_INET_PARTIAL` mean, I propose that we
clarify things in the manual page to specifically say that these only apply
to the TCP and UDP protocols.

This does mean that there are some drivers that currently provide ICMP
checksum support which will be missing out. However, the percentage of
traffic that is ICMP and the corresponding cost for such drivers will be
minimal. In addition, we should add three new flags to cover ICMP.

* `HCKSUM_INET_FULL_ICMPv4`

This indicates that hardware supports checksum offload for IPv4 ICMP
packets.

* `HCKSUM_INET_FULL_ICMPv6`

This indicates that hardware supports checksum offload for IPv6 ICMP
packets.

* `HCKSUM_INET_PARTIAL_ICMP`

This indicates that hardware supports partial checksum offload for ICMP
packets.

These could also be extended to cover SCTP in a similar fashion by
adding `HCKSUM_INET_FULL_SCTPv4`, `HCKSUM_INET_FULL_SCTPv6` and
`HCKSUM_INET_PARTIAL_SCTP`. Or if some future IPv4 protocol has hardware
checksum offload support, we can indicate it in a similar way.

One nice side effect of this is that it does not impact the set of flags
that we need to offer to use with `mac_hcksum_get(9F)` or
`mac_hcksum_set(9F)`.


### Encapsulation Checksum Offload

There are a few more challenges with checksum offload features. First,
the following hardware table of features is useful to share. This table
table surveys devices that are currently on the market and what they
support for VXLAN encapsulation. Inner L4 only refers to TCP and UDP.

| Driver | Inner L3 | Inner L4 | Outer L3 | Outer L4 | Notes |
|--------|----------|----------|----------|----------|-------|
| bnxt | yes | yes | yes | yes | no illumos driver |
| cxgbe | yes* | yes* | yes* | yes* | Pending verification |
{'fraction_non_alphanumeric': 0.05619774627408215, 'fraction_numerical': 0.011341330425299891, 'mean_word_length': 4.371339320577899, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2596679', 'n_tokens_mistral': 4120, 'n_tokens_neox': 3874, 'n_words': 2031}