| ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722 | |
| ================================================================ | |
| July 9, 2019 | |
| Contents | |
| ======== | |
| - Prerequisites | |
| - Building and Installation | |
| - Testing | |
| - Configuration | |
| - Interoperability | |
| - Known Issues | |
| Prerequisites | |
| ============= | |
| - FreeBSD version 11.2 | |
| - Kernel configuration: | |
| Please add the following kernel configuration options: | |
| include GENERIC | |
| options COMPAT_LINUXKPI | |
| options IPOIB_CM | |
| options IXL_IW | |
| nodevice ixl | |
| nodevice iavf | |
| Note: IXL_IW is required for FreeBSD-CURRENT branch. | |
| - For the iw_ixl driver to work, an if_ixl driver with iwarp interface | |
| is required. The interface is available in if_ixl version 1.7.12 or later. | |
| It should be enabled prior to usage, as the setting is switched off by | |
| default. To enable iwarp compatibility, add | |
| hw.ixl.enable_iwarp=1 | |
| to | |
| /boot/loader.conf | |
| The lan driver can be downloaded from | |
| https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-D | |
| river-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD | |
| Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network | |
| Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in | |
| the ixl driver version 1.7.12-k or later. | |
| There are some known issues with the interface on if_ixl-1.7.12. Please | |
| use version 1.7.13 or later. | |
| - fastreg memory mode in krping needs a patch applied to krping. | |
| Refer to the 'Testing' and 'Known Issues' sections for details. | |
| Building and Installation | |
| ========================= | |
| 1. Untar ixl-<version>.tar.gz and iw_ixl-<version>.tar.gz | |
| # tar -xf ixl-<version>.tar.gz | |
| # tar -xf iw_ixl-<version>.tar.gz | |
| 2. Install the if_ixl driver: | |
| # cd ixl-<version>/src directory | |
| # make | |
| # make install | |
| 3. Install the iw_ixl driver: | |
| # cd iw_ixl-<version>/src | |
| # make clean | |
| # make IXL_DIR=$PATH_TO_IXL/ixl-<version>/src | |
| # make install | |
| 4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file | |
| to the directory where manual pages are held on your system. For instance: | |
| # cp iw_ixl-<version>/doc/iw_ixl.4.gz /usr/share/man/man4/ | |
| For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow | |
| the instruction from point 3 but ensure the correct path to if_ixl source | |
| folder is supplied. For instance: | |
| IXL_DIR=/usr/src/sys/dev/ixl/ | |
| Testing | |
| ------- | |
| 1. To load the iw_ixl driver, call: | |
| # kldload iw_ixl | |
| If if_ixl is not already loaded, the system will load it on its own. | |
| Please remember to add | |
| hw.ixl.enable_iwarp=1 | |
| to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl | |
| driver has the iwarp interface enabled. | |
| 2. To validate the load of the driver, check: | |
| # sysctl -a | grep infiniband | |
| A number of sys.class.infiniband should appear, provided at least one | |
| port of the X722 is up. | |
| 3. The source code for krping software is provided with the kernel in | |
| /usr/src/sys/contrib/rdma/krping/. To compile the software, change directory | |
| to /usr/src/sys/modules/rdma/krping/ and invoke the following: | |
| # make clean | |
| # make | |
| # make install | |
| 4. Start krping server on one machine: | |
| # echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping | |
| 5. Connect client from another machine: | |
| # echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping | |
| Configuration | |
| ============= | |
| The following sysctl options are visible: | |
| - hw.iw_ixl.max_ceq | |
| determines the maximum number of msix vectors available to the driver | |
| for CEQ usage. | |
| - hw.iw_ixl.debug | |
| defines level of debug messages. | |
| - hw.iw_ixl.mpa_version | |
| shows the current MPA version used. | |
| The max_ceq setting may be changed by adding: | |
| hw.iw_ixl.max_ceq=$value | |
| to /boot/loader.conf file. The final number of CEQ is evaluated depending | |
| on the available msix vectors, number of cpu cores, and hardware limits. | |
| If max_ceq=0, the value is ignored. | |
| The debug setting may be changed either by adding: | |
| hw.iw_ixl.debug=$value | |
| to the /boot/loader.conf file or by calling | |
| sysctl hw.iw_ixl.debug=$value | |
| The mpa_version may be changed by adding: | |
| hw.iw_ixl.mpa_version=$value | |
| to the /boot/loader.conf file. | |
| Interoperability | |
| ================ | |
| To interoperate with Chelsio iWARP devices: | |
| 1. Load the ixl driver with parameter mpa_version set to 1. Add the line: | |
| hw.iw_ixl.mpa_version=1 | |
| to /boot/loader.conf | |
| 2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameter dack_mode set to 0. | |
| Known Issues | |
| ============ | |
| - Loopback is not supported. | |
| - MTU changes are not supported. | |
| - IPv6 is not supported. | |
| - MW memory mode is not supported. | |
| - MR memory mode supports only single buffer. | |
| - The function ib_cq_resize is not supported. | |
| - The max number of registered cq, qp, pd or mr reported by the device may | |
| differ from the actual number of registrations achievable. | |
| - A kernel crash may occur when trying to run krping without ensuring that the | |
| two machines are able to ping each other. | |
| - A kernel crash may occur when trying to load the iw_ixl driver when | |
| hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13). | |
| - A kernel crash may occur when loading the iw_ixl driver on a card that is | |
| supported by if_ixl driver, but does not have iWARP capability (fixed with | |
| if_ixl 1.7.13). | |
| - Krping with fastreg memory mode will not work unless some changes are made | |
| to krping. To work around the issue, modify the krping_rdma_rkey function | |
| such that, in the case of FASTREG memory mode, the ib_post_send function | |
| with &cd->invalidate_wr parameter is not called during the first run of | |
| the function. | |
| Support | |
| ======= | |
| For general information, go to the Intel support website at: | |
| http://www.intel.com/support/ | |
| If an issue is identified with the released source code on a supported kernel | |
| with a supported adapter, email the specific information related to the issue | |
| to [email protected] | |
| Copyright(c) 2017-2019 Intel Corporation. | |
| Trademarks | |
| ========== | |
| Intel is a trademark or registered trademark of Intel Corporation or its | |
| subsidiaries in the United States and/or other countries. | |
| * Other names and brands may be claimed as the property of others. | |