readme.txt

ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722
================================================================
July 9, 2019

Contents
========

- Prerequisites
- Building and Installation
- Testing
- Configuration
- Interoperability
- Known Issues


Prerequisites
=============

- FreeBSD version 11.2
- Kernel configuration:
    Please add the following kernel configuration options:
	include GENERIC
	options COMPAT_LINUXKPI
	options IPOIB_CM
	options IXL_IW

	nodevice ixl
	nodevice iavf
	Note: IXL_IW is required for FreeBSD-CURRENT branch.
- For the iw_ixl driver to work, an if_ixl driver with iwarp interface
  is required. The interface is available in if_ixl version 1.7.12 or later.
  It should be enabled prior to usage, as the setting is switched off by
  default. To enable iwarp compatibility, add
	hw.ixl.enable_iwarp=1
to
	/boot/loader.conf

The lan driver can be downloaded from
https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-D
river-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD
Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network
Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in
the ixl driver version 1.7.12-k or later.

There are some known issues with the interface on if_ixl-1.7.12. Please
use version 1.7.13 or later.

- fastreg memory mode in krping needs a patch applied to krping.
  Refer to the 'Testing' and 'Known Issues' sections for details.


Building and Installation
=========================

1. Untar ixl-<version>.tar.gz and iw_ixl-<version>.tar.gz

	# tar -xf ixl-<version>.tar.gz
	# tar -xf iw_ixl-<version>.tar.gz

2. Install the if_ixl driver:

	# cd ixl-<version>/src directory
	# make
	# make install

3. Install the iw_ixl driver:

	# cd iw_ixl-<version>/src
	# make clean
	# make IXL_DIR=$PATH_TO_IXL/ixl-<version>/src
	# make install

4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file
   to the directory where manual pages are held on your system. For instance:

	# cp iw_ixl-<version>/doc/iw_ixl.4.gz /usr/share/man/man4/

For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow
the instruction from point 3 but ensure the correct path to if_ixl source
folder is supplied. For instance:
	IXL_DIR=/usr/src/sys/dev/ixl/


Testing
-------
1. To load the iw_ixl driver, call:

	# kldload iw_ixl

   If if_ixl is not already loaded, the system will load it on its own.
   Please remember to add
	hw.ixl.enable_iwarp=1
   to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl
   driver has the iwarp interface enabled.

2. To validate the load of the driver, check:

	# sysctl -a | grep infiniband

   A number of sys.class.infiniband should appear, provided at least one
   port of the X722 is up.

3. The source code for krping software is provided with the kernel in
/usr/src/sys/contrib/rdma/krping/. To compile the software, change directory
to /usr/src/sys/modules/rdma/krping/ and invoke the following:

	# make clean
	# make
	# make install

4. Start krping server on one machine:

	# echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping
5. Connect client from another machine:

	# echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping


Configuration
=============
The following sysctl options are visible:
  - hw.iw_ixl.max_ceq
	determines the maximum number of msix vectors available to the driver
	for CEQ usage.
  - hw.iw_ixl.debug
	defines level of debug messages.
  - hw.iw_ixl.mpa_version
	shows the current MPA version used.

The max_ceq setting may be changed by adding:
	hw.iw_ixl.max_ceq=$value
to /boot/loader.conf file. The final number of CEQ is evaluated depending
on the available msix vectors, number of cpu cores, and hardware limits.

If max_ceq=0, the value is ignored.

The debug setting may be changed either by adding:
	hw.iw_ixl.debug=$value
to the /boot/loader.conf file or by calling
	sysctl hw.iw_ixl.debug=$value

The mpa_version may be changed by adding:
	hw.iw_ixl.mpa_version=$value
to the /boot/loader.conf file.


Interoperability
================

To interoperate with Chelsio iWARP devices:

1. Load the ixl driver with parameter mpa_version set to 1. Add the line:
	hw.iw_ixl.mpa_version=1
to /boot/loader.conf

2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameter dack_mode set to 0.


Known Issues
============

- Loopback is not supported.
- MTU changes are not supported.
- IPv6 is not supported.
- MW memory mode is not supported.
- MR memory mode supports only single buffer.
- The function ib_cq_resize is not supported.
- The max number of registered cq, qp, pd or mr reported by the device may
  differ from the actual number of registrations achievable.
- A kernel crash may occur when trying to run krping without ensuring that the
  two machines are able to ping each other.
- A kernel crash may occur when trying to load the iw_ixl driver when
  hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13).
- A kernel crash may occur when loading the iw_ixl driver on a card that is
  supported by if_ixl driver, but does not have iWARP capability (fixed with
  if_ixl 1.7.13).
- Krping with fastreg memory mode will not work unless some changes are made
  to krping. To work around the issue, modify the krping_rdma_rkey function
  such that, in the case of FASTREG memory mode, the ib_post_send function
  with &cd->invalidate_wr parameter is not called during the first run of
  the function.


Support
=======
For general information, go to the Intel support website at:
http://www.intel.com/support/

If an issue is identified with the released source code on a supported kernel
with a supported adapter, email the specific information related to the issue
to [email protected]


Copyright(c) 2017-2019 Intel Corporation.


Trademarks
==========
Intel is a trademark or registered trademark of Intel Corporation or its
subsidiaries in the United States and/or other countries.

* Other names and brands may be claimed as the property of others.