<commit_before>..  SPDX-License-Identifier: BSD-3-Clause
    Copyright(c) 2021 Intel Corporation.

.. include:: <isonum.txt>

IDXD DMA Device Driver
======================

The ``idxd`` dmadev driver provides a poll-mode driver (PMD) for Intel\ |reg|
Data Streaming Accelerator `(Intel DSA)
<https://software.intel.com/content/www/us/en/develop/articles/intel-data-streaming-accelerator-architecture-specification.html>`_.
This PMD can be used in conjunction with Intel\ |reg| DSA devices to offload
data operations, such as data copies, to hardware, freeing up CPU cycles for
other tasks.

Hardware Requirements
----------------------

The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the
presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma``
will show all the DMA devices on the system, including IDXD supported devices.
Intel\ |reg| DSA devices, are currently (at time of writing) appearing
as devices with type “0b25”, due to the absence of pci-id database entries for
them at this point.

Compilation
------------

For builds using ``meson`` and ``ninja``, the driver will be built when the
target platform is x86-based. No additional compilation steps are necessary.

Device Setup
-------------

Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers,
such as ``vfio-pci``. Both are supported by the IDXD PMD.

Intel\ |reg| DSA devices using IDXD kernel driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To use an Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured.
The `accel-config <https://github.com/intel/idxd-config>`_ utility library can be used for configuration.

.. note::
        The device configuration can also be done by directly interacting with the sysfs nodes.
        An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py``
        included in the driver source directory.

There are some mandatory configuration steps before being able to use a device with an application.
The internal engines, which do the copies or other operations,
and the work-queues, which are used by applications to assign work to the device,
need to be assigned to groups, and the various other configuration options,
such as priority or queue depth, need to be set for each queue.

To assign an engine to a group::

        $ accel-config config-engine dsa0/engine0.0 --group-id=0
        $ accel-config config-engine dsa0/engine0.1 --group-id=1

To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used.
However, the work queues also need to be configured depending on the use case.
Some configuration options include:

* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously.
* priority: WQ priority between 1 and 15. Larger value means higher priority.
* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device.
* type: WQ type (kernel/mdev/user). Determines how the device is presented.
* name: identifier given to the WQ.

Example configuration for a work queue::

        $ accel-config config-wq dsa0/wq0.0 --group-id=0 \
           --mode=dedicated --priority=10 --wq-size=8 \
           --type=user --name=dpdk_app1

Once the devices have been configured, they need to be enabled::

        $ accel-config enable-device dsa0
        $ accel-config enable-wq dsa0/wq0.0

Check the device configuration::

        $ accel-config list

Devices using VFIO/UIO drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The HW devices to be used will need to be bound to a user-space IO driver for use.
The ``dpdk-devbind.py`` script can be used to view the state of the devices
and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``.
For example::

	$ dpdk-devbind.py -b vfio-pci 6a:01.0

Device Probing and Initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will
be found as part of the device scan done at application initialization time without
the need to pass parameters to the application.

For Intel\ |reg| DSA devices, DPDK will automatically configure the device with the
maximum number of workqueues available on it, partitioning all resources equally
among the queues.
If fewer workqueues are required, then the ``max_queues`` parameter may be passed to
the device driver on the EAL commandline, via the ``allowlist`` or ``-a`` flag e.g.::

	$ dpdk-test -a <b:d:f>,max_queues=4
<commit_msg>doc: improve configuration examples in idxd guide<commit_after>..  SPDX-License-Identifier: BSD-3-Clause
    Copyright(c) 2021 Intel Corporation.

.. include:: <isonum.txt>

IDXD DMA Device Driver
======================

The ``idxd`` dmadev driver provides a poll-mode driver (PMD) for Intel\ |reg|
Data Streaming Accelerator `(Intel DSA)
<https://software.intel.com/content/www/us/en/develop/articles/intel-data-streaming-accelerator-architecture-specification.html>`_.
This PMD can be used in conjunction with Intel\ |reg| DSA devices to offload
data operations, such as data copies, to hardware, freeing up CPU cycles for
other tasks.

Hardware Requirements
----------------------

The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the
presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma``
will show all the DMA devices on the system, including IDXD supported devices.
Intel\ |reg| DSA devices, are currently (at time of writing) appearing
as devices with type “0b25”, due to the absence of pci-id database entries for
them at this point.

Compilation
------------

For builds using ``meson`` and ``ninja``, the driver will be built when the
target platform is x86-based. No additional compilation steps are necessary.

Device Setup
-------------

Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers,
such as ``vfio-pci``. Both are supported by the IDXD PMD.

Intel\ |reg| DSA devices using IDXD kernel driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To use an Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured.
The `accel-config <https://github.com/intel/idxd-config>`_ utility library can be used for configuration.

.. note::
        The device configuration can also be done by directly interacting with the sysfs nodes.
        An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py``
        included in the driver source directory.

There are some mandatory configuration steps before being able to use a device with an application.
The internal engines, which do the copies or other operations,
and the work-queues, which are used by applications to assign work to the device,
need to be assigned to groups, and the various other configuration options,
such as priority or queue depth, need to be set for each queue.

To assign an engine to a group::

        $ accel-config config-engine dsa0/engine0.0 --group-id=0

To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used.
However, the work queues also need to be configured depending on the use case.
Some configuration options include:

* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously.
* priority: WQ priority between 1 and 15. Larger value means higher priority.
* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device.
* type: WQ type (kernel/mdev/user). Determines how the device is presented.
* name: identifier given to the WQ.

Example configuration for a work queue::

        $ accel-config config-wq dsa0/wq0.0 --group-id=0 \
           --mode=dedicated --priority=10 --wq-size=8 \
           --max-batch-size=512 --type=user --name=dpdk_app1

Once the devices have been configured, they need to be enabled::

        $ accel-config enable-device dsa0
        $ accel-config enable-wq dsa0/wq0.0

Check the device configuration::

        $ accel-config list

Every Intel\ |reg| DSA instance supports multiple queues and each should be similarly configured.
As a further example, the following set of commands will configure and enable 4 queues on instance 0,
giving each an equal share of resources::

        # configure 4 groups, each with one engine
        accel-config config-engine dsa0/engine0.0 --group-id=0
        accel-config config-engine dsa0/engine0.1 --group-id=1
        accel-config config-engine dsa0/engine0.2 --group-id=2
        accel-config config-engine dsa0/engine0.3 --group-id=3

        # configure 4 queues, putting each in a different group, so each
        # is backed by a single engine
        accel-config config-wq dsa0/wq0.0 --group-id=0 --type=user --wq-size=32 \
            --priority=10 --max-batch-size=1024 --mode=dedicated --name=dpdk_app1
        accel-config config-wq dsa0/wq0.1 --group-id=1 --type=user --wq-size=32 \
            --priority=10 --max-batch-size=1024 --mode=dedicated --name=dpdk_app1
        accel-config config-wq dsa0/wq0.2 --group-id=2 --type=user --wq-size=32 \
            --priority=10 --max-batch-size=1024 --mode=dedicated --name=dpdk_app1
        accel-config config-wq dsa0/wq0.3 --group-id=3 --type=user --wq-size=32 \
            --priority=10 --max-batch-size=1024 --mode=dedicated --name=dpdk_app1

        # enable device and queues
        accel-config enable-device dsa0
        accel-config enable-wq dsa0/wq0.0 dsa0/wq0.1 dsa0/wq0.2 dsa0/wq0.3


Devices using VFIO/UIO drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The HW devices to be used will need to be bound to a user-space IO driver for use.
The ``dpdk-devbind.py`` script can be used to view the state of the devices
and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``.
For example::

	$ dpdk-devbind.py -b vfio-pci 6a:01.0

Device Probing and Initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will
be found as part of the device scan done at application initialization time without
the need to pass parameters to the application.

For Intel\ |reg| DSA devices, DPDK will automatically configure the device with the
maximum number of workqueues available on it, partitioning all resources equally
among the queues.
If fewer workqueues are required, then the ``max_queues`` parameter may be passed to
the device driver on the EAL commandline, via the ``allowlist`` or ``-a`` flag e.g.::

	$ dpdk-test -a <b:d:f>,max_queues=4

{'fraction_non_alphanumeric': 0.11106934786694231, 'fraction_numerical': 0.015034767900770532, 'mean_word_length': 3.5078356628547227, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1752481', 'n_tokens_mistral': 3127, 'n_tokens_neox': 3017, 'n_words': 1359}