<commit_before>.. sectnum::

How to use Deckard
==================
.. contents::

Deckard runs one or more binaries in isolated network which is described by so-called *scenario*.
There are four components in play:

- Deckard itself (test orchestrator)
- binary under test (your own)
- configuration for the binary (generated by Deckard from *template* and *YaML configuration*, i.e. ``.j2`` and ``.yaml`` files)
- environment description and test data (Deckard *scenario*, i.e. ``.rpl`` file)

It is easy to run tests if everything is already prepared and running tests gets harder
as number of components you have to prepare yourself raises.

Let's start with the easiest case:

First run
---------
Easiest way to run Deckard is using one of the prepared Shell scripts in Deckard repository (``{kresd,unbound,pdns}_run.sh`` for Knot Resolver, Unbound and PowerDNS Recursor respectively).

Deckard uses `pytest` to generate and run the tests as well as collect the results.
Output is therefore generated by `pytest` as well (``.`` for passed test, ``F`` for failed test and ``s`` for skipped test) and will look something like this:

.. code-block::

   $ ./kresd_run.sh
   deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/black_data.rpl',
   qmin=False, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional':
   ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
   SKIPPED [  0%]

   […many lines later…]

   deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/world_mx_nic_www.rpl',
   qmin=None, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'],
   'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] PASSED [100%]

   ======================= 275 passed, 97 skipped in 316.61s (0:05:16) =======================

.. note:: There is a lot of tests skipped because we run them with query minimization both on and off and some of the scenarios work only with query minimization on (or off respectively). For details see `Scenario guide#Configuration <scenario_guide.rst#configuration-config-end>`_.

Command line arguments
----------------------
As mentioned above we use `py.test` to run the tests so all possible command line arguments for the ``*run.sh`` scripts can be seen by running ``py.test -h`` in the root of Deckard repository.

Here is a list of the most useful ones:

- ``-n number`` – runs the testing in parallel with ``number`` of processes (this requires `pytest-xdist` to be installed)
- ``-k EXPRESSION`` – only run tests which match the given substring expression (e.g. ``./kresd_run -k "world_"`` will only run the scenarios with `world_` in their file name.
- ``--collectonly`` – only print the names of selected tests, no tests will be run
- ``--log-level DEBUG`` – print all debug information for failed tests
- ``--scenarios path`` – specifies where to look for `.rpl` files (``sets/resolver`` is the default)

YaML configuration
------------------
All ``*_run.sh`` scripts internally call the ``run.sh`` script and pass command line arguments to it. For example:

.. code-block::

   # running ./kresd_run.sh -n 4 -k "iter_" will result in running
   ./run.sh --config configs/kresd.yaml  -n 4 -k "iter_"

As you can see, path to YaML configuration file is passed to ``run.sh``. You can edit one of the prepared ones stored in `configs/` or write your own.

Commented contents of ``kresd.yaml`` follows:

.. code-block:: yaml

  programs:
  - name: kresd             # path to binary under test
    binary: kresd
    additional:             # list additional parameters for binary under test (e.g. path to configuration files)
      - -f
      - "1"                 # CAUTION: All parameters must be strings.
    conncheck: True         # wait until TCP port 53 accepts connections (enabled by default)
    templates:
      - template/kresd.j2   # list of Jinja2_ template files to generate configuration files
    configs:
      - config              # list of names of configuration files to be generated from Jinja2_ templates
  noclean: True             # optional, do not remove working dir after a successful test

- 'configs' files will be generated from respective files in 'templates' list
- i.e. the first file in 'configs' list is the result of processing of the first file from 'templates' list and so on
- generated files are stored in a new working directory created by Deckard for each binary

Most often it is sufficient to use these files for basic configuration changes. Read next section for details about config file templates.

Running multiple binaries
^^^^^^^^^^^^^^^^^^^^^^^^^
You can specify multiple programs to run in the YaML configuration. Deckard executes all binaries using parameters from the file. This is handy for testing interoperability of multiple binaries, e.g. when one program is configured as DNS recursor and other program is using it as forwarder.

The YAML file contains **ordered** list of binaries and their parameters. Deckard will send queries to the binary listed first.
<commit_msg>doc: don't recommend the deprecated -f parameter<commit_after>.. sectnum::

How to use Deckard
==================
.. contents::

Deckard runs one or more binaries in isolated network which is described by so-called *scenario*.
There are four components in play:

- Deckard itself (test orchestrator)
- binary under test (your own)
- configuration for the binary (generated by Deckard from *template* and *YaML configuration*, i.e. ``.j2`` and ``.yaml`` files)
- environment description and test data (Deckard *scenario*, i.e. ``.rpl`` file)

It is easy to run tests if everything is already prepared and running tests gets harder
as number of components you have to prepare yourself raises.

Let's start with the easiest case:

First run
---------
Easiest way to run Deckard is using one of the prepared Shell scripts in Deckard repository (``{kresd,unbound,pdns}_run.sh`` for Knot Resolver, Unbound and PowerDNS Recursor respectively).

Deckard uses `pytest` to generate and run the tests as well as collect the results.
Output is therefore generated by `pytest` as well (``.`` for passed test, ``F`` for failed test and ``s`` for skipped test) and will look something like this:

.. code-block::

   $ ./kresd_run.sh
   deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/black_data.rpl',
   qmin=False, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional':
   ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
   SKIPPED [  0%]

   […many lines later…]

   deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/world_mx_nic_www.rpl',
   qmin=None, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'],
   'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] PASSED [100%]

   ======================= 275 passed, 97 skipped in 316.61s (0:05:16) =======================

.. note:: There is a lot of tests skipped because we run them with query minimization both on and off and some of the scenarios work only with query minimization on (or off respectively). For details see `Scenario guide#Configuration <scenario_guide.rst#configuration-config-end>`_.

Command line arguments
----------------------
As mentioned above we use `py.test` to run the tests so all possible command line arguments for the ``*run.sh`` scripts can be seen by running ``py.test -h`` in the root of Deckard repository.

Here is a list of the most useful ones:

- ``-n number`` – runs the testing in parallel with ``number`` of processes (this requires `pytest-xdist` to be installed)
- ``-k EXPRESSION`` – only run tests which match the given substring expression (e.g. ``./kresd_run -k "world_"`` will only run the scenarios with `world_` in their file name.
- ``--collectonly`` – only print the names of selected tests, no tests will be run
- ``--log-level DEBUG`` – print all debug information for failed tests
- ``--scenarios path`` – specifies where to look for `.rpl` files (``sets/resolver`` is the default)

YaML configuration
------------------
All ``*_run.sh`` scripts internally call the ``run.sh`` script and pass command line arguments to it. For example:

.. code-block::

   # running ./kresd_run.sh -n 4 -k "iter_" will result in running
   ./run.sh --config configs/kresd.yaml  -n 4 -k "iter_"

As you can see, path to YaML configuration file is passed to ``run.sh``. You can edit one of the prepared ones stored in `configs/` or write your own.

Commented contents of ``kresd.yaml`` follows:

.. code-block:: yaml

  programs:
  - name: kresd             # path to binary under test
    binary: kresd
    additional:             # list additional parameters for binary under test (e.g. path to configuration files)
      - --noninteractive
    conncheck: True         # wait until TCP port 53 accepts connections (enabled by default)
    templates:
      - template/kresd.j2   # list of Jinja2_ template files to generate configuration files
    configs:
      - config              # list of names of configuration files to be generated from Jinja2_ templates
  noclean: True             # optional, do not remove working dir after a successful test

- 'configs' files will be generated from respective files in 'templates' list
- i.e. the first file in 'configs' list is the result of processing of the first file from 'templates' list and so on
- generated files are stored in a new working directory created by Deckard for each binary

Most often it is sufficient to use these files for basic configuration changes. Read next section for details about config file templates.

Running multiple binaries
^^^^^^^^^^^^^^^^^^^^^^^^^
You can specify multiple programs to run in the YaML configuration. Deckard executes all binaries using parameters from the file. This is handy for testing interoperability of multiple binaries, e.g. when one program is configured as DNS recursor and other program is using it as forwarder.

The YAML file contains **ordered** list of binaries and their parameters. Deckard will send queries to the binary listed first.

{'fraction_non_alphanumeric': 0.12119426438813592, 'fraction_numerical': 0.006187389510901591, 'mean_word_length': 3.767322097378277, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3781692', 'n_tokens_mistral': 2918, 'n_tokens_neox': 2812, 'n_words': 1327}