<commit_before>.. _conftest:

The Pytest hooks
================

One of the many useful features of ``pytest`` is its easy extensibility. The
``pytest`` documentation has a full description of
`Working with non-python tests <http://pytest.org/latest/example/nonpython.html#non-python-tests>`_
on which this C unit test extension is heavily based.

At its simplest, ``pytest`` supports the creation of hooks in a file called
``conftest.py`` that can be placed in the root folder where we wish to run the
tests. This allows us to define, in Python, code to find test files, run them,
and collect the results. The nice thing about this is that we can create a
hook layer between ``pytest`` and the C unit tests without changing code in
either one.

The ``conftest.py`` file is actually a per-directory file, so you can have
different files for different test directories if required.


The conftest.py file
--------------------

First let's look at the C unit test ``conftest.py`` in its entirety:

.. literalinclude:: ../../conftest.py
   :lines: 7-


This is less than 100 lines of Python including comments but it provides a lot
of functionality. However, before we look at that functionality let's first
look at the main parts of the ``conftest.py`` program.




Finding test files
------------------

The first function in the code is a ``pytest`` hook function called
``pytest_collect_file()``::

    def pytest_collect_file(parent, path):
        if path.ext == ".c" and path.basename.startswith("test_"):
            return CTestFile(path, parent)

This collects files based on any rule that we wish to write. In this case it
collects files that look like
``test_*.c`` but we could make the rule as specific as we wanted.

Once a file of the correct type is found a ``pytest.Node`` object of the
corresponding type is created and returned. In our case this is a
``CTestFile`` object which is derived from the ``pytest.File`` class.


The CTestFile Class
-------------------

The ``CTestFile`` object that we instantiated in the ``pytest_collect_file()``
hook inherits from the ``pytest.File`` class which in turn inherits from
``pytest.Collection`` and ``pytest.Node``. The ``Node`` class and its
subclasses have a ``collect()`` method which returns ``pytest.Items`` objects::

    class CTestFile(pytest.File):

        def collect(self):
            # Returns pytest.Items.

These ``pytest`` hierarchy and method concepts are explained in more detail in
the
`Working with plugins and conftest files <http://pytest.org/latest/plugins.html>`_
Pytest documentation.

Depending on the types of tests that are being run the collected items might be
individual test results or even test cases that are being stage to run.

However, in our case we are going to take a simplified approach that lends
itself to statically compiled test cases such as C unit tests:

* For each ``test_something.c`` file assume there is a ``test_something``
  executable.
* Run the ``test_something`` executable and capture the output.
* Parse the Pass/Fail results and any available metadata.
* Return each test results as a python dict with information that can be used
  for reporting.

Collecting the C unit test data
-------------------------------

So with this methodology in mind the first step is to run the collected C unit
tests and capture the output.

As stated above we are going to assume that the C unit tests are structured so
that for each ``test_something.c`` source file there is a ``test_something``
executable. This is a reasonably assumption based on the way most test suites
are laid out but it may not hold for specific implementations where, for
example, multiple ``.c`` files might be compiled to object files and linked
into a single test runner executable. For cases like that more sophisticated
``pytest_collect_file()`` can be used.

For the sake of this exercise we are going to assume that the unit test
executables are tied into a build system and have already been built via
``make``, ``make test`` or something similar.

In ``conftest.py`` the following code runs the executable that corresponds to
the ``.c`` file and captures the output::

    def collect(self):

        test_exe = str(self.fspath)[0:-2]
        test_output = subprocess.check_output(test_exe)

Extracting the test data
------------------------

The captured output will look something like the following in our case::

    [PASS] test_basic_strings.c:test_some_strings():8

    [FAIL] test_basic_strings.c:test_some_strings():9
      [TST] ASSERT_EQUAL_STR(foo, bar)
      [EXP] This is foo
      [GOT] This is bar


We clean up the unit test output and remove non test data lines as follows::
<commit_msg>Update conftest.rst<commit_after>.. _conftest:

The Pytest hooks
================

One of the many useful features of ``pytest`` is its easy extensibility. The
``pytest`` documentation has a full description of
`Working with non-python tests <http://pytest.org/latest/example/nonpython.html#non-python-tests>`_
on which this C unit test extension is heavily based.

At its simplest, ``pytest`` supports the creation of hooks in a file called
``conftest.py`` that can be placed in the root folder where we wish to run the
tests. This allows us to define, in Python, code to find test files, run them,
and collect the results. The nice thing about this is that we can create a
hook layer between ``pytest`` and the C unit tests without changing code in
either one.

The ``conftest.py`` file is actually a per-directory file, so you can have
different files for different test directories if required.


The conftest.py file
--------------------

First let's look at the C unit test ``conftest.py`` in its entirety:

.. literalinclude:: ../../conftest.py
   :lines: 7-


This is less than 100 lines of Python including comments but it provides a lot
of functionality. However, before we look at that functionality let's first
look at the main parts of the ``conftest.py`` program.




Finding test files
------------------

The first function in the code is a ``pytest`` hook function called
``pytest_collect_file()``::

    def pytest_collect_file(parent, path):
        if path.ext == ".c" and path.basename.startswith("test_"):
            return CTestFile(path, parent)

This collects files based on any rule that we wish to write. In this case it
collects files that look like
``test_*.c`` but we could make the rule as specific as we wanted.

Once a file of the correct type is found a ``pytest.Node`` object of the
corresponding type is created and returned. In our case this is a
``CTestFile`` object which is derived from the ``pytest.File`` class.


The CTestFile Class
-------------------

The ``CTestFile`` object that we instantiated in the ``pytest_collect_file()``
hook inherits from the ``pytest.File`` class which in turn inherits from
``pytest.Collection`` and ``pytest.Node``. The ``Node`` class and its
subclasses have a ``collect()`` method which returns ``pytest.Items`` objects::

    class CTestFile(pytest.File):

        def collect(self):
            # Returns pytest.Items.

These ``pytest`` hierarchy and method concepts are explained in more detail in
the
`Working with plugins and conftest files <http://pytest.org/latest/plugins.html>`_
Pytest documentation.

Depending on the types of tests that are being run the collected items might be
individual test results or even test cases that are being stage to run.

However, in our case we are going to take a simplified approach that lends
itself to statically compiled test cases such as C unit tests:

* For each ``test_something.c`` file assume there is a ``test_something``
  executable.
* Run the ``test_something`` executable and capture the output.
* Parse the Pass/Fail results and any available metadata.
* Return each test results as a python dict with information that can be used
  for reporting.

Collecting the C unit test data
-------------------------------

So with this methodology in mind the first step is to run the collected C unit
tests and capture the output.

As stated above we are going to assume that the C unit tests are structured so
that for each ``test_something.c`` source file there is a ``test_something``
executable. This is a reasonably assumption based on the way most test suites
are laid out but it may not hold for specific implementations where, for
example, multiple ``.c`` files might be compiled to object files and linked
into a single test runner executable. For cases like that more sophisticated
``pytest_collect_file()`` can be used.

For the sake of this exercise we are going to assume that the unit test
executables are tied into a build system and have already been built via
``make``, ``make test`` or something similar.

In ``conftest.py`` the following code runs the executable that corresponds to
the ``.c`` file and captures the output::

    def collect(self):

        test_exe = os.path.splitext(str(self.fspath))[0]
        test_output = subprocess.check_output(test_exe)

Extracting the test data
------------------------

The captured output will look something like the following in our case::

    [PASS] test_basic_strings.c:test_some_strings():8

    [FAIL] test_basic_strings.c:test_some_strings():9
      [TST] ASSERT_EQUAL_STR(foo, bar)
      [EXP] This is foo
      [GOT] This is bar


We clean up the unit test output and remove non test data lines as follows::

{'fraction_non_alphanumeric': 0.09813928761297182, 'fraction_numerical': 0.001594896331738437, 'mean_word_length': 3.661050545094153, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7214500', 'n_tokens_mistral': 2532, 'n_tokens_neox': 2484, 'n_words': 1313}