.. _set_up_omnigraph_tests:

Set Up OmniGraph Tests
======================

Much of the testing for a node can be set up through the .ogn file's :ref:`test section<ogn_defining_automatic_tests>`,
however there are a number of situations that require more detailed setup or more flexible checking than the
automatically generated tests can provide. For such tests you will want to hook into Kit's testing system.

Some examples of these situations are when you need to check for attributes that are one-of a set of allowed results
rather than a fixed value, where you need to check node or attribute information that is more than just the current
value, where you need to call utility scripts to set up desired configurations, or when your results depend on some
external condition such as the graph state.

Described here are some best practices for writing such tests. This is only meant to describe setting up Python
regression tests that use the Kit extensions to the Python `unittest` modeule. It is not meant to describe setting up
C++ unit tests.

.. note::

    For clarity a lot of recommended coding practices, like adding docstrings to all classes, functions, and modules,
    or checking for unexpected exceptions in order to provide better error messages, are not followed. Please do use
    them when you write your actual tests though.

Locating The Tests
------------------

The Kit extension system uses an automatic module recognition algorithm to detect directories in which test cases
may be found. In particular it looks for **.tests** submodules. So if your Python module is named `omni.graph.foo` it
will check the contents of the module `omni.graph.foo.tests`, if it exists, and attempt to find files containing
classes derived from `unittest.TestCase`, or the Kit version `omni.kit.test.AsyncTestCase`.

The usual way of structuring extensions provides a directory structure that looks like this:

.. code-block:: text

    omni.graph.foo/
        python/
            tests/
                test_some_stuff.py

The *tests/* subdirectory would be linked into the build using these lines in your *premake5.lua* file inside the
python project definition:

.. code-block:: lua

    add_files("python/tests", "python/tests/*.py")
    repo_build.prebuild_link {
        { "python/tests", ogn.python_tests_target_path },
    }

This creates a link that creates a **.tests** submodule for your extension.

The files containing tests should all begin with the prefix *test_*.

Creating A Test Class
---------------------

OmniGraph tests have some shared setUp and tearDown operations so the easiest way to set up your test class is to
have it derive from the derived test case class that implements them:

.. code-block:: python

    import omni.graph.core.tests as ogts

    class TestsForMe(ogts.OmniGraphTestCase):
        pass

This will ensure your tests are part of the Kit regression test system. The parent class will define some temporary
settings as required by OmniGraph, and will clear the scene when the test is done so as not to influence the results
of the test after it (barring any other side effects the test itself causes of course).

.. tip::

    Although the name of the class is not significant it's helpful to prefix it with *Tests* to make it easy to identify.

Specialized SetUp And TearDown
++++++++++++++++++++++++++++++

If you have some other setUp or tearDown functions you wish to perform you do it in the usual Pythonic manner:

.. code:: python

    import omni.graph.core.tests as ogts

    class TestsForMe(ogts.OmniGraphTestCase):
        async def setUp(self):
            await super().setUp()
            do_my_setup()

        async def tearDown(self):
            do_my_teardown()
            await super().tearDown()

.. note::

    The tests are async so both they and the setUp/tearDown will be "awaited" when running. This was done to facilitate
    easier access to some of the Kit async functions, though normally you want to ensure your test steps run
    sequentially.

Adding Tests
++++++++++++

Tests are added in the usual way for the Python `unittest` framework, by creating a function with the prefix *test_*.
As the tests are all awaited your functions should be async.

.. code:: python

    import omni.graph.core.tests as ogts

    class TestsForMe(ogts.OmniGraphTestCase):
        async def setUp(self):
            await super().setUp()
            do_my_setup()

        async def tearDown(self):
            do_my_teardown()
            await super().tearDown()

        async def test1(self):
            self.assertTrue(run_first_test())

        async def test2(self):
            self.assertTrue(run_second_test())

How you divide your test bodies is up to you. You'll want to balance the slower performance of repetitive setup
against the isolation of specific test conditions.

Your best friend in setting up test conditions is the :ref:`og.Controller<howto_omnigraph_controller>` class. It
provides a lot of what you will need for setting up and inspecting your graph, nodes, and attributes.

Here is a simple example that will create an add node with one constant input, and one input supplied from another
node that will test to make sure that results are correct over a set of inputs. It uses several concepts from the
controller to illustrate its use.

.. code-block:: python

    import omni.graph.core as og
    import omni.graph.core.tests as ogts


    class TestsForMe(ogts.OmniGraphTestCase):

        async def test_add(self):

            keys = og.Controller.Keys
            (graph, nodes, _, _) = og.Controller.edit("/TestGraph", {
                keys.CREATE_NODES: [
                    ("Add", "omni.graph.nodes.Add"),
                    ("ConstInt", "omni.graph.nodes.ConstantInt"),
                ],
                keys.CONNECT: ("ConstInt.inputs:value", "Add.inputs:a"),
                keys.SET_VALUES: [("ConstInt.inputs:value", 3), ("Add.inputs:b", {"type": "int", "value": 1})]
            })

            # Get controllers attached to the attributes since they will be accessed in a loop
            b_view = og.Controller(attribute=og.Controller.attribute("inputs:b", nodes[0]))
            sum_view = og.Controller(attribute=og.Controller.attribute("outputs:sum", nodes[0]))

            # Test configurations are pairs of (input_b_value, output_sum_expected)
            test_configurations = [
                ({"type": "int", "value": 1}, 4),
                ({"type": "int", "value": -3}, 0),
                ({"type": "int", "value": 1000000}, 1000003),
            ]
            for b_value, sum_expected in test_configurations:
                b_view.set(b_value)
                # Before checking computed values you must ensure the graph has evaluated
                await og.Controller.evaluate(graph)
                self.assertAlmostEqual(sum_expected, sum_view.get())

Expected Errors
+++++++++++++++

When writing tests, it can be desirable to test error conditions. However, this may cause errors to be 
displayed to the console, which can cause tests to fail when running in batch mode. 

One way to have the testing system ignore these errors is to prepend a additional text to the error line.

.. code-block:: python

    # write to the console, skipping the newline to prepend the expected error message 
    print("[Ignore this error/warning] ", end ="") 
    self.assertFalse(self.function_that_causes_error())

Then, in your packages extensions.toml file, tell the test system to ignore error output when the prepended output 
occurs.

.. code-block:: rst

    [[test]]
    stdoutFailPatterns.exclude = [
        # Exclude messages which say they should be ignored
        "*Ignore this error/warning*",
    ]

This allows for the test to specify which errors should be ignored without ignoring all errors with the same 
output in the entire package, or by disabling all error checking from your test class. 

Executing Your Tests
--------------------

Now that your tests are visible to Kit they will be automatically run by TeamCity. To run them yourself locally you
have two options.

Batch Running Tests
+++++++++++++++++++

All tests registered in the manner described above will be added to a batch file that runs
all tests in your extension. You can find this file at *$BUILD/tests-omni.graph.foo.{bat|sh}*. Executing this file
will run your tests in a minimal Kit configuration. (It basically loads your extension and all dependent extensions,
including the test manager.) Look up the documentation on ``omni.kit.test`` for more information on how the tests
can be configured.

Test Runner
+++++++++++

Kit's *Test Runner* window is a handy way to interactively run one or more of your tests at a finer granularity through
a UI. By default none of the tests are added to the window, however, so you must add a line like this to your
user configuration file, usually in **~/Documents/Kit/shared/user.toml**, to specify which tests it should load:

.. code-block:: toml

    exts."omni.kit.test".includeTests = ["omni.graph.foo.*"]


Debugging Your Tests
--------------------

Whether you're tracking down a bug or engaging in test-driven-development eventually you will end up in a situation
where you need to debug your tests.

One of the best tools is to use the script editor and the UI in Kit to inspect and manipulate your test scene.
While the normal OmniGraph test case class deletes the scene at the end of the test you can make a temporary change
to instead use a variation of the test case that does not do that, so that you can examine the failing scene.

.. code-block:: python
    :emphasize-lines: 3

    import omni.graph.core.tests as ogts

    class TestsForMe(ogts.OmniGraphTestCaseNoClear):
        pass

Running Test Batch Files With Attached Debuggers
++++++++++++++++++++++++++++++++++++++++++++++++

You're probably familiar with the debugging extensions ``omni.kit.debug.python`` and ``omni.kit.debug.vscode``,
with which you can attach debuggers to running versions of Kit.

If you are running the test batch file, however, you have to ensure that these extensions are running as part of the
test environment. To do so you just need to add these flags to your invocation of the test ``.bat`` file:

.. code-block:: bash

    $BUILD/tests-omni.graph.foo.bat --enable omni.kit.debug.vscode --enable omni.kit.debug.python

This will enable the extra extensions required to attach the debuggers while the test scripts are running. Of course
you still have to manually attach them from your IDE in the same way you usually do.

.. note::

    As of this writing you might see two ``kit.exe`` processes when attaching a C++ debugger to a test script. The
    safest thing to do is attach to both of them.

The .bat files also support the flag ``-d`` flag to wait for the debugger to attach before executing. If you're running
a debug cut this probably won't be necessary as the startup time is ample for attaching a debugger.
