.. _ogn_build_conversion:

Tutorial - OmniGraph Build Environment Conversion
=================================================

These steps will set up the extension to enable building the .ogn file generation and access the generated tests,
include file, and Python file. This only has to be done once per extension, after which any newly converted or
created nodes will be automatically picked up by the build.

Once the build has been set up any existing nodes can be converted to the .ogn format using the steps described
in :ref:`ogn_node_conversion`.

.. note::

    If you are building outside Kit you will have to add access to the OGN processing tools in your main premake5.lua
    file. See :ref:`ogn_building_outside_kit` for more details.

.. contents:: Build Conversion Steps

Step 1 : File Structure
-----------------------

These are a set of guidelines for file locations and contents that make things consistent, which in turn makes the
lives of developers easier when looking at different sections of code. See :ref:`ogn_recommended_directory_structure`
for more information on why the directory is structured this way.

Here is the file structure of the extension containing the jiggle node before conversion:

.. code-block:: text

    omni.anim.jiggle/
    ├── bindings/
    |   └── BindingsPython.cpp
    ├── config/
    |   └── extension.toml
    ├── plugins/
    |   ├── Deformer.cpp
    |   ├── IJiggleDeformer.h
    |   └── JiggleDeformer.cpp
    ├── premake5.lua
    └── python/
        ├── __init__.py
        └── scripts/
            ├── commands.py
            ├── extension.py
            ├── jiggleDeformer.py
            ├── kit_utils.py
            ├── menu.py
            └── test.py

Python Scripts
++++++++++++++
First thing to note is that we'd like to keep the utility scripts in the ``python`` subdirectory separate from
any tests. The directories can be rearranged with a simple git command:

.. code-block:: bash

    git mv python/scripts/tests.py python/tests

That leaves the python directory with this structure:

.. code-block:: text

    python/
    ├── __init__.py
    └── scripts/
        ├── commands.py
        ├── extension.py
        ├── jiggleDeformer.py
        ├── kit_utils.py
        └── menu.py
    └── tests/
            └── test.py

.. note::

    The reason that ``scripts/`` is a separate subdirectory is to allow the Python code to be linked as a whole
    in the build rather than copied. That way you can drop any file into the ``scripts/`` directory in your source
    tree, edit it in place, and have the extension system hot-reload it so that the new script is immediately
    available while the app is running.


Nodes
+++++

There is no restriction on how many nodes an extension can support though in this example there is only one. Each
node will have two files; one with the suffix ``.ogn``, containing the node description, and one with the suffix
``.cpp``, containing the node class.

It is helpful to keep the node naming and location consistent. The convention established is to use **Ogn** as a
prefix to the class name, and to put the node files in a ``nodes/`` directory, parallel to ``plugins/``. In the
example here the means creating a new directory and renaming the node as follows:

.. code-block:: bash

    mkdir nodes
    git mv plugins/JiggleDeformer.cpp nodes/OgnJiggleDeformer.cpp
    touch nodes/OgnJiggleDeformer.ogn

The last command creates a placeholder for the file to be populated later.

At this point all of the files are in their final location and the directory structure changes look like this:

.. code:: text

    omni.anim.jiggle/                           omni.anim.jiggle/
    ├── bindings/                               ├── bindings/
    |   └── BindingsPython.cpp                  |   └── BindingsPython.cpp
    ├── config/                                 ├── config/
    |   └── extension.toml                      |   └── extension.toml
    ├── nodes/                                  ├── plugins/
    |   ├── OgnJiggleDeformer.cpp               |   ├── JiggleDeformer.cpp
    |   └── OgnJiggleDeformer.ogn               |   |
    ├── plugins/                                |   |
    |   ├── Deformer.cpp                        |   ├── Deformer.cpp
    |   └── IJiggleDeformer.h                   |   └── IJiggleDeformer.h
    ├── premake5.lua                            ├── premake5.lua
    └── python/                                 └── python/
        ├── __init__.py                             ├── __init__.py
        ├── scripts/                                └─── scripts/
        |   ├── commands.py                             ├── commands.py
        |   ├── extension.py                            ├── extensions.py
        |   ├── jiggleDeformer.py                       ├── jiggleDeformer.py
        |   ├── kit_utils.py                            ├── kit_utils.py
        |   ├── menu.py                                 ├── menu.py
        └── tests/                                      |
            └── test.py                                 └── test.py


Step 2 : Fix Python Imports
---------------------------
Since the python files have move the python import directives have to be updated to point to the new locations.
Most of them are okay as they are relative, only those whose relative position will change need to be adjusted
such as the import of the bindings from ``extension.py``. In addition, the generated ``ogn/`` subdirectory must
also be imported if you have any nodes implemented in Python.

.. code-block:: python
    :linenos:
    :emphasize-lines: 2,6-7

    import omni.ext
    from .bindings._anim_jiggle import *

    from .menu import DeformerMenu
    from .commands import *
    from .test import *


    class Extension(omni.ext.IExt):
        def on_startup(self):
            self._ext = acquire_interface()
            self._menu = DeformerMenu(self._ext)

        def on_shutdown(self):
            release_interface(self._ext)
            self._menu.on_shutdown()
            self._menu = None

The bindings will now be at the same relative level so the first highlighted line changes to:

.. code-block:: python

    from .bindings._anim_jiggle import *

and the tests will be automatically detected via a mechanism to be added later, :ref:`ogn_test_detection`, so the
second highlighted line will be deleted.

Also, due to the changing locations, the import of ``extension.py`` in the file ``__init__.py`` changes from this:

.. code-block:: python

    from .scripts.extension import *

to this

.. code-block:: python

    from .extension import *


Step 3 : Add Build Support
--------------------------
The OmniGraph nodes, and the .ogn files in particular, require some build support in order to be processed
correctly.

.. note:: This process assumes you are using the
          `*extension 2.0* <https://drive.google.com/file/d/1R35g67nzD8tS-NUI0wiAFNyDWoCiklZk/view>`_ plugin interface

Here are the original contents of the Jiggle extension's ``premake5.lua`` file, with the areas that will be modified
highlighted:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 3-8,11,14,17-23,25,36,40,42-45,47-65

    local ext = get_current_extension_info()

    -- change name of plugins here
    local namespace = "omni/anim/jiggle"

    -- lines below can be copied for other plugins
    local python_source_path = "python"
    local plugin_source_path = "plugins"

    ext.group = "animation"
    project_ext(ext)


    project_ext_plugin(ext, "omni.anim.jiggle.plugin")

        add_files("impl", "plugins")
        add_files("iface", "%{root}/include/"..namespace)

        exceptionhandling "On"
        rtti "On"
        removeflags { "NoPCH" } -- to speed up USD includes compiling
        filter {}
        includedirs {
            "%{target_deps}/nv_usd/%{config}/include",
            "%{target_deps}/rtx_plugins/include",
            "%{target_deps}/cuda/include"
        }
        filter { "system:linux" }
            removeflags { "FatalCompileWarnings", "UndefinedIdentifiers" }
            includedirs { "%{target_deps}/python/include/python3.7m" }
        filter { "system:windows" }
            libdirs { "%{target_deps}/tbb/lib/intel64/vc14" }
        filter {}
        libdirs { "%{target_deps}/nv_usd/%{config}/lib" }
        links {"ar","vt", "gf", "pcp", "sdf", "arch", "usd", "tf", "usdUtils", "usdGeom", "usdSkel", "omni.usd"}
        filter {}


    project_ext_bindings {
        ext = ext,
        project_name = "omni.anim.jiggle.python",
        module = "_anim_jiggle",
        src = "bindings",
        target_subdir = namespace.."/bindings"
    }
        vpaths {
            ['bindings'] = "bindings/*.*",
            ['python'] = python_source_path.."/*.py",
            ['python/scripts'] = python_source_path.."/scripts/*.py"
        }
        files {
            python_source_path.."/*.py",
            python_source_path.."/scripts/*.py"
        }

        includedirs { plugin_source_path }

        repo_build.prebuild_link {
            { "python/scripts", ext.target_dir.."/"..namespace.."/scripts" },
        }


        repo_build.prebuild_copy {
            { "python/*.py", ext.target_dir.."/"..namespace },
        }

Set up Helper Variables With Paths
++++++++++++++++++++++++++++++++++

The first change is to add a few helper variables containing directories of interest (fixing the generic
copy/pasted comments while we are in there). Similar to the *get_current_extension_info()* helper function there
is *get_ogn_project_information()* that defines a table of useful information for OGN projects, assuming a
common layout. See below for details on what it returns.

.. code-block:: lua
    :linenos:
    :emphasize-lines: 3-6,9

    local ext = get_current_extension_info()

    -- Helper variable containing standard configuration information for projects containing OGN files
    local ogn = get_ogn_project_information(ext, "omni/anim/jiggle")

    -- Grouping also tells the name of the premake file that calls this one - $TOP/premake5-{ext.group}.lua
    ext.group = "animation"

    -- Set up common project variables
    project_ext( ext )

The usage of the ogn table will be illuminated below, as will the function definition that was
loaded from omni.graph.tools.

This is extracted from the documentation of the *get_ogn_project_information()* function:

.. code-block:: lua

    -- These are the generated table contents:
    --    Source Paths
    --      bindings_path : Subdirectory containing the C++ implementing Python bindings
    --      docs_path     : Subdirectory containing the documentation to publish
    --      nodes_path    : Subdirectory containing the C++ OmniGraph nodes
    --      plugin_path   : Subdirectory containing the C++ implementing the plugin
    --      python_path   : Subdirectory containing the Python code
    --    Target Paths
    --      python_target_path       : Path to the Python import directory of the extension
    --      python_tests_target_path : Path to the Python tests import directory of the extension
    --      bindings_module          : Name of the Python bindings module for the extension
    --      bindings_target_path     : Pyton bindings module import path, relative to root of the extension's Python tree
    --    Project Information
    --      import_path    : Subdirectory for both Python and C++ import/include
    --      namespace      : Prefix for project names (import_path with "/" replace by ".")
    --      plugin_project : Name of the project that builds the C++ code
    --      ogn_project    : Name of the project that processes the .ogn files
    --      python_project : Name of the project that processes the Python code


Add the .ogn files to the project
+++++++++++++++++++++++++++++++++

Although this project doesn't use the .ogn files directly it's useful to have them available in the project file.
Comments are also added while in here to make it clear what each section is doing.

.. code-block:: lua
    :linenos:
    :emphasize-lines: 1-3,6

    -- ----------------------------------------------------------------------
    -- Define the project containing the C++ code
    project_ext_plugin(ext, ogn.plugin_project)

        add_files("impl", ogn.plugin_path)
        add_files("ogn", ogn.nodes_path)

The reference to the iface section was removed as it didn't contain any files and under extension 2.0 that is an error.

Add the generated code to the include path
++++++++++++++++++++++++++++++++++++++++++

The generated files must be accessible to the project so a helper function *add_ogn_dependencies()* was added to
make this common set of dependencies easy to implement. It handles *includedirs*, *libdirs*, *links*, and *dependson*
relationships needed by the OGN component of the project,  as well as common project compiler configuration.
Thanks to those definitions the rest of the project is pared down to this:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 1-2

    -- Add the standard dependencies all OGN projects have
    add_ogn_dependencies(ogn)

    includedirs {
        "%{target_deps}/rtx_plugins/include",
        "%{target_deps}/cuda/include"
    }

    filter { "system:linux" }
        removeflags { "FatalCompileWarnings", "UndefinedIdentifiers" }
        includedirs { "%{target_deps}/python/include/python3.7m" }
    filter { "system:windows" }
        libdirs { "%{target_deps}/tbb/lib/intel64/vc14" }
    filter {}

    links {"ar", "vt", "gf", "pcp", "sdf", "arch", "usd", "tf", "usdUtils", "usdGeom", "usdSkel", "omni.usd"}

Insert .ogn project
+++++++++++++++++++

The .ogn processing must precede the compiling of files including the generated headers so it must always happen
first. Unfortunately the current build system doesn't allow for individual file dependencies like that so instead
a secondary project must be set up that will process your .ogn files. The easiest place to insert it is just before
the Python project definition:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 1-3

    -- ----------------------------------------------------------------------
    -- Breaking this out as a separate project ensures the .ogn files are processed before their results are needed
    project_ext_ogn( ext, ogn )

Notice the *ogn* table defined earlier being passed in to the project definition function.

Update extension definition
+++++++++++++++++++++++++++

In some cases this may not be necessary but our example extension uses the old extension definition for its
Python files as follows:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 8-16

    project_ext_bindings {
        ext = ext,
        project_name = namespace..".python",
        module = "_anim_jiggle",
        src = "bindings",
        target_subdir = import_path.."/bindings"
    }
        vpaths {
            ['bindings'] = "bindings/*.*",
            ['python'] = python_source_path.."/*.py",
            ['python/scripts'] = python_source_path.."/scripts/*.py"
        }
        files {
            python_source_path.."/*.py",
            python_source_path.."/scripts/*.py"
        }

This combination of **vpaths** and **files** definitions is replaced under **extension 2.0** with the following,
also adding the new ``python/tests`` directory to the definition:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 3,9-12

    project_ext_bindings {
        ext = ext,
        project_name = python_project,
        module = "_anim_jiggle",
        src = "bindings",
        target_subdir = import_path.."/bindings"
    }

        -- All Python script and documentation files are part of this project
        add_files("bindings", "bindings/*.*")
        add_files("python", "python/*.py")
        add_files("python/scripts", "python/scripts/**.py")
        add_files("python/tests", "python/tests/**.py")

The parameters to the project can be replaced with the ones defined in the *ogn* table:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 1-2,5-8

    -- ----------------------------------------------------------------------
    -- Python support and scripts for the Jiggle deformer
    project_ext_bindings {
        ext = ext,
        project_name = ogn.python_project,
        module = ogn.bindings_module,
        src = ogn.bindings_path,
        target_subdir = ogn.bindings_target_path
    }


Set up OGN Dependencies for Python Nodes
++++++++++++++++++++++++++++++++++++++++

Python nodes will function best if they have hot reloading capability (i.e. a change you make to the source file will
be immediately reflected in the running application). The function that sets up project dependencies for OGN,
**add_ogn_dependencies** has an extra parameter that is a table of directories in which Python OGN nodes live.

This is the directory where your OGN node files are named *OgnMyNode.py*. While it's helpful to have the .ogn file in
the same directory it's not necessary for this particular purpose.

By adding the relative path to all directories containing these nodes the extension configuration will automatically
pick them up and register the nodes with OmniGraph, as well as automatically updating them at runtime if the source
files change. Any future nodes added to the named directories will work with no further change to the build file.

The directory is linked in the build, not copied, so it picks up all subdirectories as well. The only time you need to
modify this table is if a new directory containing Python nodes is created that is a sibling to the currently named
directories. As the registration looks for the node file name pattern of **Ogn.*.py** you can put other files in the
directory without causing any registration problems. You could even include your top level **python/** directory to
ensure all Python nodes are picked up.

.. code-block:: lua
    :linenos:
    :emphasize-lines: 1-2

    -- Add the standard dependencies all OGN projects have, and link directories with Python nodes
    add_ogn_dependencies(ogn, {"python/nodes"})


Step 4 : Add Extension Support
------------------------------

The build system handles the compile-time support for the nodes, the extension support will add the link-time
support. For **extension 2.0** this exists in the form of the file ``config/extension.toml``. It must be taught
about the python test modules and the extension dependencies.

.. code-block:: lua
    :linenos:
    :emphasize-lines: 4,8-15,17,20,22-24

    [package]
    title = "Omniverse Jiggle Deformer"

    # Main module of the python interface
    [[python.module]]
    name = "omni.anim.jiggle"

    # Additional python module used to make .ogn test files auto-discoverable
    [[python.module]]
    name = "omni.anim.jiggle.ogn.tests"

    # Watch the .ogn files for hot reloading (only for Python files)
    [fswatcher.patterns]
    include = ["*.ogn", "*.py"]
    exclude = ["Ogn*Database.py"]

    # Other extensions that must be loaded before this one
    [dependencies]
    "omni.graph" = {}
    "omni.graph.tools" = {}
    "omni.kit.usd_undo" = {}
    # The generated tests will make use of these modules
    "omni.usd" = {}
    "omni.kit.async_engine" = {}

    [[native.plugin]]
    path = "bin/${platform}/${config}/*.plugin"
    recursive = false

.. _ogn_test_detection:

Step 5 : Add Python Initialization
----------------------------------

Even nodes written entirely in C++ will have Pythonic interfaces, and possibly Python test scripts. It's best if you
initialize the OGN code on startup in order to take advantage of it rather than trying to do it manually later.

So long as you have an `ogn/` subdirectory in your Python import path this will all be taken care of automatically for
your; both registration and deregistration of your nodes when the extension loads and unloads respectively.


Step 6 : Set up automatic test detection (optional)
---------------------------------------------------

While you could set up test inclusion manually, you would have to add every new test file you create as well.
A mechanism was created to automatically scan a directory for test files and add them to the list testrunner uses
for regression tests. This operates in a similar manner to how the build looks for test files, but at runtime rather
than build time.

To enable this test detection, create ``python/tests/__init__.py`` with the content below. (Other content is fine if
you need it for other tests, this code can appear anywhere in that file.)

.. code-block:: python

    """
    Presence of this file allows the tests directory to be imported as a module so that all of its contents
    can be scanned to automatically add tests that are placed into this directory.
    """
    scan_for_test_modules = True

Now that the build can handle the .ogn files, existing nodes can be converted to the .ogn format using the steps
described in :ref:`ogn_node_conversion`.

Final Result - The Converted Build Script
-----------------------------------------
As reference, here is the final version of the file ``omni.anim.jiggle/premake5.lua`` with all new lines
highlighted:

.. code-block:: lua
    :linenos:
    :emphasize-lines: 4,7,14,17,20,23,39,46-49,58,63-66

    local ext = get_current_extension_info()

    -- Helper variable containing standard configuration information for projects containing OGN files
    local ogn = get_ogn_project_information(ext, "omni/anim/jiggle")

    -- Grouping also tells the name of the premake file that calls this one - $TOP/premake5-{ext.group}.lua
    ext.group = "animation"

    -- Set up common project variables
    project_ext( ext )

    -- ----------------------------------------------------------------------
    -- Define the project containing the C++ code
    project_ext_plugin(ext, ogn.plugin_project)

        add_files("impl", ogn.plugin_path)
        add_files("ogn", ogn.nodes_path)

        -- Add the standard dependencies all OGN projects have
        add_ogn_dependencies(ogn)

        includedirs {
            "%{target_deps}/rtx_plugins/include",
            "%{target_deps}/cuda/include"
        }

        filter { "system:linux" }
            removeflags { "FatalCompileWarnings", "UndefinedIdentifiers" }
            includedirs { "%{target_deps}/python/include/python3.7m" }
        filter { "system:windows" }
            libdirs { "%{target_deps}/tbb/lib/intel64/vc14" }
        filter {}

        links {"ar", "vt", "gf", "pcp", "sdf", "arch", "usd", "tf", "usdUtils", "usdGeom", "usdSkel", "omni.usd"}


    -- ----------------------------------------------------------------------
    -- Breaking this out as a separate project ensures the .ogn files are processed before their results are needed
    project_ext_ogn( ext, ogn )


    -- ----------------------------------------------------------------------
    -- Python support and scripts for the Jiggle deformer
    project_ext_bindings {
        ext = ext,
        project_name = ogn.python_project,
        module = ogn.bindings_module,
        src = ogn.bindings_path,
        target_subdir = ogn.bindings_target_path
    }

        -- All Python script and documentation files are part of this project
        add_files("bindings", "bindings/*.*")
        add_files("python", "python/*.py")
        add_files("python/tests", "python/tests/**.py")

        -- Add the standard dependencies all OGN projects have, and link directories with Python nodes
        add_ogn_dependencies(ogn, {"python/nodes"})

        -- Copy the init script directly into the build tree to avoid reload conflicts.
        repo_build.prebuild_copy {
            { "python/__init__.py", ogn.python_target_path },
        }
        -- Linking directories allows them to hot reload when files are modified in the source tree
        repo_build.prebuild_link {
            { "python/scripts", ogn.python_target_path.."/scripts" },
            { "python/tests", ogn.python_tests_target_path },
        }
