<commit_before>.. _generate-your-engines:

Create/Implement your own UNode
===============================

Before implementing your own UNode, make sure that you have read about
the :ref:`General structure<general_structure>` of Ursgal.
This page will explain how to integrate a standalone executable or Python
script into Ursgal's structure of **resources**, **knowledge base (kb)**
and **engines**, based on two examples:
    * A Python script: filter_csv_1_0_0.py
    * A standalone search engine: MS-GF+ v9979

1. Integration into Resources
-----------------------------

The **resources/** folder contains the main code of each UNode (an executable
or Python script). This executable should be standalone and executable from the command line.
Each UNode requires its own subfolder in the **resources/** folder, which contains the
executable.

Note:
    The UNodes' **resources/** subfolder, **engine/** class and **kb/** file should all have the same name
    (lowercase and underscores instead of spaces, e.g. 'msgfplus_v9979' or 'filter_csv_1_0_0').


    #. Platfom dependent engines need to be placed according to the platform:
       darwin (OS X), linux or win32 (Windows 32 or 64 bit)

        * <ursgal_path>\ **/resources/**\ <platform>/\ <architecture>/<name_of_engine>/source (executable + potential additional files)

       **Example**: MS-GF+ on windows 64 bit:

        * <ursgal_path>\ **/resources/**\ win32/\ 64bit/msgfplus_v9979/MSGFPlus.jar


    #. Architecture independent engines, like Python scripts or Java packages
       should be placed in **/resources/**\platform_independent/arc_independent/

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/<name_of_engine>/engine.py

       **Example**: filter_csv_1_0_0.py:

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/filter_csv_1_0_0/filter_csv_1_0_0.py

       Actually, MS-GF+ is platform independent as well (since it it based on Java) and can therefore also be placed in:

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/msgfplus_v9979/MSGFPlus.jar


2. Integration into uparams.py
------------------------------

Each parameter that is used by an engine needs to be included in uparams.py.
This is a dictionary containing all parameters that are available in ursgal, its structure is explained here **insert link**.

For every parameter that can be used by a new engine, it should be checked if a corresponding parameter is already present in uparams.py.
If this is the case, the new engine (unode name) needs to be included in 'available_in_unode'. Furthermore, 'ukey_translation' needs to contain
the utranslation_style that is defined in the engines META_INFO (see General Structure) translating the ursgal parameter into the engine specific
parameter name. The parameter values can be translated in 'uvalue_translation' using the utranslation_style as well (only if a translation is necessary).

**Example**: include the parameter '-e' for MS-GF+ ::
  
    # -e defines the enzyme that has been used for digestion. This is called 'enzyme' in ursgal.
    'enzyme' : {
        # include msgfplus_v9979 in available_in_unode
        'available_in_unode' : [
            xtandem_vengeance',
            'msgfplus_v9979',
        ],
        # default_value, description, trigger_rerun, utag and uvalue_type don't need to be changed
        'default_value' : "trypsin",
        'description' :  ''' Enzyme: Rule of protein cleavage
            Possible cleavages are ...
        'trigger_rerun' : True,
        # Translate the ursgal parameter name ('enzyme') to the MS-GF+ parameter name ('-e') using the translation style (msgfplus_style_1) in ukey_translation
        'ukey_translation' : {
            'msgfplus_style_1' : '-e',
            'xtandem_style_1' : 'protein, cleavage site',
        },
        # Translate the ursgal parameter values (e.g. 'trypsin') to the MS-GF+ parameter value (e.g. '1') using the translation style (msgfplus_style_1) in uvalue_translation
        'uvalue_translation' : {
            'msgfplus_style_1' : {
                'alpha_lp' : '8',
                'argc' : '6',
                'aspn' : '7',
                'chymotrypsin' : '2',
                'glutamyl_endopeptidase' : '5',
                'lysc' : '3',
                'lysn' : '4',
                'no_cleavage' : '9',
                'nonspecific' : '0',
                'trypsin' : '1',
            },
            'xtandem_style_1' : {
                'argc' : '[R]|{P}',
                'aspn' : '[X]|[D]',
                'chymotrypsin' : '[FMWY]|{P}',
                'chymotrypsin_p' : '[FMWY]|[X]',
                'clostripain' : '[R]|[X]',
                'cnbr' : '[M]|{P}',
                'elastase' : '[AGILV]|{P}',
                'formic_acid' : '[D]|{P}',
                'gluc' : '[DE]|{P}',
                'gluc_bicarb' : '[E]|{P}',
                'iodosobenzoate' : '[W]|[X]',
                'lysc' : '[K]|{P}',
                'lysc_p' : '[K]|[X]',
                'lysn' : '[X]|[K]',
                'lysn_promisc' : '[X]|[AKRS]',
                'nonspecific' : '[X]|[X]',
                'pepsina' : '[FL]|[X]',
                'protein_endopeptidase' : '[P]|[X]',
                'staph_protease' : '[E]|[X]',
                'tca' : '[FMWY]|{P},[KR]|{P},[X]|[D]',
                'trypsin' : '[KR]|{P}',
                'trypsin_cnbr' : '[KR]|{P},[M]|{P}',
                'trypsin_gluc' : '[DEKR]|{P}',
                'trypsin_p' : '[RK]|[X]',
            },


If a parameter is not yet present in uparams.py, you can add a new paramter containing all necessary informations.

**Example** add write_unfiltered_results for filter_csv_1_0_0 ::

    'write_unfiltered_results' : {
            'available_in_unode' : [
                'filter_csv_1_0_0',
            ],
            'default_value' : False,
            'description' :  ''' writes rejected results if True ''',
            'trigger_rerun' : True,
            'ukey_translation' : {
                'filter_csv_style_1' : 'write_unfiltered_results',
            },
            'utag' : [
                'conversion',
            ],
            'uvalue_translation' : {
            },
            'uvalue_type' : "bool",
        },

After changing uparams.py, please run rewrite_uparams.py to ckeck for duplicates/errors and to sort the dictionary.


3. Implementation into Engines
------------------------------

Each UNode has to have a Python wrapper file located in:

 * <ursgal_path>\ **/wrappers/** <unode_name>.py

The UNode has to inherit from the UNode class, which during initialization injects
the node related data into the class.


The default structure of the UNode class has to be::

    class my_unode_1_0_0(ursgal.UNode):

        META_INFO = {}

        def __init__(self, *args, **kwargs):
            super(my_unode_1_0_0, self).__init__(*args, **kwargs)

        def preflight(self):
            # code that should be run before the UNode is executed
            # e.g. writing a config file
            return

        def postflight(self):
            # code that should be run after the UNode is executed
            # e.g. formatting the output file
            return

where `my_unode_1_0_0` is the name of the UNode. The META_INFO is explained :ref:`here<metainfo>` and is available as attribute of each UNode.
One can define `preflight()` and `postflight()` methods that will be executed by the uNode during preflight and postflight
(= before execution of the main executable and after execution).


3.1 Implementaion of an engine from a command line tool<commit_msg>misc. corrections in create_your_own_engine docs<commit_after>.. _generate-your-engines:

Create/Implement your own UNode
===============================

Before implementing your own UNode, make sure that you have read about
the :ref:`General structure<general_structure>` of Ursgal.
This page will explain how to integrate a standalone executable or Python
script into Ursgal's structure of **resources**, **wrappers**
and **uparams.py**, based on two examples:
    * A Python script: filter_csv_1_0_0.py
    * A standalone search engine: MS-GF+ v9979

1. Integration into Resources
-----------------------------

The **resources/** folder contains the main code of each UNode (an executable
or Python script). This executable should be standalone and executable from the command line.
Each UNode requires its own subfolder in the **resources/** folder, which contains the
executable.

Note:
    The UNodes' **resources/** subfolder, **wrapper/** file and **wrapper/** file Python class
    should all have the same name (lowercase and underscores instead of spaces, e.g.
    'msgfplus_v9979' or 'filter_csv_1_0_0').


    #. Platfom dependent engines need to be placed according to the platform:
       darwin (OS X), linux or win32 (Windows 32 or 64 bit)

        * <ursgal_path>\ **/resources/**\ <platform>/\ <architecture>/<name_of_engine>/source (executable + potential additional files)

       **Example**: MS-GF+ on windows 64 bit:

        * <ursgal_path>\ **/resources/**\ win32/\ 64bit/msgfplus_v9979/MSGFPlus.jar


    #. Architecture independent engines, like Python scripts or Java packages
       should be placed in **/resources/**\platform_independent/arc_independent/

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/<name_of_engine>/engine.py

       **Example**: filter_csv_1_0_0.py:

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/filter_csv_1_0_0/filter_csv_1_0_0.py

       Actually, MS-GF+ is platform independent as well (since it it based on Java) and can therefore also be placed in:

        * <ursgal_path>\ **/resources/**\platform_independent/arc_independent/msgfplus_v9979/MSGFPlus.jar


2. Integration into uparams.py
------------------------------

Each parameter that is used by an engine needs to be included in the file <ursgal_path>/ursgal/uparams.py.
This is a dictionary containing all parameters that are available in ursgal, its structure is explained :ref:`here<uparams>`.

For every parameter that can be used by a new engine, it should be checked if a corresponding parameter is already present in uparams.py.
If this is the case, the new engine (unode name) needs to be included in 'available_in_unode'. Furthermore, 'ukey_translation' needs to contain
the utranslation_style that is defined in the engines :ref:`META_INFO<metainfo>` translating the ursgal parameter into the engine-specific
parameter name. The parameter values can be translated in 'uvalue_translation' using the utranslation_style as well (only if a translation is necessary).

**Example**: include the parameter '-e' for MS-GF+ ::
  
    # -e defines the enzyme that has been used for digestion. This is called 'enzyme' in ursgal.
    'enzyme' : {
        # include msgfplus_v9979 in available_in_unode
        'available_in_unode' : [
            xtandem_vengeance',
            'msgfplus_v9979',
        ],
        # default_value, description, trigger_rerun, utag and uvalue_type don't need to be changed
        'default_value' : "trypsin",
        'description' :  ''' Enzyme: Rule of protein cleavage
            Possible cleavages are ...
        'trigger_rerun' : True,
        # Translate the ursgal parameter name ('enzyme') to the MS-GF+ parameter name ('-e') using the translation style (msgfplus_style_1) in ukey_translation
        'ukey_translation' : {
            'msgfplus_style_1' : '-e',
            'xtandem_style_1' : 'protein, cleavage site',
        },
        # Translate the ursgal parameter values (e.g. 'trypsin') to the MS-GF+ parameter value (e.g. '1') using the translation style (msgfplus_style_1) in uvalue_translation
        'uvalue_translation' : {
            'msgfplus_style_1' : {
                'alpha_lp' : '8',
                'argc' : '6',
                'aspn' : '7',
                'chymotrypsin' : '2',
                'glutamyl_endopeptidase' : '5',
                'lysc' : '3',
                'lysn' : '4',
                'no_cleavage' : '9',
                'nonspecific' : '0',
                'trypsin' : '1',
            },
            'xtandem_style_1' : {
                'argc' : '[R]|{P}',
                'aspn' : '[X]|[D]',
                'chymotrypsin' : '[FMWY]|{P}',
                'chymotrypsin_p' : '[FMWY]|[X]',
                'clostripain' : '[R]|[X]',
                'cnbr' : '[M]|{P}',
                'elastase' : '[AGILV]|{P}',
                'formic_acid' : '[D]|{P}',
                'gluc' : '[DE]|{P}',
                'gluc_bicarb' : '[E]|{P}',
                'iodosobenzoate' : '[W]|[X]',
                'lysc' : '[K]|{P}',
                'lysc_p' : '[K]|[X]',
                'lysn' : '[X]|[K]',
                'lysn_promisc' : '[X]|[AKRS]',
                'nonspecific' : '[X]|[X]',
                'pepsina' : '[FL]|[X]',
                'protein_endopeptidase' : '[P]|[X]',
                'staph_protease' : '[E]|[X]',
                'tca' : '[FMWY]|{P},[KR]|{P},[X]|[D]',
                'trypsin' : '[KR]|{P}',
                'trypsin_cnbr' : '[KR]|{P},[M]|{P}',
                'trypsin_gluc' : '[DEKR]|{P}',
                'trypsin_p' : '[RK]|[X]',
            },


If a parameter is not yet present in uparams.py, you can add a new parameter containing all necessary information.

**Example** add write_unfiltered_results for filter_csv_1_0_0 ::

    'write_unfiltered_results' : {
            'available_in_unode' : [
                'filter_csv_1_0_0',
            ],
            'default_value' : False,
            'description' :  ''' writes rejected results if True ''',
            'trigger_rerun' : True,
            'ukey_translation' : {
                'filter_csv_style_1' : 'write_unfiltered_results',
            },
            'utag' : [
                'conversion',
            ],
            'uvalue_translation' : {
            },
            'uvalue_type' : "bool",
        },

After changing uparams.py, please run rewrite_uparams.py to check for duplicates/errors and to sort the dictionary.


3. Implementation of the wrapper class
------------------------------

Each UNode has to have a Python wrapper file located in:

 * <ursgal_path>\ **/wrappers/** <unode_name>.py

The UNode has to inherit from the UNode class, which during initialization injects
the node related data into the class.


The default structure of the UNode class has to be::

    class my_unode_1_0_0(ursgal.UNode):

        META_INFO = {}

        def __init__(self, *args, **kwargs):
            super(my_unode_1_0_0, self).__init__(*args, **kwargs)

        def preflight(self):
            # code that should be run before the UNode is executed
            # e.g. writing a config file
            return

        def postflight(self):
            # code that should be run after the UNode is executed
            # e.g. formatting the output file
            return

where `my_unode_1_0_0` is the name of the UNode. The META_INFO is explained :ref:`here<metainfo>` and is available as attribute of each UNode.
One can define `preflight()` and `postflight()` methods that will be executed by the uNode during preflight and postflight
(= before execution of the main executable and after execution).


3.1 Implementation of an engine from a command line tool
{'fraction_non_alphanumeric': 0.12979676644373742, 'fraction_numerical': 0.011557691059022141, 'mean_word_length': 2.3955026455026456, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '916203', 'n_tokens_mistral': 4873, 'n_tokens_neox': 4545, 'n_words': 1441}