<commit_before>Mistral DSL v2 specification
============================

Introduction
------------

This document fully describes Domain Specific Language (DSL) version 2
of Mistral Workflow Service. Since version 1 issued in May 2014 Mistral
team completely reworked the language pursuing the goal in mind to make
it easier to understand while more consistent and flexible.

Unlike Mistral DSLv1 DSL v2 assumes that all
entities that Mistral works with like workflows and actions are
completely independent in terms of how they're referenced and accessed
through API (and also Python Client API and CLI). Workbook, the entity
that can combine workflows and actions still exists in the
language but only for namespacing and convenience purposes. See
`Workbooks section <#workbooks>`__ for more details.

**NOTE**: DSL and API of version 1 has not been supported since April 2015 and
DSL and API of version 2 is now the only way to interact with Mistral
service.

Mistral DSL consists of the following main object(entity) types that
will be described in details below:

-  `Workflows <#workflows>`__
-  `Actions <#actions>`__

Prerequisites
-------------

Mistral DSL takes advantage of
`YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ expression language to
reference workflow context variables and thereby implements passing data
between workflow tasks. It's also referred to as Data Flow mechanism.
YAQL is a simple but powerful query language that allows to extract
needed information from JSON structured data. It is allowed to use YAQL
in the following sections of DSL:

-  Workflow `'output' attribute <#common-workflow-attributes>`__
-  Workflow `'task-defaults' attribute <#common-workflow-attributes>`__
-  `Direct workflow <#direct-workflow>`__ transitions
-  Task `'publish' attribute <#common-task-attributes>`__
-  Task `'input' attribute <#common-task-attributes>`__
-  Task `'with-items' attribute <#common-task-attributes>`__
-  Task `'target' attribute <#common-task-attributes>`__
-  Any attribute of `task policies <#policies>`__
-  Action `'base-input' attribute <#attributes>`__
-  Action `'output' attribute <#attributes>`__

Mistral DSL is fully based on YAML and knowledge of YAML is a plus for
better understanding of the material in this specification. It also
takes advantage of YAQL query language to define expressions in workflow
and action definitions.

-  Yet Another Markup Language (YAML): http://yaml.org
-  Yet Another Query Language (YAQL): https://pypi.python.org/pypi/yaql/1.0.0

Workflows
---------

Workflow is the main building block of Mistral DSL, the reason why the
project exists. Workflow represents a process that can be described in a
various number of ways and that can do some job interesting to the end
user. Each workflow consists of tasks (at least one) describing what
exact steps should be made during workflow execution.

YAML example
^^^^^^^^^^^^

.. code-block:: yaml

    ---
    version: '2.0'

    create_vm:
      description: Simple workflow example
      type: direct
     
      input:
        - vm_name
        - image_ref
        - flavor_ref
      output:
        vm_id: <% $.vm_id %>
     
      tasks:
        create_server:
          action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %>
          publish:
            vm_id: <% task(create_server).result.id %>
          on-success:
            - wait_for_instance

        wait_for_instance:
          action: nova.servers_find id=<% $.vm_id %> status='ACTIVE'
          retry:
            delay: 5
            count: 15

This example workflow simply sends a command to OpenStack Compute
service Nova to start creating a virtual machine and wait till it's
created using special "retry" policy.

Workflow Types
^^^^^^^^^^^^^^

Mistral DSL v2 introduces different workflow types and the structure of
each workflow type varies according to its semantics. Basically,
workflow type encapsulates workflow processing logic, a set of meta
rules defining how all workflows of this type should work. Currently,
Mistral provides two workflow types:

-  `Direct workflow <#direct-workflow>`__
-  `Reverse workflow <#reverse-workflow>`__

See corresponding sections for details.

Common Workflow Attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^

-  **type** - Workflow type. Either 'direct' or 'reverse'. *Optional*. 'direct' by default.
-  **description** - Arbitrary text containing workflow description. *Optional*.
-  **input** - List defining required input parameter names and
   optionally their default values in a form "my_param: 123". *Optional*.
-  **output** - Any data structure arbitrarily containing YAQL
   expressions that defines workflow output. May be nested. *Optional*.
-  **task-defaults** - Default settings for some of task attributes
   defined at workflow level. *Optional*. Corresponding attribute
   defined for a specific task always takes precedence. Specific task
   attributes that could be defined in **task-defaults** are the
   following:

   -  **on-error** - List of tasks which will run after the task has
      completed with an error. For `direct
      workflow <#direct-workflow>`__ only. *Optional*.
   -  **on-success** - List of tasks which will run after the task has
      completed successfully. For `direct workflow <#direct-workflow>`__
      only. *Optional*.
   -  **on-complete** - List of tasks which will run after the task has
      completed regardless of whether it is successful or not. For
      `direct workflow <#direct-workflow>`__ only. *Optional*.
   -  **requires** - List of tasks that a task depends on. For `reverse
      workflow <#Reverse_Workflow>`__ only. *Optional*.
   -  **pause-before** - Configures pause-before policy. *Optional*.
   -  **wait-before** - Configures wait-before policy. *Optional*.
   -  **wait-after** - Configures wait-after policy. *Optional*.
   -  **timeout** - Configures timeout policy. *Optional*.
   -  **retry** - Configures retry policy. *Optional*.
   -  **concurrency** - Configures concurrency policy. *Optional*.

-  **tasks** - Dictionary containing workflow tasks. See below for more
   details. *Required*.

Tasks
^^^^^

Task is what a workflow consists of. It defines a specific computational
step in the workflow. Each task can optionally take input data and
produce output. In Mistral DSL v2 task can be associated with an action
or a workflow. In the example below there are two tasks of different
types:

.. code-block:: yaml

    action_based_task:
      action: std.http url='openstack.org'

    workflow_based_task:
      workflow: backup_vm_workflow vm_id=<% $.vm_id %>

Actions will be explained below in an individual paragraph but looking
ahead it's worth saying that Mistral provides a lot of actions out of
the box (including actions for most of the core OpenStack services) and
it's also easy to plug new actions into Mistral.

Common Task Attributes
''''''''''''''''''''''

All Mistral tasks regardless of workflow type have the following common
attributes:

-  **description** - Arbitrary text containing task description.
   *Optional*.
-  **action** - Name of the action associated with the task. *Required
   but mutually exclusive with* **workflow**.
-  **workflow** - Name of the workflow associated with the task.
   *Required but mutually exclusive with* **action**.
-  **input** - Actual input parameter values of the task. *Optional*.
   Value of each parameter is a JSON-compliant type such as number,
   string etc, dictionary or list. It can also be a YAQL expression to
   retrieve value from task context or any of the mentioned types
   containing inline YAQL expressions (for example, string "<%
   $.movie_name %> is a cool movie!")
-  **publish** - Dictionary of variables to publish to the workflow
   context. Any JSON-compatible data structure optionally containing
   YAQL expression to select precisely what needs to be published.
   Published variables will be accessible for downstream tasks via using
   YAQL expressions. *Optional*.
-  **with-items** - If configured, it allows to run action or workflow
   associated with a task multiple times on a provided list of items.
   See `Processing collections using
   'with-items' <#processing-collections>`__ for details. *Optional*.
-  **keep-result** - Boolean value allowing to not store action results
   after task completion (e.g. if they are large and not needed
   afterwards). *Optional*. By default is 'true'.<commit_msg>Correct documentation about task attributes 'action' and 'workflow'<commit_after>Mistral DSL v2 specification
============================

Introduction
------------

This document fully describes Domain Specific Language (DSL) version 2
of Mistral Workflow Service. Since version 1 issued in May 2014 Mistral
team completely reworked the language pursuing the goal in mind to make
it easier to understand while more consistent and flexible.

Unlike Mistral DSLv1 DSL v2 assumes that all
entities that Mistral works with like workflows and actions are
completely independent in terms of how they're referenced and accessed
through API (and also Python Client API and CLI). Workbook, the entity
that can combine workflows and actions still exists in the
language but only for namespacing and convenience purposes. See
`Workbooks section <#workbooks>`__ for more details.

**NOTE**: DSL and API of version 1 has not been supported since April 2015 and
DSL and API of version 2 is now the only way to interact with Mistral
service.

Mistral DSL consists of the following main object(entity) types that
will be described in details below:

-  `Workflows <#workflows>`__
-  `Actions <#actions>`__

Prerequisites
-------------

Mistral DSL takes advantage of
`YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ expression language to
reference workflow context variables and thereby implements passing data
between workflow tasks. It's also referred to as Data Flow mechanism.
YAQL is a simple but powerful query language that allows to extract
needed information from JSON structured data. It is allowed to use YAQL
in the following sections of DSL:

-  Workflow `'output' attribute <#common-workflow-attributes>`__
-  Workflow `'task-defaults' attribute <#common-workflow-attributes>`__
-  `Direct workflow <#direct-workflow>`__ transitions
-  Task `'publish' attribute <#common-task-attributes>`__
-  Task `'input' attribute <#common-task-attributes>`__
-  Task `'with-items' attribute <#common-task-attributes>`__
-  Task `'target' attribute <#common-task-attributes>`__
-  Any attribute of `task policies <#policies>`__
-  Action `'base-input' attribute <#attributes>`__
-  Action `'output' attribute <#attributes>`__

Mistral DSL is fully based on YAML and knowledge of YAML is a plus for
better understanding of the material in this specification. It also
takes advantage of YAQL query language to define expressions in workflow
and action definitions.

-  Yet Another Markup Language (YAML): http://yaml.org
-  Yet Another Query Language (YAQL): https://pypi.python.org/pypi/yaql/1.0.0

Workflows
---------

Workflow is the main building block of Mistral DSL, the reason why the
project exists. Workflow represents a process that can be described in a
various number of ways and that can do some job interesting to the end
user. Each workflow consists of tasks (at least one) describing what
exact steps should be made during workflow execution.

YAML example
^^^^^^^^^^^^

.. code-block:: yaml

    ---
    version: '2.0'

    create_vm:
      description: Simple workflow example
      type: direct
     
      input:
        - vm_name
        - image_ref
        - flavor_ref
      output:
        vm_id: <% $.vm_id %>
     
      tasks:
        create_server:
          action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %>
          publish:
            vm_id: <% task(create_server).result.id %>
          on-success:
            - wait_for_instance

        wait_for_instance:
          action: nova.servers_find id=<% $.vm_id %> status='ACTIVE'
          retry:
            delay: 5
            count: 15

This example workflow simply sends a command to OpenStack Compute
service Nova to start creating a virtual machine and wait till it's
created using special "retry" policy.

Workflow Types
^^^^^^^^^^^^^^

Mistral DSL v2 introduces different workflow types and the structure of
each workflow type varies according to its semantics. Basically,
workflow type encapsulates workflow processing logic, a set of meta
rules defining how all workflows of this type should work. Currently,
Mistral provides two workflow types:

-  `Direct workflow <#direct-workflow>`__
-  `Reverse workflow <#reverse-workflow>`__

See corresponding sections for details.

Common Workflow Attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^

-  **type** - Workflow type. Either 'direct' or 'reverse'. *Optional*. 'direct' by default.
-  **description** - Arbitrary text containing workflow description. *Optional*.
-  **input** - List defining required input parameter names and
   optionally their default values in a form "my_param: 123". *Optional*.
-  **output** - Any data structure arbitrarily containing YAQL
   expressions that defines workflow output. May be nested. *Optional*.
-  **task-defaults** - Default settings for some of task attributes
   defined at workflow level. *Optional*. Corresponding attribute
   defined for a specific task always takes precedence. Specific task
   attributes that could be defined in **task-defaults** are the
   following:

   -  **on-error** - List of tasks which will run after the task has
      completed with an error. For `direct
      workflow <#direct-workflow>`__ only. *Optional*.
   -  **on-success** - List of tasks which will run after the task has
      completed successfully. For `direct workflow <#direct-workflow>`__
      only. *Optional*.
   -  **on-complete** - List of tasks which will run after the task has
      completed regardless of whether it is successful or not. For
      `direct workflow <#direct-workflow>`__ only. *Optional*.
   -  **requires** - List of tasks that a task depends on. For `reverse
      workflow <#Reverse_Workflow>`__ only. *Optional*.
   -  **pause-before** - Configures pause-before policy. *Optional*.
   -  **wait-before** - Configures wait-before policy. *Optional*.
   -  **wait-after** - Configures wait-after policy. *Optional*.
   -  **timeout** - Configures timeout policy. *Optional*.
   -  **retry** - Configures retry policy. *Optional*.
   -  **concurrency** - Configures concurrency policy. *Optional*.

-  **tasks** - Dictionary containing workflow tasks. See below for more
   details. *Required*.

Tasks
^^^^^

Task is what a workflow consists of. It defines a specific computational
step in the workflow. Each task can optionally take input data and
produce output. In Mistral DSL v2 task can be associated with an action
or a workflow. In the example below there are two tasks of different
types:

.. code-block:: yaml

    action_based_task:
      action: std.http url='openstack.org'

    workflow_based_task:
      workflow: backup_vm_workflow vm_id=<% $.vm_id %>

Actions will be explained below in an individual paragraph but looking
ahead it's worth saying that Mistral provides a lot of actions out of
the box (including actions for most of the core OpenStack services) and
it's also easy to plug new actions into Mistral.

Common Task Attributes
''''''''''''''''''''''

All Mistral tasks regardless of workflow type have the following common
attributes:

-  **description** - Arbitrary text containing task description.
   *Optional*.
-  **action** - Name of the action associated with the task. *Mutually exclusive with* **workflow**. If neither action nor workflow are provided then the action 'std.noop' will be used.
-  **workflow** - Name of the workflow associated with the task.
   *Mutually exclusive with* **action**.
-  **input** - Actual input parameter values of the task. *Optional*.
   Value of each parameter is a JSON-compliant type such as number,
   string etc, dictionary or list. It can also be a YAQL expression to
   retrieve value from task context or any of the mentioned types
   containing inline YAQL expressions (for example, string "<%
   $.movie_name %> is a cool movie!")
-  **publish** - Dictionary of variables to publish to the workflow
   context. Any JSON-compatible data structure optionally containing
   YAQL expression to select precisely what needs to be published.
   Published variables will be accessible for downstream tasks via using
   YAQL expressions. *Optional*.
-  **with-items** - If configured, it allows to run action or workflow
   associated with a task multiple times on a provided list of items.
   See `Processing collections using
   'with-items' <#processing-collections>`__ for details. *Optional*.
-  **keep-result** - Boolean value allowing to not store action results
   after task completion (e.g. if they are large and not needed
   afterwards). *Optional*. By default is 'true'.
{'fraction_non_alphanumeric': 0.09346451159522136, 'fraction_numerical': 0.003630826891543687, 'mean_word_length': 3.649332970323986, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6479085', 'n_tokens_mistral': 4913, 'n_tokens_neox': 4533, 'n_words': 2082}