<commit_before>Running Workflows
==========

Quick workflow example
-----------------------

Here's a quick workflow that starts with a Worldview 2 image over San Francisco, runs it through
DigitalGlobe's "Fast Ortho" and "Acomp" tasks, then saves to a user-specified location 
under s3://bucket/prefix.

.. code-block:: python

   data = "s3://receiving-dgcs-tdgplatform-com/054813633050_01_003" # WV02 Image over San Francisco
   aoptask = gbdx.Task("AOP_Strip_Processor", data=data, enable_acomp=True, enable_pansharpen=True)
   workflow = gbdx.Workflow([ aoptask ]) 
   workflow.savedata(aoptask.outputs.data, location='some_folder_under_your_bucket_prefix')
   workflow.execute()

At this point the workflow is launched, and you can get status as follows:

.. code-block:: pycon

   >>> workflow.status
   >>> {u'state': u'pending', u'event': u'submitted'}

You can also get workflow events:

.. code-block:: pycon

   >>> for event in workflow.events:
   >>>     print event['task'], event['event']
   
   >>> gdal-task submitted
   >>> Stage-data submitted
   >>> gdal-task scheduled
   >>> gdal-task started
   >>> gdal-task succeeded
   >>> Stage-data scheduled
   >>> Stage-data started
   >>> Stage-data succeeded

Tasks
-----------------------

A task is instantiated as follows:

.. code-block:: python

    task = gbdx.Task("Task_Name")

The task name must be a valid gbdx task name.


Setting Task Inputs
-----------------------

The following are all equivalent ways of setting the input values on a task:

.. code-block:: python

    # upon task instantiation:
    task = gbdx.Task("Task_Name", input1="value1", input2="value2")

    # with set function:
    task.set(input1="value1", input2="value2")

    # using input setter:
    task.inputs.input1 = "value1"
    task.inputs.input2 = "value2"

You can interactively determine the inputs of a task by typing:

.. code-block:: python

    >>> task = gbdx.Task("AOP_Strip_Processor")
    >>> task.inputs
    enable_tiling
    enable_dra
    bands
    enable_acomp
    ...

You can also interactively get more info on a particular input:

.. code-block:: python

    >>> task.inputs.enable_acomp
    Port enable_acomp:
       type: string
       description: Enable/disable AComp. Choices are 'true' or 'false'. Default is 'true'.
       required: False
       Value: None

Task Outputs
-----------------------

Task outputs can be interactively explored the same way as task inputs:

.. code-block:: python

    >>> task = gbdx.Task("AOP_Strip_Processor")
    >>> task.outputs
    data
    log

    >>> task.outputs.log
    Port log:
       type: directory
       description: The output log directory


Linking Outputs from one task into Inputs of Another Task
-----------------------

The whole point of the workflow system is to build complex workflows with 
automagic data movement between tasks. This can be done as follows:

.. code-block:: python

    task1 = gbdx.Task("AOP_Strip_Processor")

    task2 = gbdx.Task("Some_Other_task")
    task2.inputs.<input_name> = task1.outputs.<output_name>.value

Running a Workflow
-----------------------

A workflow is just a set of tasks with inputs and outputs linked appropriately.  Create/setup a few tasks and construct and run a workflow:

.. code-block:: python

    data = "s3://receiving-dgcs-tdgplatform-com/054813633050_01_003" # WV02 Image over San Francisco
    aoptask = gbdx.Task("AOP_Strip_Processor", data=data)

    s3task = gbdx.Task("StageDataToS3")
    s3task.inputs.data = aoptask.outputs.data.value
    s3task.inputs.destination = "s3://path/to/destination"

    workflow = gbdx.Workflow([ s3task, aoptask ])
    workflow.execute()

Note that a workflow is instantiated with a list of tasks.  The tasks will get executed when their inputs are satisfied and ready to go.


Workflow Status
-----------------------

There are a few ways to check the status of a running workflow.

Checking the status directly:

.. code-block:: python

   >>> workflow.status
   {u'state': u'pending', u'event': u'submitted'}

Checking whether a workflow is running:

.. code-block:: python
<commit_msg>added example of running workflow with 4 tasks<commit_after>Running Workflows
==========

Quick workflow example
-----------------------

Here's a quick workflow that starts with a Worldview 2 image over San Francisco, runs it through
DigitalGlobe's "Fast Ortho" and "Acomp" tasks, then saves to a user-specified location 
under s3://bucket/prefix.

.. code-block:: python

   data = "s3://receiving-dgcs-tdgplatform-com/054813633050_01_003" # WV02 Image over San Francisco
   aoptask = gbdx.Task("AOP_Strip_Processor", data=data, enable_acomp=True, enable_pansharpen=True)
   workflow = gbdx.Workflow([ aoptask ]) 
   workflow.savedata(aoptask.outputs.data, location='some_folder_under_your_bucket_prefix')
   workflow.execute()

At this point the workflow is launched, and you can get status as follows:

.. code-block:: pycon

   >>> workflow.status
   >>> {u'state': u'pending', u'event': u'submitted'}

You can also get workflow events:

.. code-block:: pycon

   >>> for event in workflow.events:
   >>>     print event['task'], event['event']
   
   >>> gdal-task submitted
   >>> Stage-data submitted
   >>> gdal-task scheduled
   >>> gdal-task started
   >>> gdal-task succeeded
   >>> Stage-data scheduled
   >>> Stage-data started
   >>> Stage-data succeeded

Tasks
-----------------------

A task is instantiated as follows:

.. code-block:: python

    task = gbdx.Task("Task_Name")

The task name must be a valid gbdx task name.


Setting Task Inputs
-----------------------

The following are all equivalent ways of setting the input values on a task:

.. code-block:: python

    # upon task instantiation:
    task = gbdx.Task("Task_Name", input1="value1", input2="value2")

    # with set function:
    task.set(input1="value1", input2="value2")

    # using input setter:
    task.inputs.input1 = "value1"
    task.inputs.input2 = "value2"

You can interactively determine the inputs of a task by typing:

.. code-block:: python

    >>> task = gbdx.Task("AOP_Strip_Processor")
    >>> task.inputs
    enable_tiling
    enable_dra
    bands
    enable_acomp
    ...

You can also interactively get more info on a particular input:

.. code-block:: python

    >>> task.inputs.enable_acomp
    Port enable_acomp:
       type: string
       description: Enable/disable AComp. Choices are 'true' or 'false'. Default is 'true'.
       required: False
       Value: None

Task Outputs
-----------------------

Task outputs can be interactively explored the same way as task inputs:

.. code-block:: python

    >>> task = gbdx.Task("AOP_Strip_Processor")
    >>> task.outputs
    data
    log

    >>> task.outputs.log
    Port log:
       type: directory
       description: The output log directory


Linking Outputs from one task into Inputs of Another Task
-----------------------

The whole point of the workflow system is to build complex workflows with 
automagic data movement between tasks. This can be done as follows:

.. code-block:: python

    task1 = gbdx.Task("AOP_Strip_Processor")

    task2 = gbdx.Task("Some_Other_task")
    task2.inputs.<input_name> = task1.outputs.<output_name>.value

Running a Workflow
-----------------------

A GBDX workflow is a set of tasks with inputs and outputs linked appropriately.  
Note that in gbdxtools, a workflow object is instantiated with a list of tasks.  
The tasks will get executed when their inputs are satisfied and ready to go.
Here is an example of a workflow which consists of the AOP_Strip_Processor task followed by 
the StageDataToS3 task.

.. code-block:: python

    data = "s3://receiving-dgcs-tdgplatform-com/054813633050_01_003" # WV02 Image over San Francisco
    aoptask = gbdx.Task("AOP_Strip_Processor", data=data)

    s3task = gbdx.Task("StageDataToS3")
    s3task.inputs.data = aoptask.outputs.data.value
    s3task.inputs.destination = "s3://path/to/destination"

    workflow = gbdx.Workflow([ s3task, aoptask ])
    workflow.execute()

Here is another example of a more complicated workflow.

.. code-block:: python

    data = "s3://receiving-dgcs-tdgplatform-com/054813633050_01_003" 
    aoptask = gbdx.Task("AOP_Strip_Processor", data=data, enable_acomp=True, enable_pansharpen=False, enable_dra=False, bands='MS')
    pp_task = gbdx.Task("ProtogenPrep",raster=aoptask.outputs.data.value)      # ProtogenPrep task is used to get AOP output into proper format for protogen task
    prot_lulc = gbdx.Task("protogenV2LULC", raster=pp_task.outputs.data.value)
    workflow = gbdx.Workflow([ aoptask, pp_task, prot_lulc ]) 
    workflow.savedata(prot_lulc.outputs.data.value, location="some_folder_under_your_bucket_prefix")
    workflow.execute()

Workflow Status
-----------------------

There are a few ways to check the status of a running workflow.

Checking the status directly:

.. code-block:: python

   >>> workflow.status
   {u'state': u'pending', u'event': u'submitted'}

Checking whether a workflow is running:

.. code-block:: python

{'fraction_non_alphanumeric': 0.13431848257609175, 'fraction_numerical': 0.01643140714600794, 'mean_word_length': 3.0631720430107525, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 121, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2235899', 'n_tokens_mistral': 2878, 'n_tokens_neox': 2732, 'n_words': 869}