<commit_before>.. _pipeline:

****************
Pipeline Example
****************

This example shows how to use the Ensemble Toolkit ``Pipeline`` pattern
to execute 16 concurrent pipeline of sequential tasks. In the first step of
each pipeline ``stage_1``, a 10 MB input file is generated and filled with
ASCII charaters. In the second step ``stage_2``, a character frequency analysis
if performed on this file. In the last step ``stage_3``, an SHA1 checksum is
calculated for the analysis result. The results of the frequency analysis and
the SHA1 checksums are copied back to the machine on which this script runs.

.. figure:: ../../images/pipeline_pattern.*
   :width: 300pt
   :align: center
   :alt: Pipeline Pattern

   Fig.: `The Pipeline Pattern.`

Run Locally
===========

.. warning:: In order to run this example, you need access to a MongoDB server and
             set the ``RADICAL_PILOT_DBURL`` in your environment accordingly.
             The format is ``mongodb://hostname:port``. Read more about it
             MongoDB in chapter :ref:`envpreparation`.

**Step 1:** View and download the example sources :ref:`below <example_source_pipeline>`.

**Step 2:** Run this example with ``RADICAL_ENTK_VERBOSE`` set to ``REPORT``::

    RADICAL_ENTK_VERBOSE=REPORT python pipeline.py<commit_msg>minor grammar fix<commit_after>.. _pipeline:

****************
Pipeline Example
****************

This example shows how to use the Ensemble Toolkit ``Pipeline`` pattern
to execute 16 concurrent pipelines of sequential tasks. In the first step of
each pipeline ``stage_1``, a 10 MB input file is generated and filled with
ASCII charaters. In the second step ``stage_2``, a character frequency analysis
if performed on this file. In the last step ``stage_3``, an SHA1 checksum is
calculated for the analysis result. The results of the frequency analysis and
the SHA1 checksums are copied back to the machine on which this script runs.

.. figure:: ../../images/pipeline_pattern.*
   :width: 300pt
   :align: center
   :alt: Pipeline Pattern

   Fig.: `The Pipeline Pattern.`

Run Locally
===========

.. warning:: In order to run this example, you need access to a MongoDB server and
             set the ``RADICAL_PILOT_DBURL`` in your environment accordingly.
             The format is ``mongodb://hostname:port``. Read more about it
             MongoDB in chapter :ref:`envpreparation`.

**Step 1:** View and download the example sources :ref:`below <example_source_pipeline>`.

**Step 2:** Run this example with ``RADICAL_ENTK_VERBOSE`` set to ``REPORT``::

    RADICAL_ENTK_VERBOSE=REPORT python pipeline.py
{'fraction_non_alphanumeric': 0.11446012972148036, 'fraction_numerical': 0.0106829454406715, 'mean_word_length': 4.013384321223709, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3811289', 'n_tokens_mistral': 734, 'n_tokens_neox': 692, 'n_words': 329}