<commit_before>.. ehive creating pipelines guide, a description of accumulators

.. The default language is set to perl. Non-perl code-blocks have to define
   their own language setting
.. highlight:: perl


Accumulators
============

Accumulators are a way of passing data from within a semaphore group
to its controlling funnel.

Accumulators are defined within pipelines as URLs. These must have the
``accu_name`` key, which indicates the name of the funnel's parameter that
will hold the data. These data come from the dataflow event, specifically
the parameter that has the name of the ``accu_name`` key. This can be
overridden with the ``accu_input_variable`` key.
There are five types of Accumulators, all described below:
scalar, pile, multiset, array and hash. For each of them we show how to
initialise them and equivalent Perl code to build the same structure.

Scalar
~~~~~~

Basic syntax:
    ``?accu_name=scalar_name``

Extended syntax:
    ``?accu_name=scalar_name&accu_input_variable=output_parameter_name``

Retrieval:
    ``my $scalar_value = $self->param('scalar_name');``

This is the simplest type of Accumulator. The basic syntax example passes
he value of the ``scalar_name`` parameter from the *fan* to the
*funnel*. The extended syntax example makes the value of the
``output_parameter_name`` parameter from the *fan* available to the
*funnel* as if it were a parameter named ``scalar_name``. If there are
multiple Jobs in the fan, eHive will arbitrarily select one of them to
define the Accumulator.

In Perl, this is equivalent to doing this:

Accumulator initialisation:
   ::

       my $scalar_name;

Accumulator extension:
   ::

       $scalar_name = $scalar_name;           # Basic syntax
       $scalar_name = $output_parameter_name; # Extended syntax

Accumulator retrieval:
   ::

       say "Value: $scalar_name";


Pile
~~~~

Basic syntax:
    ``?accu_name=pile_name&accu_address=[]``

Extended syntax:
    ``?accu_name=pile_name&accu_address=[]&accu_input_variable=pile_component``

Retrieval:
  ::

      my $pile_ref = $self->param('pile_name');
      foreach my $pile_element (@{$pile_ref}) {
          # do something with $pile_element
      }


A pile is an unordered list. All the ``pile_name`` (or ``pile_component``
in the second form) values that are dataflown
into the Accumulator are aggregated into a list named ``pile_name``
in a *random* order.

In Perl, this is similar to doing this:

Accumulator initialisation:
   ::

       my @pile_name;

Accumulator extension:
   ::

       push @pile_name, $pile_name;             # Basic syntax
       push @pile_name, $pile_component;        # Extended syntax

Accumulator retrieval:
   ::

       foreach my $v (@pile_name) {
           say "Value: $v";
       }


Multiset
~~~~~~~~

Basic syntax:
    ``?accu_name=multiset_name&accu_address=[]``

Extended syntax:
    ``?accu_name=multiset_name&accu_address=[]&accu_input_variable=multiset_component``

Retrieval:
   ::<commit_msg>docfix: the address of a multiset accumulator was wrong<commit_after>.. ehive creating pipelines guide, a description of accumulators

.. The default language is set to perl. Non-perl code-blocks have to define
   their own language setting
.. highlight:: perl


Accumulators
============

Accumulators are a way of passing data from within a semaphore group
to its controlling funnel.

Accumulators are defined within pipelines as URLs. These must have the
``accu_name`` key, which indicates the name of the funnel's parameter that
will hold the data. These data come from the dataflow event, specifically
the parameter that has the name of the ``accu_name`` key. This can be
overridden with the ``accu_input_variable`` key.
There are five types of Accumulators, all described below:
scalar, pile, multiset, array and hash. For each of them we show how to
initialise them and equivalent Perl code to build the same structure.

Scalar
~~~~~~

Basic syntax:
    ``?accu_name=scalar_name``

Extended syntax:
    ``?accu_name=scalar_name&accu_input_variable=output_parameter_name``

Retrieval:
    ``my $scalar_value = $self->param('scalar_name');``

This is the simplest type of Accumulator. The basic syntax example passes
he value of the ``scalar_name`` parameter from the *fan* to the
*funnel*. The extended syntax example makes the value of the
``output_parameter_name`` parameter from the *fan* available to the
*funnel* as if it were a parameter named ``scalar_name``. If there are
multiple Jobs in the fan, eHive will arbitrarily select one of them to
define the Accumulator.

In Perl, this is equivalent to doing this:

Accumulator initialisation:
   ::

       my $scalar_name;

Accumulator extension:
   ::

       $scalar_name = $scalar_name;           # Basic syntax
       $scalar_name = $output_parameter_name; # Extended syntax

Accumulator retrieval:
   ::

       say "Value: $scalar_name";


Pile
~~~~

Basic syntax:
    ``?accu_name=pile_name&accu_address=[]``

Extended syntax:
    ``?accu_name=pile_name&accu_address=[]&accu_input_variable=pile_component``

Retrieval:
  ::

      my $pile_ref = $self->param('pile_name');
      foreach my $pile_element (@{$pile_ref}) {
          # do something with $pile_element
      }


A pile is an unordered list. All the ``pile_name`` (or ``pile_component``
in the second form) values that are dataflown
into the Accumulator are aggregated into a list named ``pile_name``
in a *random* order.

In Perl, this is similar to doing this:

Accumulator initialisation:
   ::

       my @pile_name;

Accumulator extension:
   ::

       push @pile_name, $pile_name;             # Basic syntax
       push @pile_name, $pile_component;        # Extended syntax

Accumulator retrieval:
   ::

       foreach my $v (@pile_name) {
           say "Value: $v";
       }


Multiset
~~~~~~~~

Basic syntax:
    ``?accu_name=multiset_name&accu_address={}``

Extended syntax:
    ``?accu_name=multiset_name&accu_address={}&accu_input_variable=multiset_component``

Retrieval:
   ::
{'fraction_non_alphanumeric': 0.09443704197201866, 'fraction_numerical': 0.0, 'mean_word_length': 3.9505358615004122, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3940395', 'n_tokens_mistral': 1883, 'n_tokens_neox': 1761, 'n_words': 668}