<commit_before># Farm pattern

The **farm** pattern (or *task-farm*) is a streaming pattern that applies an operation to every element in a stream.

The interface to the **farm** pattern is provided by function `grppi::farm()`.
As all functions in *GrPPI*, this function takes as its first argument an
execution policy.

~~~{.cpp}
grppi::farm(exec, other_arguments...);
~~~

## Farm variants

There are several variants:

* Stand-alone *transforming farm*: Applies a transformation to a stream of 
  values produced by a generator and invokes a consumer for each generated result.
* Stand-alone *identity farm*: Consumes a stream of values produced by a
  generator.
* Composable *farm*: Defines a farm that can be used as a building block by another pattern (e.g. a *pipeline*).

## Key elements in a farm

The key elements of a **farm** are the **Generator**, the **Transformer**, and
the **Consumer**.

The central element in a farm is the **Transformer**. The operation may be any
C++ callable entity. This operation, is a unary operation taking a data item and
returning its transformation. Thus, a transformer `op` is any operation that,
given an input value `x` of type `T` makes valid the following:

~~~{.cpp}
U res{transformer(x)};
~~~

A *stand-alone farm* also has a **Generator**. The generator may be any C++
callable entity that produces values in a way that allows to signal an
indication of end-of stream. For this purpose, the farm requires that the
generator produces values of any type that is convertible to `bool` and can
dereferenced:

~~~{.cpp}
auto r = generator(); // r is a value generated by a generator.

if (!r) { /* ... */ } // !r is convertible to bool

if (r) {              // r is convertible to bool
  auto y = *r;        // r can be dereferenced.
}
~~~

Additionally a *farm* may have a **Consumer**. The consumer may be any C++
callable entity that takes values of the result type of the *Transformer*.

~~~{.cpp}
auto t = transformer(x);
consumer(t);
~~~

## Details on farm variants

### Stand-alone transforming farm

A *stand-alone transforming farm* transforms each generated element in a stream
and then consumes it.

---
**Example**: Generate a sequence of integers and print the square of each value.
~~~{.cpp}
int n = 10;
farm(exec,
  []() -> std::optional<int> {
    n--;
    if (n>0) return n;
    else     {};
  },
  [](int x) { return x*x; },
  [](int x) {
    std::cout << x << " ";
  }
);
~~~
---

### Stand-alone identity farm

A *stand-alone identity farm* generates a stream of data items and then consumes
them.

---
**Example**
~~~{.cpp}
ifstream file{"file.txt"};
farm(exec,
  []() -> std::optional<string> {
    std::string word;
    file >> word;
    if (file) return word;
    else return {};
  },
  [](auto word) {
    std::cout << word << " , " << word.length() << std::endl;
  },
);
~~~
---

### Composable farm

A *composable farm* applies a **Transformer** to each data item in a stream. The
**farm** can be inserted into another upper level pattern (which will
be responsible for generation and consumption)

---
**Example**
~~~{.cpp}
pipeline(exec,
  stageA,
  stageB,
  farm(exec, [](auto x) {
    return x.length();
  }),
  stageC
);
~~~
**Note**: For brevity we do not show here the details of other stages.
---

For composing complex patterns, the `farm()` function may be used to create an object that may be used later in the composition.

---
**Example**
~~~{.cpp}
auto print_long_words = farm(exec, [](auto x) {
  if (x.length() > 4) std::cout << x << std::endl;
});

pipeline(exec,
  stageA,
  stageB,
  print_long_words,
  stageC
);
~~~
**Note**: For brevity we do not show here the details of other stages.
---


<commit_msg>Revised doc and corrected typos and code mistakes<commit_after># Farm pattern

The **farm** pattern (or *task-farm*) is a streaming pattern that applies an operation to every element in a stream.

The interface to the **farm** pattern is provided by function `grppi::farm()`.
As all functions in *GrPPI*, this function takes as its first argument an
execution policy.

~~~{.cpp}
grppi::farm(exec, other_arguments...);
~~~

## Farm variants

There are several variants:

* Stand-alone *transforming farm*: Applies a transformation to a stream of 
  values produced by a generator and invokes a consumer for each generated result.
* Stand-alone *identity farm*: Consumes a stream of values produced by a
  generator.
* Composable *farm*: Defines a farm that can be used as a building block by another pattern (e.g. a *pipeline*).

## Key elements in a farm

The key elements of a **farm** are the **Generator**, the **Transformer**, and
the **Consumer**.

The central element in a farm is the **Transformer**. The operation may be any
C++ callable entity. This operation, is a unary operation taking a data item and
returning its transformation. Thus, a transformer `op` is any operation that,
given an input value `x` of type `T` makes valid the following:

~~~{.cpp}
U res{transformer(x)};
~~~

A *stand-alone farm* also has a **Generator**. The generator may be any C++
callable entity that produces values in a way that allows to signal an
indication of end-of stream. For this purpose, the farm requires that the
generator produces values of any type that is convertible to `bool` and can
be dereferenced:

~~~{.cpp}
auto r = generator(); // r is a value generated by a generator.

if (!r) { /* ... */ } // !r is convertible to bool

if (r) {              // r is convertible to bool
  auto y = *r;        // r can be dereferenced.
}
~~~

Additionally a *farm* may have a **Consumer**. The consumer may be any C++
callable entity that takes values of the result type of the *Transformer*.

~~~{.cpp}
auto t = transformer(x);
consumer(t);
~~~

## Details on farm variants

### Stand-alone transforming farm

A *stand-alone transforming farm* transforms each generated element in a stream
and then consumes it.

---
**Example**: Generate a sequence of integers and print the square of each value.
~~~{.cpp}
using namespace std;
using namespace experimental;
//...
int n = 10;
grppi::farm(exec,
  []() -> optional<int> {
    n--;
    if (n>0) return n;
    else     {};
  },
  [](int x) { return x*x; },
  [](int x) {
    cout << x << " ";
  }
);
~~~
---

### Stand-alone identity farm

A *stand-alone identity farm* generates a stream of data items and then consumes
them.

---
**Example**: Read a file and write to the standard output one word per line.
~~~{.cpp}
using namespace std;
using namespace experimental;
//...
ifstream file{"file.txt"};
grppi::farm(exec,
  [&file]() -> optional<string> {
    string word;
    file >> word;
    if (file) return word;
    else return {};
  },
  [](auto word) {
    cout << word << " , " << word.length() << endl;
  },
);
~~~
---

### Composable farm

A *composable farm* applies a **Transformer** to each data item in a stream. The
**farm** can be inserted into another upper level pattern (which will
be responsible for generation and consumption)

---
**Example**: Use a farm as a stage of a composed pipeline.
~~~{.cpp}
grppi::pipeline(exec,
  stageA,
  stageB,
  grppi::farm(exec, [](auto x) {
    return x.length();
  }),
  stageC
);
~~~
---
**Note**: For brevity we do not show here the details of other stages.

For composing complex patterns, the `farm()` function may be used to create an object that may be used later in the composition.

---
**Example**: Build a farm as a composable pattern and use later as a stage of a
pipeline.
~~~{.cpp}
auto print_long_words = grppi::farm(exec, [](auto x) {
  if (x.length() > 4) std::cout << x << std::endl;
});

grppi::pipeline(exec,
  stageA,
  stageB,
  print_long_words,
  stageC
);
~~~
---
**Note**: For brevity we do not show here the details of other stages.
{'fraction_non_alphanumeric': 0.13428608214239732, 'fraction_numerical': 0.0010299987125016094, 'mean_word_length': 3.574793875147232, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4570261', 'n_tokens_mistral': 2418, 'n_tokens_neox': 2322, 'n_words': 1066}