<commit_before>Let's get started!
==================

To begin with Bonobo, you need to install it in a working python 3.5+ environment, and you'll also need cookiecutter
to bootstrap your project.

.. code-block:: shell-session

    $ pip install bonobo cookiecutter

See :doc:`/install` for more options.


Create an empty project
:::::::::::::::::::::::

Your ETL code will live in ETL projects, which are basically a bunch of files, including python code, that bonobo
can run.

.. code-block:: shell-session

    $ bonobo init tutorial

This will create a `tutorial` directory (`content description here <https://www.bonobo-project.org/with/cookiecutter>`_).

To run this project, use:

.. code-block:: shell-session

    $ bonobo run tutorial


Write a first transformation
::::::::::::::::::::::::::::

Open `tutorial/main.py`, and delete all the code here.

A transformation can be whatever python can call. Simplest transformations are functions and generators.

Let's write one:

.. code-block:: python

    def transform(x):
        return x.upper()

Easy.

.. note::

    This function is very similar to :func:`str.upper`, which you can use directly.

Let's write two more transformations for the "extract" and "load" steps. In this example, we'll generate the data from
scratch, and we'll use stdout to "simulate" data-persistence.

.. code-block:: python

    def extract():
        yield 'foo'
        yield 'bar'
        yield 'baz'

    def load(x):
        print(x)

Bonobo makes no difference between generators (yielding functions) and regular functions. It will, in all cases, iterate
on things returned, and a normal function will just be seen as a generator that yields only once.

.. note::

    Once again, you should use the builtin :func:`print` directly instead of this `load()` function.


Create a transformation graph
:::::::::::::::::::::::::::::

Amongst other features, Bonobo will mostly help you there with the following:

* Execute the transformations in independant threads
* Pass the outputs of one thread to other(s) thread(s) inputs.

To do this, it needs to know what data-flow you want to achieve, and you'll use a :class:`bonobo.Graph` to describe it.

.. code-block:: python

    import bonobo

    graph = bonobo.Graph(extract, transform, load)

    if __name__ == '__main__':
        bonobo.run(graph)

.. graphviz::

    digraph {
        rankdir = LR;
        stylesheet = "../_static/graphs.css";

        BEGIN [shape="point"];
        BEGIN -> "extract" -> "transform" -> "load";
    }

.. note::

    The `if __name__ == '__main__':` section is not required, unless you want to run it directly using the python
    interpreter.


Execute the job
:::::::::::::::<commit_msg>Add a note about the graph variable<commit_after>Let's get started!
==================

To begin with Bonobo, you need to install it in a working python 3.5+ environment, and you'll also need cookiecutter
to bootstrap your project.

.. code-block:: shell-session

    $ pip install bonobo cookiecutter

See :doc:`/install` for more options.


Create an empty project
:::::::::::::::::::::::

Your ETL code will live in ETL projects, which are basically a bunch of files, including python code, that bonobo
can run.

.. code-block:: shell-session

    $ bonobo init tutorial

This will create a `tutorial` directory (`content description here <https://www.bonobo-project.org/with/cookiecutter>`_).

To run this project, use:

.. code-block:: shell-session

    $ bonobo run tutorial


Write a first transformation
::::::::::::::::::::::::::::

Open `tutorial/main.py`, and delete all the code here.

A transformation can be whatever python can call. Simplest transformations are functions and generators.

Let's write one:

.. code-block:: python

    def transform(x):
        return x.upper()

Easy.

.. note::

    This function is very similar to :func:`str.upper`, which you can use directly.

Let's write two more transformations for the "extract" and "load" steps. In this example, we'll generate the data from
scratch, and we'll use stdout to "simulate" data-persistence.

.. code-block:: python

    def extract():
        yield 'foo'
        yield 'bar'
        yield 'baz'

    def load(x):
        print(x)

Bonobo makes no difference between generators (yielding functions) and regular functions. It will, in all cases, iterate
on things returned, and a normal function will just be seen as a generator that yields only once.

.. note::

    Once again, you should use the builtin :func:`print` directly instead of this `load()` function.


Create a transformation graph
:::::::::::::::::::::::::::::

Amongst other features, Bonobo will mostly help you there with the following:

* Execute the transformations in independant threads
* Pass the outputs of one thread to other(s) thread(s) inputs.

To do this, it needs to know what data-flow you want to achieve, and you'll use a :class:`bonobo.Graph` to describe it.

.. code-block:: python

    import bonobo

    graph = bonobo.Graph(extract, transform, load)

    if __name__ == '__main__':
        bonobo.run(graph)

.. graphviz::

    digraph {
        rankdir = LR;
        stylesheet = "../_static/graphs.css";

        BEGIN [shape="point"];
        BEGIN -> "extract" -> "transform" -> "load";
    }

.. note::

    The `if __name__ == '__main__':` section is not required, unless you want to run it directly using the python
    interpreter.

    The name of the `graph` variable is arbitrary, but this variable must be global and available unconditionally.
    Do not put it in its own function or in the `if __name__ == '__main__':` section.


Execute the job
:::::::::::::::
{'fraction_non_alphanumeric': 0.12473572938689217, 'fraction_numerical': 0.0007047216349541931, 'mean_word_length': 3.431693989071038, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '202041', 'n_tokens_mistral': 1624, 'n_tokens_neox': 1506, 'n_words': 694}