<commit_before>===========================================
Jug: A Task-Based Parallelization Framework
===========================================

What is Jug?
------------

Jug allows you to write code that is broken up into tasks and run different
tasks on different processors.

It currently has two backends. The first uses the filesystem to communicate
between processes and works correctly over NFS, so you can coordinate processes
on different machines. The second is based on redis so the processes only need
the capability to connect to a common redis server.

Jug also takes care of saving all the intermediate results to the backend in a
way that allows them to be retrieved later.


Documentation Contents
----------------------

.. toctree::
    :maxdepth: 2

    tutorial.rst
    decrypt-example.rst
    text-example.rst
    subcommands.rst
    types.rst
    tasks.rst
    idioms.rst
    barrier.rst
    compound.rst
    backends.rst
    configuration.rst
    faq.rst
    why.rst
    api.rst

Examples
--------

Short Example
~~~~~~~~~~~~~

Here is a one minute example. Save the following to a file called ``primes.py``::

    from jug import TaskGenerator
    from time import sleep
<commit_msg>DOC: Add utilities.rst to tree<commit_after>===========================================
Jug: A Task-Based Parallelization Framework
===========================================

What is Jug?
------------

Jug allows you to write code that is broken up into tasks and run different
tasks on different processors.

It currently has two backends. The first uses the filesystem to communicate
between processes and works correctly over NFS, so you can coordinate processes
on different machines. The second is based on redis so the processes only need
the capability to connect to a common redis server.

Jug also takes care of saving all the intermediate results to the backend in a
way that allows them to be retrieved later.


Documentation Contents
----------------------

.. toctree::
    :maxdepth: 2

    tutorial.rst
    decrypt-example.rst
    text-example.rst
    subcommands.rst
    types.rst
    tasks.rst
    idioms.rst
    barrier.rst
    compound.rst
    utilities.rst
    backends.rst
    configuration.rst
    faq.rst
    why.rst
    api.rst

Examples
--------

Short Example
~~~~~~~~~~~~~

Here is a one minute example. Save the following to a file called ``primes.py``::

    from jug import TaskGenerator
    from time import sleep

{'fraction_non_alphanumeric': 0.15006100040666936, 'fraction_numerical': 0.0008133387555917039, 'mean_word_length': 3.0728476821192054, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2446500', 'n_tokens_mistral': 664, 'n_tokens_neox': 611, 'n_words': 289}