<commit_before>FATE Pipeline
=============

Pipeline is a high-level python API that allows user to design, start,
and query FATE jobs in a sequential manner. FATE Pipeline is designed to
be user-friendly and consistent in behavior with FATE command line
tools. User can customize job workflow by adding components to pipeline
and then initiate a job with one call. In addition, Pipeline provides
functionality to run prediction and query information after fitting a
pipeline. Run the `mini demo <./demo/pipeline-mini-demo.py>`__ to have a
taste of how FATE Pipeline works. Default values of party ids and work
mode may need to be modified depending on the deployment setting.

.. code:: bash

   python pipeline-mini-demo.py config.yaml

For more pipeline demo, please refer to
`examples <../../examples/pipeline>`__.

A FATE Job is A Sequence
------------------------

A FATE job includes a sequence of tasks. FATE pipeline provides
easy-to-use tools to configure order and setting of the tasks.

FATE is written in a modular style. Modules are designed to have input
and output data and model. Therefore two modules are connected when
output of one module is set to be input of another module. By tracing
how one data set is processed through FATE modules, we can see that a
FATE job is in fact formed by a sequence of sub-tasks. For example, in
the `mini demo <./demo/pipeline-mini-demo.py>`__ above, guest’s data is
first read in by ``Reader``, then loaded into ``DataIO``. Overlapping
ids between guest and host are then found by running data through
``Intersection``. Finally, ``HeteroLR`` model is fit on the data. Each
of the listed modules run a small task with the data, and together they
constitute a model training job.

Beyond the given mini demo, a job may include multiple data sets and
models. For more pipeline examples, please refer to `demos <./demo/>`__.

Install Pipeline
----------------

Pipeline CLI
~~~~~~~~~~~~

After successfully installed FATE Pipeline, user needs configure Pipeline.
Pipeline provides a command line tool for quick setup. Run the following
command for more information.

.. code:: bash
<commit_msg>fix doc link<commit_after>FATE Pipeline
=============

Pipeline is a high-level python API that allows user to design, start,
and query FATE jobs in a sequential manner. FATE Pipeline is designed to
be user-friendly and consistent in behavior with FATE command line
tools. User can customize job workflow by adding components to pipeline
and then initiate a job with one call. In addition, Pipeline provides
functionality to run prediction and query information after fitting a
pipeline. Run the `mini demo <./demo/pipeline-mini-demo.py>`__ to have a
taste of how FATE Pipeline works. Default values of party ids and work
mode may need to be modified depending on the deployment setting.

.. code:: bash

   python pipeline-mini-demo.py config.yaml

For more pipeline demo, please refer to
`examples <../../examples/pipeline>`__.

A FATE Job is A Sequence
------------------------

A FATE job includes a sequence of tasks. FATE pipeline provides
easy-to-use tools to configure order and setting of the tasks.

FATE is written in a modular style. Modules are designed to have input
and output data and model. Therefore two modules are connected when
output of one module is set to be input of another module. By tracing
how one data set is processed through FATE modules, we can see that a
FATE job is in fact formed by a sequence of sub-tasks. For example, in
the `mini demo <./demo/pipeline-mini-demo.py>`__ above, guest’s data is
first read in by ``Reader``, then loaded into ``DataIO``. Overlapping
ids between guest and host are then found by running data through
``Intersection``. Finally, ``HeteroLR`` model is fit on the data. Each
of the listed modules run a small task with the data, and together they
constitute a model training job.

Beyond the given mini demo, a job may include multiple data sets and
models. For more pipeline examples, please refer to `examples <../../examples/pipeline>`__.

Install Pipeline
----------------

Pipeline CLI
~~~~~~~~~~~~

After successfully installed FATE Pipeline, user needs configure Pipeline.
Pipeline provides a command line tool for quick setup. Run the following
command for more information.

.. code:: bash

{'fraction_non_alphanumeric': 0.08217270194986072, 'fraction_numerical': 0.0, 'mean_word_length': 3.847019122609674, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5193173', 'n_tokens_mistral': 1102, 'n_tokens_neox': 1057, 'n_words': 648}