<commit_before># Data processing in Magenta

Magenta has a lot of different models which require different types of inputs. Some models train on melodies, some on raw audio or midi data. Being able to convert easily between these different data types is essential. We define a `Pipeline` which is a data processing module that transforms input data types to output data types. By connecting pipelines together, new data pipelines can be quickly built for new models.

Files:

* [pipeline.py](/magenta/pipelines/pipeline.py) defines the `Pipeline` abstract class and utility functions for running a `Pipeline` instance.
* [pipelines_common.py](/magenta/pipelines/pipelines_common.py) contains some `Pipeline` implementations that convert to common data types, like `Melody`.
* [dag_pipeline.py](/magenta/pipelines/dag_pipeline.py) defines a `Pipeline` subclass `DAGPipeline` which connects arbitrary pipelines together inside it. These `Pipelines` can be connected into any directed acyclic graph (DAG).
* [statistics.py](/magenta/pipelines/statistics.py) defines the `Statistic` abstract class and implementations. Statistics are useful for reporting about data processing.
* [chord_pipelines.py](/magenta/pipelines/chord_pipelines.py), [drum_pipelines.py](/magenta/pipelines/chord_pipelines.py), [melody_pipelines.py](/magenta/pipelines/chord_pipelines.py), and [lead_sheet_pipelines.py](/magenta/pipelines/chord_pipelines.py) define extractor pipelines for different types of musical event sequences.

## Pipeline

All pipelines implement the abstract class Pipeline. Each Pipeline defines what its input and output look like. A pipeline can take as input a single object or dictionary mapping names to inputs. The output is given as a list of objects or a dictionary mapping names to lists of outputs. This allows the pipeline to output multiple items from a single input.

Pipeline has two methods:

* `transform(input_object)` converts a single input to one or many outputs.<commit_msg>Fix links to different extractor pipelines (#733)<commit_after># Data processing in Magenta

Magenta has a lot of different models which require different types of inputs. Some models train on melodies, some on raw audio or midi data. Being able to convert easily between these different data types is essential. We define a `Pipeline` which is a data processing module that transforms input data types to output data types. By connecting pipelines together, new data pipelines can be quickly built for new models.

Files:

* [pipeline.py](/magenta/pipelines/pipeline.py) defines the `Pipeline` abstract class and utility functions for running a `Pipeline` instance.
* [pipelines_common.py](/magenta/pipelines/pipelines_common.py) contains some `Pipeline` implementations that convert to common data types, like `Melody`.
* [dag_pipeline.py](/magenta/pipelines/dag_pipeline.py) defines a `Pipeline` subclass `DAGPipeline` which connects arbitrary pipelines together inside it. These `Pipelines` can be connected into any directed acyclic graph (DAG).
* [statistics.py](/magenta/pipelines/statistics.py) defines the `Statistic` abstract class and implementations. Statistics are useful for reporting about data processing.
* [chord_pipelines.py](/magenta/pipelines/chord_pipelines.py), [drum_pipelines.py](/magenta/pipelines/drum_pipelines.py), [melody_pipelines.py](/magenta/pipelines/melody_pipelines.py), and [lead_sheet_pipelines.py](/magenta/pipelines/lead_sheet_pipelines.py) define extractor pipelines for different types of musical event sequences.

## Pipeline

All pipelines implement the abstract class Pipeline. Each Pipeline defines what its input and output look like. A pipeline can take as input a single object or dictionary mapping names to inputs. The output is given as a list of objects or a dictionary mapping names to lists of outputs. This allows the pipeline to output multiple items from a single input.

Pipeline has two methods:

* `transform(input_object)` converts a single input to one or many outputs.
{'fraction_non_alphanumeric': 0.06736266467810093, 'fraction_numerical': 0.0007457121551081282, 'mean_word_length': 5.50080775444265, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4303608', 'n_tokens_mistral': 1073, 'n_tokens_neox': 1031, 'n_words': 475}