<commit_before>DAX Processors
===========

-----
About
-----
DAX pipelines are defined by creating a YAML text file. If you are not familiar with YAML, start here:
https://learnxinyminutes.com/docs/yaml/.

The processor YAML defines the Environment, Inputs, Commands, and Outputs of your pipeline.

----------------
Processor Repos
----------------
There are several existing processors that can be used without modification. The processors in these
repositories can also provide valuable examples.

https://github.com/bud42/dax-processors

https://github.com/MASILab/yaml_processors


----------------
Overview
----------------
The processor file defines how a script to run a pipeline should be created. DAX will use the processor to generate scripts to be submitted to your cluster as jobs. The script will contain the
commands to download the inputs from XNAT, run the pipeline, and prepare the results to be uploaded back to XNAT (the actual uploading is performed by DAX via *dax upload*).

----------------
A "Simple" Example
----------------<commit_msg>Update processors.rst<commit_after>DAX Processors
===========

-----
About
-----
DAX pipelines are defined by creating YAML text files. If you are not familiar with YAML, start here:
https://learnxinyminutes.com/docs/yaml/.

A processor YAML file defines the Environment, Inputs, Commands, and Outputs of your pipeline.

----------------
Processor Repos
----------------
There are several existing processors that can be used without modification. The processors in these
repositories can also provide valuable examples.

https://github.com/bud42/dax-processors

https://github.com/MASILab/yaml_processors


----------------
Overview
----------------
The processor file defines how a script to run a pipeline should be created. DAX will use the processor to generate scripts to be submitted to your cluster as jobs. The script will contain the
commands to download the inputs from XNAT, run the pipeline, and prepare the results to be uploaded back to XNAT (the actual uploading is performed by DAX via *dax upload*).

----------------
A "Simple" Example
----------------
{'fraction_non_alphanumeric': 0.15150093808630394, 'fraction_numerical': 0.001876172607879925, 'mean_word_length': 2.829443447037702, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5282249', 'n_tokens_mistral': 538, 'n_tokens_neox': 505, 'n_words': 261}