<commit_before># Workflow

The workflow streamlines your data imports/exports. Using the workflow, you can
re-usable pipelines for processing your data. Each pipeline takes place along 
the following lines:

1. Construct a [reader](readers.md).
2. Construct a workflow, passing the reader to it. Optionally set a logger on 
   the workflow. 
3. Add at least one [writer](writers.md) to the workflow. Optionally, customize 
   the workflow by adding filters, converters, mappers, etc.
4. Process the workflow. This will read data from the reader, filters and 
   convert the data, and write the output to each of the writers. At the end of
   the process, a [Result](#the-workflow-result) object is returned which 
   contains counts and information about (failed) reads and writes.
   <commit_msg>Fix typo<commit_after># Workflow

The workflow streamlines your data imports/exports. Using the workflow, you can create
re-usable pipelines for processing your data. Each pipeline takes place along 
the following lines:

1. Construct a [reader](readers.md).
2. Construct a workflow, passing the reader to it. Optionally set a logger on 
   the workflow. 
3. Add at least one [writer](writers.md) to the workflow. Optionally, customize 
   the workflow by adding filters, converters, mappers, etc.
4. Process the workflow. This will read data from the reader, filters and 
   convert the data, and write the output to each of the writers. At the end of
   the process, a [Result](#the-workflow-result) object is returned which 
   contains counts and information about (failed) reads and writes.
   
{'fraction_non_alphanumeric': 0.06140350877192982, 'fraction_numerical': 0.005012531328320802, 'mean_word_length': 4.006269592476489, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2707574', 'n_tokens_mistral': 436, 'n_tokens_neox': 405, 'n_words': 226}