<commit_before>Jormungand
==========

Jormungand is a plugin-based framework for creating applications that extracting data from varied sources,
processing and validated the extracted data and then storing it again in a common format.

Jormungand defines a basic model for the process defined above and allows for specific implementations of various
stages in the process to be added or removed in a pluggable fashion.

Jormungand also provides a CLI script that is used to intiate the extraction post-process with a specified
configuration of plugins and input.


The Jormungand Process Model
-------------------------

Jormungand defines its basic process model as follows:

1. Data Model
2. Jormungand
3. Post-Processing
4. Validation
5. Storage<commit_msg>Updated README with some roadmap items.<commit_after>Jormungand
==========

Jormungand is a plugin-based framework for creating applications that extracting data from varied sources,
processing and validated the extracted data and then storing it again in a common format.

Jormungand defines a basic model for the process defined above and allows for specific implementations of various
stages in the process to be added or removed in a pluggable fashion.

Jormungand also provides a CLI script that is used to intiate the extraction post-process with a specified
configuration of plugins and input.


The Jormungand Process Model
-------------------------

Jormungand defines its basic process model as follows:

1. Data Model
2. Jormungand
3. Post-Processing
4. Validation
5. Storage


Roadmap
-------

The following items would be nice to have in the future:

1. Multi-processing support
2. Transposing data from one Data Model to another.
{'fraction_non_alphanumeric': 0.06752789195537287, 'fraction_numerical': 0.007046388725778039, 'mean_word_length': 3.9391304347826086, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6009308', 'n_tokens_mistral': 423, 'n_tokens_neox': 402, 'n_words': 238}