<commit_before># Major Planned Features

## Boolean operations and type

* contains (for lists or strings)

## Query Language (Experimental)

You should be able to ask the compiler to tell you any value or set of values in the
compiled configuration.

## Translation Language (Experimental)

For some configuration file formats we need a way to specify a particular
organiztion for a given configuration structure (i.e. xml attribute or tag?).

Some options here could be:

* Simple data export (json) 
* A Functional Transform similar to xslt or css transforms
* A Templating language
* DSL's
* Annotations

## Built In testing as a part of the language

* A DSL for assertions.
* A DSL for identifying tests.
* A way to run tests only during validation stage.

# Minor Fixes and Polish

* Compiler caching?
* Strings as tuple fields?
* Streaming Parsing
* Casting between types?
* Better error messages.
* Flags should allow different seperators for prefixed flags.
* YAML export
* HCL export

# Release Checklist

* Cargo test
* Cargo fmt
* Update Cargo.toml version.
* Tag git commit with version tag.
* Cargo publish<commit_msg>minor updates to our TODO list.<commit_after># Major Planned Features

## Boolean operations and type

* contains (for lists or strings)

## Query Language (Experimental)

You should be able to ask the compiler to tell you any value or set of values in the
compiled configuration.

## Translation Language (Experimental)

For some configuration file formats we need a way to specify a particular
organiztion for a given configuration structure (i.e. xml attribute or tag?).

Some options here could be:

* Simple data export (json) 
* A Functional Transform similar to xslt or css transforms
* A Templating language
* DSL's
* Annotations

## Built In testing as a part of the language

* A DSL for assertions.
* A DSL for identifying tests.
* A way to run tests only during validation stage.

# Minor Fixes and Polish

* Compiler caching (interface has been defined)
* Strings as tuple fields?
* Streaming Parsing?
* Casting between types?
* Better error messages.
* Flags should allow different seperators for prefixed flags.
* YAML export
* HCL export

# Release Checklist

* Cargo test
* Cargo fmt
* Update Cargo.toml version.
* Tag git commit with version tag.
* Cargo publish
{'fraction_non_alphanumeric': 0.056636402939904885, 'fraction_numerical': 0.0, 'mean_word_length': 4.247165532879818, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2504884', 'n_tokens_mistral': 643, 'n_tokens_neox': 607, 'n_words': 330}