<commit_before>parse
----------

Simple parser constructor inspired by [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) which
describes grammars with composition of go types.

It generates syntax tree which could be used to fold the data inside into something useful.

This project is in **alpha state**, API may change in future.

## Limitations

- Has no concept of `string literal`, you could parse `"foo\"bar"` but you should fold the AST by hands
  This may change in future, I think we could introduce a separate rule type for this.
- Line reporting in AST is not implemented at this time, it reports only position in the string.
  This will change in the future, I think we could introduce an option to create a parser
  which will configure the line-break symbols.

## Example

> Example from `examples/` directory.

``` go<commit_msg>Fixed readme<commit_after>parse
----------

Simple parser constructor inspired by [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) which
describes grammars with composition of go types.

It generates syntax tree which could be used to fold the data inside into something useful.

This project is in **alpha state**, API may change in future.

## Limitations

- Has no concept of `string literal`, you could parse `"foo\"bar"` but you should fold the AST by hands.
  This may change in future, I think we could introduce a separate rule type for this.
- Line reporting in AST is not implemented at this time, it reports only position in the string.
  This will change in the future, I think we could introduce an option to create a parser
  which will configure the line-break symbols.

## Example

> Example from `examples/` directory.

``` go
{'fraction_non_alphanumeric': 0.075635103926097, 'fraction_numerical': 0.005773672055427252, 'mean_word_length': 4.023188405797101, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3153258', 'n_tokens_mistral': 464, 'n_tokens_neox': 445, 'n_words': 246}