<commit_before>text-parser
===========
[![Build status](https://ci.appveyor.com/api/projects/status/01jxm32c7i9769ef?svg=true)](https://ci.appveyor.com/project/StevenLiekens/text-parser)

Base classes for object-oriented recursive descent parsers, designed to be extended with custom grammars and grammar parsers.<commit_msg>Updating README<commit_after>Structured Language (SLANG)
===========
[![Build status](https://ci.appveyor.com/api/projects/status/01jxm32c7i9769ef?svg=true)](https://ci.appveyor.com/project/StevenLiekens/text-parser)

# About
This repository contains library classes that deal with structured languages.

Examples of structured languages include...

* Programming languages
    * C#, JavaScript, SQL
* Data interchange formats
    * XML, JSON
* Text-based protocols
    * HTTP, FTP


Background and Goals
===========

All structured languages have one thing in common: each language has a formal syntax specification.
These specifications define a set of grammar rules for the language. Programmers use these grammar rules to create programs that split structured text into its core components before actually processing the text.

Today, grammars are most commonly defined in what is called Augmented BNF (ABNF) notation. The ABNF specification itself defines a set of core grammar rules, which other specifications can then use to define a formal syntax for their language.

An example that is commonly used is the formal syntax specification for integer numbers. An integer has one or more digits. A digit can be any decimal digit, and is a core ABNF rule.

```abnf
INT   = 1*DIGIT                      ; INT is a custom rule, invented for this demonstration
DIGIT = "0" / "1" / "2" / "3" / "4"  ; DIGIT is a core ABNF rule
        / "5" / "6" / "7" /"8" / "9"
```



The goal of this project is to provide a set of base classes that you can use to implement your own grammar parsers.

See also:
* RFC 5234: [Augmented BNF for Syntax Specifications](https://tools.ietf.org/html/rfc5234)

Important: there is currently no support for automatically generating parsers. The idea here is that if you want to do it properly, then do it yourself.
{'fraction_non_alphanumeric': 0.08948339483394833, 'fraction_numerical': 0.017066420664206643, 'mean_word_length': 4.239130434782608, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '273787', 'n_tokens_mistral': 636, 'n_tokens_neox': 577, 'n_words': 265}