<commit_before>## Welcome to the PEGTL

The Parsing Expression Grammar Template Library (PEGTL) is a C++11 header-only library for creating parsers according to a Parsing Expression Grammar (PEG).

## Intro

Grammars are written as regular C++ code, created with template programming (not template meta programming), i.e. nested template instantiations that naturally correspond to the inductive definition of PEGs or other parser combinator approaches.

A comprehensive set of parser rules that can be combined and extended by the user is included, as are mechanisms for debugging grammars and attaching user-defined actions to grammar rules.
Here is an example of how a PEG grammar rule is implemented as C++ class with the PEGTL.

```c++
  // PEG rule for integers consisting of a non-empty
  // sequence of digits with an optional sign:

  // integer ::= ( '+' / '-' )? digit+

  // The same parsing rule implemented with the PEGTL:

  struct integer
     : pegtl::seq< pegtl::opt< pegtl::one< '+', '-' > >,
                   pegtl::plus< pegtl::digit > > {};
```

PEGs are superficially similar to Context-Free Grammars (CFGs), however the more deterministic and procedural nature of PEGs gives rise to some very important differences.
The included [grammar analysis](https://github.com/ColinH/PEGTL/wiki/Grammar-Analysis) helps to find several typical errors in PEGs including **left recursion**.
For an in-depth description of PEGs see [Wikipedia page on PEGs](http://en.wikipedia.org/wiki/Parsing_expression_grammar) or [this paper on PEGs](http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf) by <NAME>.

## Design

The PEGTL is mostly concerned with parsing combinators and grammar rules and giving the user control over what else happens during a parsing run.

Whether and which actions are taken and which data structures are created during a parsing run are entirely up to the user of the library, however we provide some examples for typical situation like unescaping escape sequences in literal strings, building a generic JSON data structure and on-the-fly evaluation of arithmetic expressions.

Through the use of template programming and template specialisation it is possible to write a grammar once and use it in multiple ways by using different actions in different (or the same) parsing run.

Unlike Antlr and Yacc/Bison the grammar is expressed in C++ and part of the C++ source code.
Also the PEG formalism removes the separation into lexer and parser stages -- everything is done in a single grammar.

Unlike Spirit the grammar is implemented with compile-time template instantiations rather than run-time operator calls.
This leads to slightly increased compile times as the C++ compiler is given the task of optimising PEGTL grammars.

The PEGTL is designed to be "lean and mean".
The actual core library has about 3000 lines of code.
Emphasis is on simplicity and efficiency but without adding any large constructions for optimising parsers.

## Status

The master branch of the PEGTL is **always** stable in the sense that all known bugs are fixed and all unit tests run without errors. It is not stable in the sense that incompatible API changes can happen before/between major version releases.

[![Build Status](https://secure.travis-ci.org/ColinH/PEGTL.png)](http://travis-ci.org/ColinH/PEGTL)

For questions and suggestions please contact the authors at **pegtl (at) colin-hirsch.net**.

## Documentation

* [Getting Started](https://github.com/ColinH/PEGTL/wiki/Getting-Started)
* [Rules and Grammars](https://github.com/ColinH/PEGTL/wiki/Rules-and-Grammars)
* [Actions and States](https://github.com/ColinH/PEGTL/wiki/Actions-and-States)
* [Rule Reference](https://github.com/ColinH/PEGTL/wiki/Rule-Reference)
  * [Meta Rules](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#meta-rules)
  * [Combinators](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#combinators)
  * [Convenience](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#convenience)
  * [Atomic Rules](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#atomic-rules)<commit_msg>Use SVG instead of PNG from Travis CI<commit_after>## Welcome to the PEGTL

[![Build Status](https://travis-ci.org/ColinH/PEGTL.svg)](https://travis-ci.org/ColinH/PEGTL)

The Parsing Expression Grammar Template Library (PEGTL) is a C++11 header-only library for creating parsers according to a Parsing Expression Grammar (PEG).

## Intro

Grammars are written as regular C++ code, created with template programming (not template meta programming), i.e. nested template instantiations that naturally correspond to the inductive definition of PEGs or other parser combinator approaches.

A comprehensive set of parser rules that can be combined and extended by the user is included, as are mechanisms for debugging grammars and attaching user-defined actions to grammar rules.
Here is an example of how a PEG grammar rule is implemented as C++ class with the PEGTL.

```c++
  // PEG rule for integers consisting of a non-empty
  // sequence of digits with an optional sign:

  // integer ::= ( '+' / '-' )? digit+

  // The same parsing rule implemented with the PEGTL:

  struct integer
     : pegtl::seq< pegtl::opt< pegtl::one< '+', '-' > >,
                   pegtl::plus< pegtl::digit > > {};
```

PEGs are superficially similar to Context-Free Grammars (CFGs), however the more deterministic and procedural nature of PEGs gives rise to some very important differences.
The included [grammar analysis](https://github.com/ColinH/PEGTL/wiki/Grammar-Analysis) helps to find several typical errors in PEGs including **left recursion**.
For an in-depth description of PEGs see [Wikipedia page on PEGs](http://en.wikipedia.org/wiki/Parsing_expression_grammar) or [this paper on PEGs](http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf) by <NAME>ord.

## Design

The PEGTL is mostly concerned with parsing combinators and grammar rules and giving the user control over what else happens during a parsing run.

Whether and which actions are taken and which data structures are created during a parsing run are entirely up to the user of the library, however we provide some examples for typical situation like unescaping escape sequences in literal strings, building a generic JSON data structure and on-the-fly evaluation of arithmetic expressions.

Through the use of template programming and template specialisation it is possible to write a grammar once and use it in multiple ways by using different actions in different (or the same) parsing run.

Unlike Antlr and Yacc/Bison the grammar is expressed in C++ and part of the C++ source code.
Also the PEG formalism removes the separation into lexer and parser stages -- everything is done in a single grammar.

Unlike Spirit the grammar is implemented with compile-time template instantiations rather than run-time operator calls.
This leads to slightly increased compile times as the C++ compiler is given the task of optimising PEGTL grammars.

The PEGTL is designed to be "lean and mean".
The actual core library has about 3000 lines of code.
Emphasis is on simplicity and efficiency but without adding any large constructions for optimising parsers.

## Status

The master branch of the PEGTL is **always** stable in the sense that all known bugs are fixed and all unit tests run without errors. It is not stable in the sense that incompatible API changes can happen before/between major version releases.

For questions and suggestions please contact the authors at **pegtl (at) colin-hirsch.net**.

## Documentation

* [Getting Started](https://github.com/ColinH/PEGTL/wiki/Getting-Started)
* [Rules and Grammars](https://github.com/ColinH/PEGTL/wiki/Rules-and-Grammars)
* [Actions and States](https://github.com/ColinH/PEGTL/wiki/Actions-and-States)
* [Rule Reference](https://github.com/ColinH/PEGTL/wiki/Rule-Reference)
  * [Meta Rules](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#meta-rules)
  * [Combinators](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#combinators)
  * [Convenience](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#convenience)
  * [Atomic Rules](https://github.com/ColinH/PEGTL/wiki/Rule-Reference#atomic-rules)
{'fraction_non_alphanumeric': 0.0835258358662614, 'fraction_numerical': 0.0024316109422492403, 'mean_word_length': 4.546864463924478, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 21, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1902643', 'n_tokens_mistral': 2318, 'n_tokens_neox': 2211, 'n_words': 982}