<commit_before># Incremental Input

The PEGTL is designed and optimised for parsing single contiguous blocks of memory, e.g. the contents of a file made available via `mmap(2)`, or the contents of a `std::string`.

In cases where the input does not fit into memory, or there are other reasons to not create a single memory block containing all input data, it is possible, with a little help from the grammar, to perform incremental parsing, where the data is incrementally made available, e.g. when reading from a stream.

## Grammars and Buffering

A buffer is used to keep a portion of the input data in a contiguous memory block.
The buffer is allocated at the begin of the parsing run with a user-supplied maximum size.

The maximum buffer size usually depends on the grammar, the actions, and the input data.
It must be chosen large enough to keep the data required for (a) any backtracking, and (b) all action invocations.

The buffer is automatically filled by the parsing rules that require input data, however **discarding data from the buffer is** (currently) **not automatic**:
The grammar has to call [`discard`](Rule-Reference.md#discard) in appropriate places to free the buffer again.

More precisely, each rule that uses one of the following methods on the input will implicitly make a call to `tao::pegtl::buffer_input<>::require( amount )`.
(The `empty()`-method uses a hard-coded `amount` of 1.)

```c++
namespace tao
{
   namespace pegtl
   {
      template< class Reader,
                typename Eol = lf_crlf_eol >
      class buffer_input
      {
         empty();
         size( const std::size_t amount );
         end( const std::size_t amount );
         ...<commit_msg>Fix eol parameter<commit_after># Incremental Input

The PEGTL is designed and optimised for parsing single contiguous blocks of memory, e.g. the contents of a file made available via `mmap(2)`, or the contents of a `std::string`.

In cases where the input does not fit into memory, or there are other reasons to not create a single memory block containing all input data, it is possible, with a little help from the grammar, to perform incremental parsing, where the data is incrementally made available, e.g. when reading from a stream.

## Grammars and Buffering

A buffer is used to keep a portion of the input data in a contiguous memory block.
The buffer is allocated at the begin of the parsing run with a user-supplied maximum size.

The maximum buffer size usually depends on the grammar, the actions, and the input data.
It must be chosen large enough to keep the data required for (a) any backtracking, and (b) all action invocations.

The buffer is automatically filled by the parsing rules that require input data, however **discarding data from the buffer is** (currently) **not automatic**:
The grammar has to call [`discard`](Rule-Reference.md#discard) in appropriate places to free the buffer again.

More precisely, each rule that uses one of the following methods on the input will implicitly make a call to `tao::pegtl::buffer_input<>::require( amount )`.
(The `empty()`-method uses a hard-coded `amount` of 1.)

```c++
namespace tao
{
   namespace pegtl
   {
      template< class Reader,
                typename Eol = eol::lf_crlf >
      class buffer_input
      {
         empty();
         size( const std::size_t amount );
         end( const std::size_t amount );
         ...
{'fraction_non_alphanumeric': 0.06821523081446633, 'fraction_numerical': 0.0011761246692149367, 'mean_word_length': 3.7184466019417477, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6403631', 'n_tokens_mistral': 912, 'n_tokens_neox': 860, 'n_words': 492}