<commit_before>17.10 Step location accessors

This section describes step accessors which are not basic to stack manipulation,
but that provide other Earley set location information about the parse.
A step's location in terms of Earley sets is called its ES location.
Every ES location is the ID of an Earley set.

These step accessors are implemented as macros.
All of these accessors always succeed, but if called when they are
irrelevant they return an unspecified value. In this context, an
“unspecified value” is a value that is either -1 or the ID of a
valid Earley set, but which is otherwise unpredictable.

ES location is only relevant for parse nodes.
A parse node is either a leaf node or a rule node.
The step type for a rule node is MARPA_STEP_RULE.
A leaf node is either a nulled node,
or a token node.
The step type for a nulled node is MARPA_STEP_NULLING_SYMBOL.
The step type for a token node is MARPA_STEP_TOKEN.

Every parse node has both a start ES location
and an end ES location.
The start ES location is the first ES location of
that parse node.

The end ES location of a leaf is the ES location where
the next leaf symbol in the fringe of the current parse would start.
Typically, this is the location where a leaf node actually starts,
but toward the end of a parse, there may not be an actual next leaf node.

The start ES location of a MARPA_RULE_STEP is the start ES location
of its first child node in the current parse tree.
The end ES location of a MARPA_RULE_STEP is the end ES location
of its last child node in the current parse tree.

Macro: Marpa_Earley_Set_ID marpa_v_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_RULE, MARPA_STEP_TOKEN, or MARPA_STEP_NULLING_SYMBOL,
the return value is the end ES location of the parse node.
If the current step type is anything else, the return value is an unspecified value. 

Macro: Marpa_Earley_Set_ID marpa_v_rule_start_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_RULE, the start ES location of the rule node.
If the current step type is anything else, an unspecified value. 

Macro: Marpa_Earley_Set_ID marpa_v_token_start_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_TOKEN or MARPA_STEP_NULLING_SYMBOL,
the start ES location of the leaf node.
If the current step type is anything else, an unspecified value. 

For every parse node of the current parse tree,
the Earley set length (ES length) of the node is the end ES location,
less the start ES location.
The ES length of a nulled node is always 0.

If v is a valuator whose current step type is MARPA_STEP_NULLING_SYMBOL, it is always the case that

  marpa_v_token_start_es_id(v) == marpa_v__es_id(v)

If v is a valuator whose current step type is MARPA_STEP_RULE or MARPA_STEP_TOKEN,
it is always the case that

  marpa_v_token_start_es_id(v) <= marpa_v__es_id(v)

As an example, let Null be a nulling symbol and let Tok be a non-nullable symbol.
Then a possible fringe is

  Null@0-0, Tok@0-1, Null@1-1, Tok@1-2, Null@2-2

where the fringe is ordered from left to right, and the notation Sym@m-n indicates that the symbol
Sym has ES start location m and ES end location n.
Another example is

  Null@0-0, Null@0-0, Tok@0-1, Null@1-1, Null@1-1, Tok@1-2, Null@2-2, Null@2-2

In this second example note that when a nulled leaf immediately follows another
nulled leaf, both leaves has the same start ES location and the same end ES location.
This makes sense, because nulled symbol instances do not advance the current ES location,
but it also implies that the ES locations cannot be used to uniquely identify a parse node.<commit_msg>Issue #92 -- Add doc fix suggested by @dabrahams.<commit_after>17.10 Step location accessors

This section describes step accessors which are not basic to stack manipulation,
but that provide other Earley set location information about the parse.
A step's location in terms of Earley sets is called its ES location.
Every ES location is the ID of an Earley set.

These step accessors are implemented as macros.
All of these accessors always succeed, but if called when they are
irrelevant they return an unspecified value. In this context, an
“unspecified value” is a value that is either -1 or the ID of a
valid Earley set, but which is otherwise unpredictable.

ES location is only relevant for parse nodes.
A parse node is either a leaf node or a rule node.
The step type for a rule node is MARPA_STEP_RULE.
A leaf node is either a nulled node,
or a token node.
The step type for a nulled node is MARPA_STEP_NULLING_SYMBOL.
The step type for a token node is MARPA_STEP_TOKEN.

Every parse node has both a start ES location
and an end ES location.
The start ES location is the first ES location of
that parse node.

The end ES location of a leaf is the ES location where
the next leaf symbol in the fringe of the current parse would start.
Typically, this is the location where a leaf node actually starts,
but toward the end of a parse, there may not be an actual next leaf node.

The start ES location of a MARPA_RULE_STEP is the start ES location
of its first child node in the current parse tree.
The end ES location of a MARPA_RULE_STEP is the end ES location
of its last child node in the current parse tree.

Macro: Marpa_Earley_Set_ID marpa_v_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_RULE, MARPA_STEP_TOKEN, or MARPA_STEP_NULLING_SYMBOL,
the return value is the end ES location of the parse node.
If the current step type is anything else, the return value is an unspecified value. 

Macro: Marpa_Earley_Set_ID marpa_v_rule_start_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_RULE, the start ES location of the rule node.
If the current step type is anything else, an unspecified value. 

Macro: Marpa_Earley_Set_ID marpa_v_token_start_es_id (Marpa_Value v)

    Return value: If the current step type is MARPA_STEP_TOKEN or MARPA_STEP_NULLING_SYMBOL,
the start ES location of the leaf node.
If the current step type is anything else, an unspecified value. 

For every parse node of the current parse tree,
the Earley set length (ES length) of the node is the end ES location,
less the start ES location.
The ES length of a nulled node is always 0.

If v is a valuator whose current step type is MARPA_STEP_NULLING_SYMBOL, it is always the case that

  marpa_v_token_start_es_id(v) == marpa_v__es_id(v)

If v is a valuator whose current step type is MARPA_STEP_RULE or MARPA_STEP_TOKEN,
it is always the case that

  marpa_v_token_start_es_id(v) <= marpa_v__es_id(v)

As an example, let Null be a nulling symbol and let Tok be a non-nullable symbol.
Then a possible fringe is

  Null@0-0, Tok@0-1, Null@1-1, Tok@1-2, Null@2-2

where the fringe is ordered from left to right, and the notation Sym@m-n indicates that the symbol
Sym has ES start location m and ES end location n.
Another example is

  Null@0-0, Null@0-0, Tok@0-1, Null@1-1, Null@1-1, Tok@1-2, Null@2-2, Null@2-2

In this second example note that when a nulled leaf immediately follows another
nulled leaf, both leaves has the same start ES location and the same end ES location.
This makes sense, because nulled symbol instances do not advance the current ES location,
but it also implies that the ES locations and LHS symbol
cannot be used to uniquely identify a parse node.
{'fraction_non_alphanumeric': 0.035574191803056945, 'fraction_numerical': 0.008927363722440147, 'mean_word_length': 4.29277022190408, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7076014', 'n_tokens_mistral': 2273, 'n_tokens_neox': 2175, 'n_words': 1207}