<commit_before># Copyright 2012 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::BNF - BNF Interface

=head1 Synopsis

=for Marpa::R2::Display
name: Stuifzand Synopsis
partial: 1
normalize-whitespace: 1

    use Marpa::R2;

    my $grammar = Marpa::R2::Grammar->new(
        {   
            actions        => 'My_Actions',
            default_action => 'do_first_arg',
            source          => \(<<'END_OF_SOURCE'),
    :start ::= Script
    Script ::= Expression+ separator => <op comma> action => do_script
    Expression ::=
        Number
        | (<op lparen>) Expression (<op rparen>) action => do_parens assoc => group
       || Expression (<op pow>) Expression action => do_pow assoc => right
       || Expression (<op times>) Expression action => do_multiply
        | Expression (<op divide>) Expression action => do_divide
       || Expression (<op add>) Expression action => do_add
        | Expression (<op subtract>) Expression action => do_subtract
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 Overview

This page is the reference for the Marpa's "BNF interface",
also called its "Stuifzand interface".
The Stuifzand interface allows strings
in place of the rule descriptors as value
of the C<rules> named
arguments of Marpa grammars.
These strings are called BNF source strings
or Stuifzand source strings.
BNF source strings are specified using the 
L<the C<source> named argument of Marpa::R2's grammars|Marpa::R2::Grammar/"source">.

=head1 The source file

=head2 Structure

Whitespace separates tokens, but is otherwise ignored.
Comments are equivalent to whitespace.
A hash ("C<#>") character starts a comment,
which continues to the end of the line.

The Stuifzand format consists of rule declarations.
Each rule declaraton consists of a left hand side (LHS),
followed by the declaration operator ("C<::=>")
and a right hand side (RHS).
The left and right hand sides
are described in detail below,
in the discussion of each rule by type.

=head2 Symbol names

Symbol names can be either "bare" or enclosed in angle brackets.
Bare symbols names must consist entirely of Perl word characters
(alphanumerics, plus the underscore).
The angle brackets, if used, serve to "quote"
the symbol name,
and will not be part of the symbol name.

If angle brackets are used, symbol names may also contain whitespace,
as in

=for Marpa::R2::Display
ignore: 1

    <op comma>

=for Marpa::R2::Display::End

A whitespace sequence inside angle brackets can
include any whitespace character that is legal in Perl,
including newlines.
This allows very long symbol names to be line wrapped,
if necessary.

Unlike angle brackets, the whitespace does become part of the symbol name,
but it does so in a "normalized" form.
Leading and trailing whitespace in the name is discarded,
and all other whitespace sequences are converted to a single ASCII
space character.
This means that

=for Marpa::R2::Display
ignore: 1

    < op comma  >
    <op   comma>
    <     op comma>

=for Marpa::R2::Display::End

and even


=for Marpa::R2::Display
ignore: 1

    <op
    comma>

=for Marpa::R2::Display::End

will all be treated as if they were a single symbol name,
one which is in effect equivalent to C<< <op comma> >>.

=head2 Types of rules

Rules are of four kinds:
the start rule,
empty rules,
quantified rules
and prioritized rules.
A simple rule,
with only a series of RHS symbols
on their RHS,
is treated as a
prioritized rule with only one alternative
and only one priority.
In a large grammar,
most rules are prioritized rules.

=head2 Start rule

Every source file should have a start rule.
The LHS of this rule is the C<:start> pseudo-symbol.
The RHS must be a single symbol name.
The start rule does not accept any rule adverbs.

=head2 Empty rules

An empty rule is a rule with an empty RHS.
A empty right hand side is what the name suggests,
no symbols, adverbs or punctuation.
A empty rule makes its LHS symbol a nullable symbol.
Empty rules may have an C<action> adverb.

=head2 Quantified rules

A quantified right hand side defines a sequence rule.
Its RHS consists of a single symbol name,
followed by a "quantifier".
The quantifer is either
a star ("C<*>"),
or a plus sign ("C<+>")<commit_msg>Documenting Stuifzand interface<commit_after># Copyright 2012 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::BNF - BNF Interface

=head1 Synopsis

=for Marpa::R2::Display
name: Stuifzand Synopsis
partial: 1
normalize-whitespace: 1

    use Marpa::R2;

    my $grammar = Marpa::R2::Grammar->new(
        {   
            actions        => 'My_Actions',
            default_action => 'do_first_arg',
            source          => \(<<'END_OF_SOURCE'),
    :start ::= Script
    Script ::= Expression+ separator => <op comma> action => do_script
    Expression ::=
        Number
        | (<op lparen>) Expression (<op rparen>) action => do_parens assoc => group
       || Expression (<op pow>) Expression action => do_pow assoc => right
       || Expression (<op times>) Expression action => do_multiply
        | Expression (<op divide>) Expression action => do_divide
       || Expression (<op add>) Expression action => do_add
        | Expression (<op subtract>) Expression action => do_subtract
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 Overview

This page is the reference for the Marpa's "BNF interface",
also called its "Stuifzand interface".
The Stuifzand interface allows strings
in place of the rule descriptors as value
of the C<rules> named
arguments of Marpa grammars.
These strings are called BNF source strings
or Stuifzand source strings.
BNF source strings are specified using the 
L<the C<source> named argument of Marpa::R2's grammars|Marpa::R2::Grammar/"source">.

=head1 The source file

=head2 Structure

Whitespace separates tokens, but is otherwise ignored.
Comments are equivalent to whitespace.
A hash ("C<#>") character starts a comment,
which continues to the end of the line.

The Stuifzand format consists of rule declarations.
Each rule declaraton consists of, in order:

=over 4

=item * A left hand side (LHS).
This will be a
symbol or a pseudo-symbol.

=item * The declaration operator ("C<::=>").

=item * A right hand side (RHS).
For each type of rule,
the RHS will be described in detail below.
In the case of empty rules,
the RHS will be empty.

=item * An adverb list.
Each adverbs consist of a keyword,
the adverb operator ("C<< => >>"),
and the adverb's value.
The adverb list may be zero length.

=back

=head2 Symbol names

Symbol names can be either "bare" or enclosed in angle brackets.
Bare symbols names must consist entirely of Perl word characters
(alphanumerics, plus the underscore).
The angle brackets, if used, serve to "quote"
the symbol name,
and will not be part of the symbol name.

If angle brackets are used, symbol names may also contain whitespace,
as in

=for Marpa::R2::Display
ignore: 1

    <op comma>

=for Marpa::R2::Display::End

A whitespace sequence inside angle brackets can
include any whitespace character that is legal in Perl,
including newlines.
This allows very long symbol names to be line wrapped,
if necessary.

Unlike angle brackets, the whitespace does become part of the symbol name,
but it does so in a "normalized" form.
Leading and trailing whitespace in the name is discarded,
and all other whitespace sequences are converted to a single ASCII
space character.
This means that

=for Marpa::R2::Display
ignore: 1

    < op comma  >
    <op   comma>
    <     op comma>

=for Marpa::R2::Display::End

and even


=for Marpa::R2::Display
ignore: 1

    <op
    comma>

=for Marpa::R2::Display::End

will all be treated as if they were a single symbol name,
one which is in effect equivalent to C<< <op comma> >>.

=head2 Types of rules

Rules are of four kinds:
the start rule,
empty rules,
quantified rules
and prioritized rules.
A simple rule,
with only a series of RHS symbols
on their RHS,
is treated as a
prioritized rule with only one alternative
and only one priority.
In a large grammar,
most rules are prioritized rules.

=head2 Start rule

Every source file should have a start rule.
The LHS of this rule is the C<:start> pseudo-symbol.
The RHS must be a single symbol name.
The start rule does not accept any adverbs.

=head2 Empty rules

An empty rule is a rule with an empty RHS.
A empty right hand side is what the name suggests,
no symbols, adverbs or punctuation.
A empty rule makes its LHS symbol a nullable symbol.
Empty rules may have an C<action> adverb.

=head2 Quantified rules

A quantified right hand side defines a sequence rule.
Its RHS consists of a single symbol name,
followed by a "quantifier".
The quantifer is either
a star ("C<*>"),
or a plus sign ("C<+>")
{'fraction_non_alphanumeric': 0.07695362923244961, 'fraction_numerical': 0.0074471254095918975, 'mean_word_length': 3.8122312470138557, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6763793', 'n_tokens_mistral': 3007, 'n_tokens_neox': 2884, 'n_words': 1435}