<commit_before># Copyright 2014 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::Scanless::DSL - The DSL for the Scanless interface

=head1 Synopsis

=for Marpa::R2::Display
name: SLIF DSL synopsis
partial: 1
normalize-whitespace: 1

    use Marpa::R2;

    my $grammar = Marpa::R2::Scanless::G->new(
        {   bless_package => 'My_Nodes',
            source        => \(<<'END_OF_SOURCE'),
    :default ::= action => [values] bless => ::lhs
    lexeme default = action => [start,length,value] bless => ::name

    :start ::= Script
    Script ::= Expression+ separator => comma
    comma ~ [,]
    Expression ::=
        Number bless => primary
        | '(' Expression ')' bless => paren assoc => group
       || Expression '**' Expression bless => exponentiate assoc => right
       || Expression '*' Expression bless => multiply
        | Expression '/' Expression bless => divide
       || Expression '+' Expression bless => add
        | Expression '-' Expression bless => subtract

    Number ~ [\d]+
    :discard ~ whitespace
    whitespace ~ [\s]+
    # allow comments
    :discard ~ <hash comment>
    <hash comment> ~ <terminated hash comment> | <unterminated
       final hash comment>
    <terminated hash comment> ~ '#' <hash comment body> <vertical space char>
    <unterminated final hash comment> ~ '#' <hash comment body>
    <hash comment body> ~ <hash comment char>*
    <vertical space char> ~ [\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    <hash comment char> ~ [^\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 About this document

This is the reference document for
the domain-specific language (DSL)
of Marpa's Scanless interface (SLIF).
The SLIF's DSL is
an extension of BNF.
The SLIF DSL is used to specify other
DSL's, and is therefore a "meta-DSL".

=head1 The structure of SLIF source strings

The SLIF source string consists of a series of rules,
pseudo-rules and statements.
These are made up of tokens,
as described below.
Whitespace separates tokens, but is otherwise ignored.

A hash ("C<#>") character starts a comment,
which continues to the end of the line.
Comments are equivalent to whitespace.

=head1 L0, G1 and lexemes

In reading this document, it is important to keep in mind
the distinction, on one hand, between L0 and G1 rules and,
on the other hand, between rules and lexemes.
G1 rules have a semantics,
which can be specified as described
in this document.
L0 rules simply recognize symbols in the input.
L0 rules do not have a semantics.

Top-level L0 rules correspond to a string in the input.
The top-level L0 rules are seen by G1 as lexemes,
and the string to which a top-level L0 rule
corresponds becomes the default value of the lexeme.
The L0 grammar can be thought of as similar
in behavior to a set of regular expressions
with the lexemes being seen as similar to named captures.

Lexemes are the symbols which form the interface between
G1 and L0.  Lexemes, like G1 rules, have a semantics.
The semantics of lexemes is
specified separately from the
semantics of G1 rules,
as described below.

=head1 Symbol names

Symbol names can be either "bare" or enclosed in angle brackets.
Bare symbol names must consist entirely of Perl word characters
(alphanumerics, plus the underscore).
Symbol names are case-sensitive.

The angle brackets, if used, serve to "quote"
the symbol name,
and will not be part of the explicit symbol name.

If angle brackets are used, symbol names may also contain whitespace,
as in

=for Marpa::R2::Display
ignore: 1

    <op comma>

=for Marpa::R2::Display::End

A whitespace sequence inside angle brackets can
include any whitespace character that is legal in Perl,
including newlines.
This allows very long symbol names to be line wrapped,
if necessary.

Unlike the angle brackets,
the whitespace in a bracketed symbol token
B<does> become part of
of the explicit symbol name,
but it does so in a "normalized" form.
Leading and trailing whitespace in the name is discarded,
and all other whitespace sequences are converted to a single ASCII
space character.
This means that

=for Marpa::R2::Display
ignore: 1

    < op comma  >
    <op   comma>
    <     op comma>

=for Marpa::R2::Display::End

and even


=for Marpa::R2::Display
ignore: 1

    <op
    comma>

=for Marpa::R2::Display::End

will all be regarded as the same symbol name.
The explicit form of that symbol name
is C<< <op comma> >>, except that, again, the
angle brackets are for clarity,
and are not part of the explicity name.<commit_msg>Fixed a typo.<commit_after># Copyright 2014 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::Scanless::DSL - The DSL for the Scanless interface

=head1 Synopsis

=for Marpa::R2::Display
name: SLIF DSL synopsis
partial: 1
normalize-whitespace: 1

    use Marpa::R2;

    my $grammar = Marpa::R2::Scanless::G->new(
        {   bless_package => 'My_Nodes',
            source        => \(<<'END_OF_SOURCE'),
    :default ::= action => [values] bless => ::lhs
    lexeme default = action => [start,length,value] bless => ::name

    :start ::= Script
    Script ::= Expression+ separator => comma
    comma ~ [,]
    Expression ::=
        Number bless => primary
        | '(' Expression ')' bless => paren assoc => group
       || Expression '**' Expression bless => exponentiate assoc => right
       || Expression '*' Expression bless => multiply
        | Expression '/' Expression bless => divide
       || Expression '+' Expression bless => add
        | Expression '-' Expression bless => subtract

    Number ~ [\d]+
    :discard ~ whitespace
    whitespace ~ [\s]+
    # allow comments
    :discard ~ <hash comment>
    <hash comment> ~ <terminated hash comment> | <unterminated
       final hash comment>
    <terminated hash comment> ~ '#' <hash comment body> <vertical space char>
    <unterminated final hash comment> ~ '#' <hash comment body>
    <hash comment body> ~ <hash comment char>*
    <vertical space char> ~ [\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    <hash comment char> ~ [^\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 About this document

This is the reference document for
the domain-specific language (DSL)
of Marpa's Scanless interface (SLIF).
The SLIF's DSL is
an extension of BNF.
The SLIF DSL is used to specify other
DSL's, and is therefore a "meta-DSL".

=head1 The structure of SLIF source strings

The SLIF source string consists of a series of rules,
pseudo-rules and statements.
These are made up of tokens,
as described below.
Whitespace separates tokens, but is otherwise ignored.

A hash ("C<#>") character starts a comment,
which continues to the end of the line.
Comments are equivalent to whitespace.

=head1 L0, G1 and lexemes

In reading this document, it is important to keep in mind
the distinction, on one hand, between L0 and G1 rules and,
on the other hand, between rules and lexemes.
G1 rules have a semantics,
which can be specified as described
in this document.
L0 rules simply recognize symbols in the input.
L0 rules do not have a semantics.

Top-level L0 rules correspond to a string in the input.
The top-level L0 rules are seen by G1 as lexemes,
and the string to which a top-level L0 rule
corresponds becomes the default value of the lexeme.
The L0 grammar can be thought of as similar
in behavior to a set of regular expressions
with the lexemes being seen as similar to named captures.

Lexemes are the symbols which form the interface between
G1 and L0.  Lexemes, like G1 rules, have a semantics.
The semantics of lexemes is
specified separately from the
semantics of G1 rules,
as described below.

=head1 Symbol names

Symbol names can be either "bare" or enclosed in angle brackets.
Bare symbol names must consist entirely of Perl word characters
(alphanumerics, plus the underscore).
Symbol names are case-sensitive.

The angle brackets, if used, serve to "quote"
the symbol name,
and will not be part of the explicit symbol name.

If angle brackets are used, symbol names may also contain whitespace,
as in

=for Marpa::R2::Display
ignore: 1

    <op comma>

=for Marpa::R2::Display::End

A whitespace sequence inside angle brackets can
include any whitespace character that is legal in Perl,
including newlines.
This allows very long symbol names to be line wrapped,
if necessary.

Unlike the angle brackets,
the whitespace in a bracketed symbol token
B<does> become part of
of the explicit symbol name,
but it does so in a "normalized" form.
Leading and trailing whitespace in the name is discarded,
and all other whitespace sequences are converted to a single ASCII
space character.
This means that

=for Marpa::R2::Display
ignore: 1

    < op comma  >
    <op   comma>
    <     op comma>

=for Marpa::R2::Display::End

and even


=for Marpa::R2::Display
ignore: 1

    <op
    comma>

=for Marpa::R2::Display::End

will all be regarded as the same symbol name.
The explicit form of that symbol name
is C<< <op comma> >>, except that, again, the
angle brackets are for clarity,
and are not part of the explicit name.
{'fraction_non_alphanumeric': 0.08506761292797545, 'fraction_numerical': 0.012084012659441834, 'mean_word_length': 3.6972972972972973, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6782304', 'n_tokens_mistral': 3162, 'n_tokens_neox': 2991, 'n_words': 1435}