<commit_before># Copyright 2012 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::Scanless::G - Scanless interface grammars

=head1 This document is UNDER CONSTRUCTION

=head1 Synopsis

=for Marpa::R2::Display
name: Scanless grammar synopsis
partial: 1
normalize-whitespace: 1

    my $grammar = Marpa::R2::Scanless::G->new(
        {   
            action_object  => 'My_Actions',
            default_action => 'do_first_arg',
            source          => \(<<'END_OF_SOURCE'),
    :start ::= Script
    Script ::= Expression+ separator => comma action => do_script
    comma ~ [,]
    Expression ::=
        Number
        | '(' Expression ')' action => do_parens assoc => group
       || Expression '**' Expression action => do_pow assoc => right
       || Expression '*' Expression action => do_multiply
        | Expression '/' Expression action => do_divide
       || Expression '+' Expression action => do_add
        | Expression '-' Expression action => do_subtract
    Number ~ [\d]+

    :discard ~ whitespace
    whitespace ~ [\s]+
    # allow comments
    :discard ~ <hash comment>
    <hash comment> ~ <terminated hash comment> | <unterminated
       final hash comment>
    <terminated hash comment> ~ '#' <hash comment body> <vertical space char>
    <unterminated final hash comment> ~ '#' <hash comment body>
    <hash comment body> ~ <hash comment char>*
    <vertical space char> ~ [\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    <hash comment char> ~ [^\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 About this document

This page is the reference for the grammar objects
of Marpa's Scanless interface.

=head1 Methods

=head2 new()

=head2 show_rules()

=head1 Source files

The format of
Scanless source files is based
on L<the format of
BNF source files|Marpa::R2::BNF>,
but with the following changes
and extensions:

=head2 Structural rules

The BNF operator ("C<::=>")
has the same function as before,
but in addition it indicates that the rule is
a G1 rule C<::=>.
Start rules must be G1 rules.

=head2 Lexical rules

The match operator ("C<~>")
can be used between the LHS and RHS of a rule,
instead of the BNF operator.
Rules which use the match operator are G0 rules.
Discard rules must be G0 rules.
A lexical rule cannot have an C<action> adverb.

=head2 Discard rules

A discard rule is a rule whose LHS is
the C<:discard> pseudo-symbol,
and whose RHS is a single symbol name,
called the B<discarded symbol>.
These rules indicate that the discarded symbol is a top-level G0
symbol, but one which is not a lexeme.
When a discarded symbol is recognized,
it is not passed as a lexeme to the G1 parser, but is
(as the name suggests) discarded.

=head2 Single quoted strings

Single quotes can be used in prioritized
rules to indicate character strings.
The characters inside the single quote
will be matched in the input, literally and one-for-one.
Single quoted strings can contain any character except
a single quote and vertical whitespace.

Single quotes does not have "escaped" characters.
A backslash ("C<\>") represents itself and has no effect
on the interpretation of the next character.
If a rule needs to match one of the forbidden characters
(single quote or vertical whitespace), it must use a
character class.

Single quoted strings are always interpreted at the
G0 level, but they may be used in either structural
or lexical rules.
When a single quoted string is used in a structural rule,
Marpa creates a virtual G0 rule on behalf of the application.
This is handy, but it does have a real disadvantage --
the name of the virtual rule's LHS
will be one assigned automatically by Marpa,
When tracing and debugging parses and grammar,
these virtual LHS's can be harder for a programmer
to interpret.

=head2 Character classes

A character class in square brackets ("C<[]>")
can be used in a RHS alternative of a prioritized rule,
or on the RHS of a quantified rule or a discard rule.
Marpa character classes may contain anything acceptable to Perl,
and follow the same escaping conventions as Perl's character classes.

Character classes are always interpreted at the
G0 level, but they may be used in either structural
or lexical rules.
When a character class is used in a structural rule,
Marpa creates a virtual G0 rule on behalf of the application.
This is handy, but it does have a real disadvantage --
the name of the virtual rule's LHS
will be one assigned automatically by Marpa,
When tracing and debugging parses and grammar,
these virtual LHS's can be harder for a programmer
to interpret.

An implementation note: character classes are
interpreted by Perl, but this involves minimal overhead
when the parse is of any length.
Each character class is
passed to Perl to interpret exactly once and the result is
memoized in a C structure for future use.

=head1 Copyright and License

=for Marpa::R2::Display
ignore: 1

  Copyright 2012 <NAME>
  This file is part of Marpa::R2.  Marpa::R2 is free software: you can
  redistribute it and/or modify it under the terms of the GNU Lesser
  General Public License as published by the Free Software Foundation,
  either version 3 of the License, or (at your option) any later version.

  Marpa::R2 is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser
  General Public License along with Marpa::R2.  If not, see
  http://www.gnu.org/licenses/.

=for Marpa::R2::Display::End

=cut

# Local Variables:
#   mode: cperl
#   cperl-indent-level: 4
#   fill-column: 100
# End:
# vim: expandtab shiftwidth=4:

<commit_msg>Documenting scanless interface<commit_after># Copyright 2012 <NAME>
# This file is part of Marpa::R2.  Marpa::R2 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R2 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R2.  If not, see
# http://www.gnu.org/licenses/.

=head1 Name

Marpa::R2::Scanless::G - Scanless interface grammars

=head1 This document is UNDER CONSTRUCTION

=head1 Synopsis

=for Marpa::R2::Display
name: Scanless grammar synopsis
partial: 1
normalize-whitespace: 1

    my $grammar = Marpa::R2::Scanless::G->new(
        {   
            action_object  => 'My_Actions',
            default_action => 'do_first_arg',
            source          => \(<<'END_OF_SOURCE'),
    :start ::= Script
    Script ::= Expression+ separator => comma action => do_script
    comma ~ [,]
    Expression ::=
        Number
        | '(' Expression ')' action => do_parens assoc => group
       || Expression '**' Expression action => do_pow assoc => right
       || Expression '*' Expression action => do_multiply
        | Expression '/' Expression action => do_divide
       || Expression '+' Expression action => do_add
        | Expression '-' Expression action => do_subtract
    Number ~ [\d]+

    :discard ~ whitespace
    whitespace ~ [\s]+
    # allow comments
    :discard ~ <hash comment>
    <hash comment> ~ <terminated hash comment> | <unterminated
       final hash comment>
    <terminated hash comment> ~ '#' <hash comment body> <vertical space char>
    <unterminated final hash comment> ~ '#' <hash comment body>
    <hash comment body> ~ <hash comment char>*
    <vertical space char> ~ [\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    <hash comment char> ~ [^\x{A}\x{B}\x{C}\x{D}\x{2028}\x{2029}]
    END_OF_SOURCE
        }
    );

=for Marpa::R2::Display::End

=head1 About this document

This page is the reference for the grammar objects
of Marpa's Scanless interface.

=head1 Constructor

The C<new()> is the constructor for Scanless grammars.
An example of its use is L<above|Synopsis>.
The C<new()> constructor accepts a hash of named arguments.
Allowed named arguments are the following:

=head2 action_object

=head2 default_action

=head2 source

The value of the C<source> named argument must be a reference
to a string which contains a description of the grammar.
The string's format is described L<below|/"Source strings">.

=head2 trace_file_handle

=head1 Trace method

=head2 show_rules()

=head1 Source strings

The format of
Scanless source files is based
on L<the format of
BNF source files|Marpa::R2::BNF>,
but with the following changes
and extensions:

=head2 Structural rules

The BNF operator ("C<::=>")
has the same function as before,
but in addition it indicates that the rule is
a G1 rule C<::=>.
Start rules must be G1 rules.

=head2 Lexical rules

The match operator ("C<~>")
can be used between the LHS and RHS of a rule,
instead of the BNF operator.
Rules which use the match operator are G0 rules.
Discard rules must be G0 rules.
A lexical rule cannot have an C<action> adverb.

=head2 Discard rules

A discard rule is a rule whose LHS is
the C<:discard> pseudo-symbol,
and whose RHS is a single symbol name,
called the B<discarded symbol>.
These rules indicate that the discarded symbol is a top-level G0
symbol, but one which is not a lexeme.
When a discarded symbol is recognized,
it is not passed as a lexeme to the G1 parser, but is
(as the name suggests) discarded.

=head2 Single quoted strings

Single quotes can be used in prioritized
rules to indicate character strings.
The characters inside the single quote
will be matched in the input, literally and one-for-one.
Single quoted strings can contain any character except
a single quote and vertical whitespace.

Single quotes does not have "escaped" characters.
A backslash ("C<\>") represents itself and has no effect
on the interpretation of the next character.
If a rule needs to match one of the forbidden characters
(single quote or vertical whitespace), it must use a
character class.

Single quoted strings are always interpreted at the
G0 level, but they may be used in either structural
or lexical rules.
When a single quoted string is used in a structural rule,
Marpa creates a virtual G0 rule on behalf of the application.
This is handy, but it does have a real disadvantage --
the name of the virtual rule's LHS
will be one assigned automatically by Marpa,
When tracing and debugging parses and grammar,
these virtual LHS's can be harder for a programmer
to interpret.

=head2 Character classes

A character class in square brackets ("C<[]>")
can be used in a RHS alternative of a prioritized rule,
or on the RHS of a quantified rule or a discard rule.
Marpa character classes may contain anything acceptable to Perl,
and follow the same escaping conventions as Perl's character classes.

Character classes are always interpreted at the
G0 level, but they may be used in either structural
or lexical rules.
When a character class is used in a structural rule,
Marpa creates a virtual G0 rule on behalf of the application.
This is handy, but it does have a real disadvantage --
the name of the virtual rule's LHS
will be one assigned automatically by Marpa,
When tracing and debugging parses and grammar,
these virtual LHS's can be harder for a programmer
to interpret.

An implementation note: character classes are
interpreted by Perl, but this involves minimal overhead
when the parse is of any length.
Each character class is
passed to Perl to interpret exactly once and the result is
memoized in a C structure for future use.

=head1 Copyright and License

=for Marpa::R2::Display
ignore: 1

  Copyright 2012 <NAME>
  This file is part of Marpa::R2.  Marpa::R2 is free software: you can
  redistribute it and/or modify it under the terms of the GNU Lesser
  General Public License as published by the Free Software Foundation,
  either version 3 of the License, or (at your option) any later version.

  Marpa::R2 is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser
  General Public License along with Marpa::R2.  If not, see
  http://www.gnu.org/licenses/.

=for Marpa::R2::Display::End

=cut

# Local Variables:
#   mode: cperl
#   cperl-indent-level: 4
#   fill-column: 100
# End:
# vim: expandtab shiftwidth=4:


{'fraction_non_alphanumeric': 0.07224787725309102, 'fraction_numerical': 0.011172352152539849, 'mean_word_length': 3.8613323678493847, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 84, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4700503', 'n_tokens_mistral': 3860, 'n_tokens_neox': 3698, 'n_words': 1899}