<commit_before># Copyright (C) 2008, The Perl Foundation.
# $Id$

=head1 NAME

docs/pdds/draft/pdd29_compiler_tools.pod - Parrot Compiler Tools

=head1 VERSION

$Revision$

=head1 MAINTAINER

<NAME>
Klaas-<NAME>

=head1 ABSTRACT

This PDD specifies the Parrot Compiler Tools (PCT).

=head1 SYNOPSIS

Creating a PCT-based compiler can be done as follows:

 .sub 'onload' :anon :load :init
     load_bytecode 'PCT.pbc'
     $P0 = get_hll_global ['PCT'], 'HLLCompiler'
     $P1 = $P0.'new'()
     $P1.'language'('Foo')
     $P1.'parsegrammar'('Foo::Grammar')
     $P1.'parseactions'('Foo::Grammar::Actions')
 .end

 .sub 'main' :main
     .param pmc args
     $P0 = compreg 'Foo'
     $P1 = $P0.'command_line'(args)
 .end

{{ this is the most important part; is this enough? }}

The Parrot distribution contains a Perl script to generate a compiler
stub, containing all necessary files. This generated compiler will
compile out of the box. It is highly suggested to use this script to
get started with the PCT.
The script is located in C<tools/dev/mk_language_shell.pl>.

{{ Not sure whether the mk_language_shell.pl script should be mentioned long
term }}


Running the compiler is then done as follows:

 $ parrot foo.pbc [--target=[parse|past|post|pir]] <file>

{{ other options? }}

=head1 DESCRIPTION

The Parrot Compiler Tools are specially designed to easily create a
compiler targeting the Parrot Virtual Machine. The tools themselves
run on Parrot, which implies that no other external programs are
needed.

The PCT is a set of libraries and programs, to:

=over 4

=item * create a parser

=item * create an intermediate data structure (Abstract Syntax Tree)

=item * generate executable code (Parrot Intermediate Representation)

=back



=head1 IMPLEMENTATION

The PCT is made up of the following libraries and programs:

=over 4

=item * Parrot Grammar Engine (PGE)

=item * Parrot Abstract Syntax Tree (PAST) classes

=item * Parrot Opcode Syntax Tree (POST) classes

=item * PCT::HLLCompiler class

=item * PCT::Grammar class

=back

Although strictly speaking it is not part of the PCT, the
Not Quite Perl (6) (NQP) language is typically used in all PCT-based
compilers. NQP is a subset of the Perl 6 language, and is a
high-level language as an alternative for PIR.

=head2 Compilation phases

A PCT-based compiler has by default four compilation phases, or
I<transformations>. Phases can be removed and added through the API of
the C<HLLCompiler> class. These are:

=over 4

=item * source to parse tree

The source is read, parsed and stored in a parse tree.

=item * parse tree to PAST

The parse tree is converted into a Parrot Abstract Syntax Tree.

=item * PAST to POST

The PAST is converted into a Parrot Opcode Syntax Tree.

=item * POST to PIR

The POST is converted into executable Parrot Intermediate Representation.

=back

=head3 Source to Parse Tree

{{ XXX }}

=head3 Parse tree to PAST

{{ XXX }}

=head3 PAST to POST

{{ XXX }}

=head3 POST to PIR

{{ XXX }}

=head2 Parrot Grammar Engine

The Parrot Grammar Engine (PGE) is a component that I<executes> regular
expressions. Besides I<classic> regular expressions, it also understands
Perl 6 Rules. Such rules are special regular expressions to define a grammar.

=head2 PCT::Grammar

The class C<PCT::Grammar> is a built-in grammar class that can be used as
a parent class for a custom grammar. This class defines a number of rules and
tokens that are inherited by child classes. Note that the concept of C<class>
and C<grammar> are equivalent.

The following rules are predefined:

{{ is this necessary, or just a reference to the file? }}

=over 4

=item ident

=item ws

=back

=head2 PCT::HLLCompiler

All PCT-based compilers use a HLLCompiler object as a compiler driver.
This object invokes the different compiler phases.

=head3 HLLCompiler API Methods

{{ TODO: complete this }}

=over 4

=item commandline_prompt

$P0.'commandline_prompt'($S0)
<commit_msg>[pdd29] add more stuff to pdd29<commit_after># Copyright (C) 2008, The Perl Foundation.
# $Id$

=head1 NAME

docs/pdds/draft/pdd29_compiler_tools.pod - Parrot Compiler Tools

=head1 VERSION

$Revision$

=head1 MAINTAINER

<NAME>
<NAME>

=head1 DEFINITIONS

=head2 Compiler

In this document, when we speak of a I<compiler>, we mean
PCT-based compilers.

=head2 HLL

A High-Level Language. Examples are: Perl, Ruby, Python, Lua, Tcl, etc.

=head1 ABSTRACT

This PDD specifies the Parrot Compiler Tools (PCT).

=head1 SYNOPSIS

Creating a PCT-based compiler can be done as follows:

 .sub 'onload' :anon :load :init
     load_bytecode 'PCT.pbc'
     $P0 = get_hll_global ['PCT'], 'HLLCompiler'
     $P1 = $P0.'new'()
     $P1.'language'('Foo')
     $P1.'parsegrammar'('Foo::Grammar')
     $P1.'parseactions'('Foo::Grammar::Actions')
 .end

 .sub 'main' :main
     .param pmc args
     $P0 = compreg 'Foo'
     $P1 = $P0.'command_line'(args)
 .end

{{ this is the most important part; is this enough? }}

The Parrot distribution contains a Perl script to generate a compiler
stub, containing all necessary files. This generated compiler will
compile out of the box. It is highly suggested to use this script to
get started with the PCT.
The script is located in C<tools/dev/mk_language_shell.pl>.

{{ Not sure whether the mk_language_shell.pl script should be mentioned long term
   In a sense, this script can also be considered part of the parrot compiler "tools",
   as it is used to create a compiler.
}}

=head2 Parser Synopsis

 grammar Foo is PCT::Grammar;

 rule TOP {
     <statement>*
     {*}
 }

 rule statement {
     <ident> '=' <expression>
     {*}
 }

 rule expression is optable { ... }

 proto infix:<+> is precedence('1') is pirop('n_add') { ... }

 rule 'term:' is tighter(infix:<+>) is parsed(&term) { ... }

 rule term {
     | <ident> {*}       #= ident
     | <number> {*}      #= number
 }

=head2 Actions Synopsis

{{ Is this a good idea? }}

class Foo::Grammar::Actions;

 method TOP($/) {
     my $past := PAST::Block.new( :blocktype('declaration') );
     for $<statement> {
         $past.push( $( $_ ) );
     }
     make $past;
 }

 method statement($/) {
     ...
 }

 method expression($/, $key) {
     ...
 }

 method term($/, $key) {
     make $( $/{$key} );
 }

=head2 Running the compiler

Running the compiler is then done as follows:

 $ parrot foo.pbc [--target=[parse|past|post|pir]] <file>

{{ other options? Maybe --target=pbc in the future, once PBC can be
   generated? }}

=head1 DESCRIPTION

The Parrot Compiler Tools are specially designed to easily create a
compiler targeting the Parrot Virtual Machine. The tools themselves
run on Parrot, which implies that no other external programs are
needed.

The PCT is a set of libraries and programs, to:

=over 4

=item * create a parser

=item * create an intermediate data structure (Abstract Syntax Tree)

=item * generate executable Parrot code

=back



=head1 IMPLEMENTATION

The PCT is made up of the following libraries and programs:

=over 4

=item * Parrot Grammar Engine (PGE)

=item * Parrot Abstract Syntax Tree (PAST) classes

=item * Parrot Opcode Syntax Tree (POST) classes

=item * PCT::HLLCompiler class

=item * PCT::Grammar class

=back

Although strictly speaking it is not part of the PCT, the
Not Quite Perl (6) (NQP) language is typically used in all PCT-based
compilers. NQP is a subset of the Perl 6 language, and is a
high-level language as an alternative for PIR.

=head2 Compilation phases

A PCT-based compiler has by default four compilation phases, or
I<transformations>. Phases can be removed and added through the API of
the C<HLLCompiler> class. These are:

=over 4

=item * source to parse tree

The source is read, parsed and stored in a parse tree.

=item * parse tree to PAST

The parse tree is converted into a Parrot Abstract Syntax Tree.

=item * PAST to POST

The PAST is converted into a Parrot Opcode Syntax Tree.

=item * POST to PIR

The POST is converted into executable Parrot Intermediate Representation.

=back

=head3 Source to Parse Tree

The first stage of a PCT-based compiler is done by the C<parser>. The
parser is defined as a set of Perl 6 Rules, which is processed by the
Perl 6 Rules compiler. This results in a generated PIR file that
implements the parser.

{{ Doesn't this make the Perl 6 rules compiler part of the PCT? }}

During the first stage, the source (input string) is parsed, resulting
in a C<parse tree>.

=head3 Parse tree to PAST

The second stage converts the parse tree into a Parrot Abstract Syntax Tree
(PAST). PAST is a data structure consisting of PAST nodes, each of which
represents a common HLL construct. While all languages differ in syntax,
many constructs in different HLLs map to the same semantics. This second
transformation is executed during the parse stage. The transformations
of the parse tree nodes into PAST nodes is done by so-called parse actions,
which are methods of a class that is specified through the C<parseactions>
attribute of the HLLCompiler. Such classes are implemented in NQP.

{{ How do we say that this is not obligatory; you could also use PIR,
   and in the future maybe other languages.
}}

=head3 PAST to POST

The third transformation converts the PAST into a Parrot Opcode Syntax Tree
(POST). PAST nodes represent HLL constructs, which are transformed into a
set of low-level POST nodes. A POST node is a low-level node, representing
a single instruction, label, or a subroutine. While a PAST is very close to
a HLL program, a POST is much closer to PIR code.

=head3 POST to PIR

The last transformation generates PIR code from the POST.

The generated PIR is then fed into the Parrot executable, and processed
into Parrot Byte Code (PBC) by the PIR compiler.

=head2 Parrot Grammar Engine

The Parrot Grammar Engine (PGE) is a component that I<executes> regular
expressions. Besides I<classic> regular expressions, it also understands
Perl 6 Rules. Such rules are special regular expressions to define a grammar.

=head2 Parrot Abstract Syntax Tree

The PCT includes a set of PAST classes. PAST classes represent common language
constructs, such as a C<while statement>.
These are described extensively in L<docs/pdds/pdd26_ast.pod>.

=head2 Parrot Opcode Syntax Tree

=head3 POST::Node

POST::Node is the base class for all other POST classes.

=head3 POST::Op

=head3 POST::Ops

=head3 POST::Label

=head3 POST::Sub



=head2 PCT::Grammar

The class C<PCT::Grammar> is a built-in grammar class that can be used as
a parent class for a custom grammar. This class defines a number of rules and
tokens that are inherited by child classes. Note that the concept of C<class>
and C<grammar> are equivalent.

The following rules are predefined:

{{ is this necessary, or just a reference to the file? }}

=over 4

=item ident

=item ws

=back

=head2 PCT::HLLCompiler

All PCT-based compilers use a HLLCompiler object as a compiler driver.
It acts as a I<facade> for the compiler. This object invokes the different
compiler phases.

=head3 HLLCompiler API Methods

{{ TODO: complete this }}

=over 4

=item commandline_prompt

$P0.'commandline_prompt'($S0)

{'fraction_non_alphanumeric': 0.07897814158495997, 'fraction_numerical': 0.009444994153098858, 'mean_word_length': 3.917293233082707, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7646359', 'n_tokens_mistral': 3545, 'n_tokens_neox': 3350, 'n_words': 1574}