<commit_before>## $Id$

=head1 RAKUDO COMPILER OVERVIEW

=head2 How the Rakudo Perl 6 compiler works

This document describes the architecture and operation of the Rakudo
Perl 6 (or simply Rakudo) compiler.  The F<README> describes how to
build and run Rakudo.

Rakudo has six main parts (source code paths are relative to Rakudo's
F<src/> directory):

=over 4

=item 1.

Not Quite Perl builds Perl 6 source code parts into Rakudo

=item 2.

A main program drives parsing, code generation and runtime execution
(F<Perl6/Compiler.pir>)

=item 3.

A grammar parses user programs (F<Perl6/Grammar.pm>)

=item 4.

Action methods build a Parrot Abstract Syntax Tree (F<Perl6/Actions.pm>)

=item 5.

Parrot extensions provide Perl 6 run time behavior (TODO: describe)
(F<binder/*>, F<ops/*>, F<pmc/*>)

=item 6.

Libraries provide functions at run time (F<builtins/*.pir>, F<cheats/*>,
F<core/*.pm>, F<glue/*.pir>, F<metamodel/*>)

=back

The F<Makefile> (generated from F<build/Makefile.in> by F<Configure.pl>)
compiles all the parts to form the F<perl6.pbc> executable and the
F<perl6> or F<perl6.exe> "fake executable".  We call it fake because it
has only a small stub of code to launch the Parrot executable and pass
itself as a chunk of bytecode for Parrot to execute.

=head2 1. NQP[-RX]

The source files of Rakudo are preferably and increasingly written in
Perl 6, the remainder in Parrot Intermediate Representation (PIR) or C.
Not Quite Perl (nqp) provides the bootstrap step of compiling compiler
code (yes!) written in a subset of Perl 6, into PIR.

The latest version is called B<nqp-rx> because it now also includes a
powerful Perl 6 regex engine.  This has produced a streamlined compiler
framework on which to build a very functional Perl 6 implementation.

NQP itself is also written in PIR, is an important part of the Parrot
Compiler Toolkit (PCT), and is installed with Parrot.  PCT is a standard
framework to make and use Parrot based languages.  The source code of
NQP is in F<../parrot/ext/nqp-rx/> and the resulting (compiler-)
compiler is F<../parrot_install/bin/parrot-nqp>.  Note, NQP only
I<builds> the Rakudo compiler, and does not compile or run user
programs.

We can conceivably use the Rakudo compiler to compile itself to PIR and
eliminate the need for NQP entirely.  At some point as Rakudo matures we
will probably do this.  However, for the time being it's slightly easier
to manage the process if we keep a distinction between the two tools,
and using NQP for this stage also helps us to limit ourselves to using a
regular, well-defined, and relatively easy-to-implement subset of Perl 6
for the core compiler.  So, while it's possible for us to eliminate NQP
from the process, there are some good reasons not to do so just yet.
(If at some point we discover that we need something for the compiler
that NQP can't or won't support, then that will probably be a good point
to switch.)

=head2 2. Compiler main program

A subroutine called C<'main'>, in F<Perl6/Compiler.pir>, starts the
source parsing and bytecode generation work.  It creates a
C<Perl6::Compiler> object for the C<'perl6'> source type.  The
C<Perl6::Compiler> class inherits from the C<HLLCompiler> class of the
Parrot Compiler Toolkit, look in
F<../parrot/compilers/pct/src/PCT/HLLCompiler.pir>.

Before tracing Rakudo's execution further, a few words about Parrot
process and library initialization.

Parrot execution does not simply begin with 'main'.  When Parrot
executes a bytecode file, it first calls all subroutines in it that are
marked with the C<:init> modifier.  Rakudo has over 50 such subroutines,
brought in by C<.include> directives in F<Perl6/Compiler.pir>, to create
classes and objects in Parrot's memory.

Similarly, when the executable loads libraries, Parrot automatically
calls subs having the C<:load> modifier.  The Rakudo C<:init> subs are
usually also C<:load>, so that the same startup sequence occurs whether
Rakudo is run as an executable or loaded as a library.

F<Perl6/Compiler.pir> has three C<.loadlib> commands early on, for
C<perl6_group>, C<perl6_ops> and C<math_ops>.  All three dynamically
extend Parrot with respectively Rakudo specific PMC's (Poly Morphic
Containers, formerly Parrot Magic Cookies), opcodes, and mathematical
operators.  The source is in F<pmc/*>, F<ops/*> and
F<parrot/src/ops/math.ops>.

So, that Rakudo 'main' subroutine had created a C<Perl6::Compiler>
object.  Next, 'main' invokes the C<'command_line'> method on this
object, passing the command line arguments in a PMC called C<args_str>.
The C<'command_line'> method is inherited from the C<HLLCompiler> parent
class (part of the PCT, remember).

And that's it, apart from a C<'!fire_phasers'('END')> and an C<exit>.
Well, as far a C<'main'> is concerned.  The remaining work is divided
between PCT, grammar and actions.

=head2 2. Grammar

Using C<parrot-nqp>, C<make> target C<PERL6_G> uses F<parrot-nqp> to
compile F<Perl6/Grammar.pm> to F<gen/perl6-grammar.pir>.

The top-level portion of the grammar is written using Perl 6 rules
(Synopsis 5) and is based on the STD.pm grammar in the Pugs repository
(L<http://svn.pugscode.org/pugs/src/perl6/STD.pm>).  There are a few
places where Rakudo's grammar deviates from STD.pm, but the ultimate
goal is for the two to converge.  The grammar inherits from
C<HLL::Grammar>, which provides the C<< <.panic> >> rule to throw
exceptions for syntax errors.

The compiler works by calling C<TOP> method in F<Perl6/Grammar.pm>.
After some initialization, TOP matches the user program to the comp_unit
(meaning compilation unit) token.  That triggers a series of matches to
other tokens and rules (two kinds of regex) depending on the source in
the user program.

Lastly, the F<src/parser/quote_expression.pir> file implements
code to parse the various forms of Perl 6 quoting rules.  It's
far easier to write this component using PIR instead of a
regular expression, but otherwise it acts just like any other
rule in the grammar.

=head2 3. Actions

The C<Perl6/Actions.pm> file defines what the compiler must output when
it matches certain tokens or rules.  The output is a tree hierarchy of
objects representing language syntax elements, such as a statement.
The tree is called a Parrot Abstract Syntax Tree (PAST).

The PAST representation is the final stage of processing in Rakudo
itself. The PAST datastructure is then passed on to Parrot directly.
Parrot does the remainder of the work translating from PAST to pir and
then to bytecode.

--- ng update progress point

The action methods (in F<src/parser/actions.pm>) are used to convert the nodes
of the parse tree (produced by the parse grammar) into an equivalent Parrot
Abstract Syntax Tree (PAST) representation, which is then passed on to Parrot.

The action methods are where the Rakudo compiler does the bulk of the work of
creating an executable program.  Action methods are written in Perl 6, but we
use NQP to compile them into PIR as F<src/gen_actions.pir>.

When Rakudo is compiling a Perl 6 program, action methods are invoked
by the C< {*} > symbols in the parse grammar.  Each C< {*} > in a rule
causes the action method corresponding to the rule's name to be
invoked, passing the current match object as an argument.  If the
rule source line containing C< {*} > also contains a comment
starting with C< #= >, any text after the comment is passed as a
separate key argument to the action method.  (This is similar to
the approach that STD.pm uses to mark and distinguish actions.)

For example, here's the parse rule for Rakudo's C<unless> statement
(in src/parser/grammar.pg):

    rule unless_statement {
        $<sym>=[unless] <EXPR> <block><commit_msg>[docs/compiler_overview.pod] any excuse for a commit to the new master branch formerly known as ng<commit_after>## $Id$

=head1 RAKUDO COMPILER OVERVIEW

=head2 How the Rakudo Perl 6 compiler works

This document describes the architecture and operation of the Rakudo
Perl 6 (or simply Rakudo) compiler.  The F<README> describes how to
build and run Rakudo.

Rakudo has six main parts (source code paths are relative to Rakudo's
F<src/> directory):

=over 4

=item 1.

Not Quite Perl builds Perl 6 source code parts into Rakudo

=item 2.

A main program drives parsing, code generation and runtime execution
(F<Perl6/Compiler.pir>)

=item 3.

A grammar parses user programs (F<Perl6/Grammar.pm>)

=item 4.

Action methods build a Parrot Abstract Syntax Tree (F<Perl6/Actions.pm>)

=item 5.

Parrot extensions provide Perl 6 run time behavior (TODO: describe)
(F<binder/*>, F<ops/*>, F<pmc/*>)

=item 6.

Libraries provide functions at run time (F<builtins/*.pir>, F<cheats/*>,
F<core/*.pm>, F<glue/*.pir>, F<metamodel/*>)

=back

The F<Makefile> (generated from F<build/Makefile.in> by F<Configure.pl>)
compiles all the parts to form the F<perl6.pbc> executable and the
F<perl6> or F<perl6.exe> "fake executable".  We call it fake because it
has only a small stub of code to launch the Parrot executable and pass
itself as a chunk of bytecode for Parrot to execute.

=head2 1. NQP[-RX]

The source files of Rakudo are preferably and increasingly written in
Perl 6, the remainder in Parrot Intermediate Representation (PIR) or C.
Not Quite Perl (nqp) provides the bootstrap step of compiling compiler
code (yes!) written in a subset of Perl 6, into PIR.

The latest version is called B<nqp-rx> because it now also includes a
powerful Perl 6 regex engine.  This has produced a streamlined compiler
framework on which to build a very functional Perl 6 implementation.

NQP itself is also written in PIR, is an important part of the Parrot
Compiler Toolkit (PCT), and is installed with Parrot.  PCT is a standard
framework to make and use Parrot based languages.  The source code of
NQP is in F<../parrot/ext/nqp-rx/> and the resulting (compiler-)
compiler is F<../parrot_install/bin/parrot-nqp>.  Note, NQP only
I<builds> the Rakudo compiler, and does not compile or run user
programs.

We can conceivably use the Rakudo compiler to compile itself to PIR and
eliminate the need for NQP entirely.  At some point as Rakudo matures we
will probably do this.  However, for the time being it's slightly easier
to manage the process if we keep a distinction between the two tools,
and using NQP for this stage also helps us to limit ourselves to using a
regular, well-defined, and relatively easy-to-implement subset of Perl 6
for the core compiler.  So, while it's possible for us to eliminate NQP
from the process, there are some good reasons not to do so just yet.
(If at some point we discover that we need something for the compiler
that NQP can't or won't support, then that will probably be a good point
to switch.)

=head2 2. Compiler main program

A subroutine called C<'main'>, in F<Perl6/Compiler.pir>, starts the
source parsing and bytecode generation work.  It creates a
C<Perl6::Compiler> object for the C<'perl6'> source type.  The
C<Perl6::Compiler> class inherits from the C<HLLCompiler> class of the
Parrot Compiler Toolkit, look in
F<../parrot/compilers/pct/src/PCT/HLLCompiler.pir>.

Before tracing Rakudo's execution further, a few words about Parrot
process and library initialization.

Parrot execution does not simply begin with 'main'.  When Parrot
executes a bytecode file, it first calls all subroutines in it that are
marked with the C<:init> modifier.  Rakudo has over 50 such subroutines,
brought in by C<.include> directives in F<Perl6/Compiler.pir>, to create
classes and objects in Parrot's memory.

Similarly, when the executable loads libraries, Parrot automatically
calls subs having the C<:load> modifier.  The Rakudo C<:init> subs are
usually also C<:load>, so that the same startup sequence occurs whether
Rakudo is run as an executable or loaded as a library.

F<Perl6/Compiler.pir> has three C<.loadlib> commands early on, for
C<perl6_group>, C<perl6_ops> and C<math_ops>.  All three dynamically
extend Parrot with respectively Rakudo specific PMC's (Poly Morphic
Containers, formerly Parrot Magic Cookies), opcodes, and mathematical
operators.  The source is in F<pmc/*>, F<ops/*> and
F<parrot/src/ops/math.ops>.

So, that Rakudo 'main' subroutine had created a C<Perl6::Compiler>
object.  Next, 'main' invokes the C<'command_line'> method on this
object, passing the command line arguments in a PMC called C<args_str>.
The C<'command_line'> method is inherited from the C<HLLCompiler> parent
class (part of the PCT, remember).

And that's it, apart from a C<'!fire_phasers'('END')> and an C<exit>.
Well, as far a C<'main'> is concerned.  The remaining work is divided
between PCT, grammar and actions.

=head2 2. Grammar

Using C<parrot-nqp>, C<make> target C<PERL6_G> uses F<parrot-nqp> to
compile F<Perl6/Grammar.pm> to F<gen/perl6-grammar.pir>.

The top-level portion of the grammar is written using Perl 6 rules
(Synopsis 5) and is based on the STD.pm grammar in the Pugs repository
(L<http://svn.pugscode.org/pugs/src/perl6/STD.pm>).  There are a few
places where Rakudo's grammar deviates from STD.pm, but the ultimate
goal is for the two to converge.  The grammar inherits from
C<HLL::Grammar>, which provides the C<< <.panic> >> rule to throw
exceptions for syntax errors.

The compiler works by calling C<TOP> method in F<Perl6/Grammar.pm>.
After some initialization, TOP matches the user program to the comp_unit
(meaning compilation unit) token.  That triggers a series of matches to
other tokens and rules (two kinds of regex) depending on the source in
the user program.

=head2 3. Actions

The C<Perl6/Actions.pm> file defines what the compiler must output when
it matches certain tokens or rules.  The output is a tree hierarchy of
objects representing language syntax elements, such as a statement.
The tree is called a Parrot Abstract Syntax Tree (PAST).

The PAST representation is the final stage of processing in Rakudo
itself. The PAST datastructure is then passed on to Parrot directly.
Parrot does the remainder of the work translating from PAST to pir and
then to bytecode.

--- ng update progress point

Lastly, the F<src/parser/quote_expression.pir> file implements
code to parse the various forms of Perl 6 quoting rules.  It's
far easier to write this component using PIR instead of a
regular expression, but otherwise it acts just like any other
rule in the grammar.

The action methods (in F<src/parser/actions.pm>) are used to convert the nodes
of the parse tree (produced by the parse grammar) into an equivalent Parrot
Abstract Syntax Tree (PAST) representation, which is then passed on to Parrot.

The action methods are where the Rakudo compiler does the bulk of the work of
creating an executable program.  Action methods are written in Perl 6, but we
use NQP to compile them into PIR as F<src/gen_actions.pir>.

When Rakudo is compiling a Perl 6 program, action methods are invoked
by the C< {*} > symbols in the parse grammar.  Each C< {*} > in a rule
causes the action method corresponding to the rule's name to be
invoked, passing the current match object as an argument.  If the
rule source line containing C< {*} > also contains a comment
starting with C< #= >, any text after the comment is passed as a
separate key argument to the action method.  (This is similar to
the approach that STD.pm uses to mark and distinguish actions.)

For example, here's the parse rule for Rakudo's C<unless> statement
(in src/parser/grammar.pg):

    rule unless_statement {
        $<sym>=[unless] <EXPR> <block>
{'fraction_non_alphanumeric': 0.07011189444408512, 'fraction_numerical': 0.006985317896643166, 'mean_word_length': 4.415761821366025, 'pattern_counts': {'":': 0, '<': 157, '<?xml version=': 0, '>': 157, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4481082', 'n_tokens_mistral': 4781, 'n_tokens_neox': 4518, 'n_words': 2319}