<commit_before>=head1 S11-Modules proposal for Rakudo * implementation

The aim will be to implement it all in NQP if possible.

=head1 Overriding Principle

The source code must always be the absolute source of all information.
Everything else should act as a cache.  That means *.pir files and
databases of metadata may be automatically or manually derived from the
*.pm files, but they may not add information (from the command line, for
example).

If there is to be cached metadata in future, it should be stored in
files as close to the corresponding source files as possible.
If modules are precompiled to *.pir files, those files will be stored
in the same directories as their corresponding *.pm source files, with
identical names apart from the extension.

=head1 Restrictions and limitations

=head2 No C<package> keyword

Classes may contain other classes, which provides sufficient hierarchy.

Rakudo will implement C<module>, in order to contain sub definitions
that do not belong to any class.

=head2 Only simplest :ver and :auth implementation

There should be only one C<:ver> and C<:auth> name part per source code
file, in order to keep the implementation simple.  In order to keep
users sane, multiple C<:ver> or C<:auth> name parts in the same source
file will make the using program die with a NYI error.

The following is ok (loaded by "use Foo::Bar:ver<1.2.3>:auth<baz>"):

    # in Foo/Bar.pm
    class Foo::Bar:ver<1.2.3>:auth<baz> {       # or module, grammar etc
        ...
        class Foo::Bar::Baz {
            ...
        }
    }

The following (nested class declarations) will not be implemented in
Rakudo *:

    # in Foo.pm
    module Foo {                         # or class Foo, grammar Foo etc
        class Bar:ver<1.2.3>:auth<baz> {
            ...
        }
    }

=head2 No Unicode mangling in file names

If you want to use Unicode in your module names, your file system must
support Unicode as well.  If you want users without Unicode file names
to use your modules, don't use Unicode in your module names.

=head2 Retain @*INC to specify where searching begins

Rakudo effectively gives the following output to -e'.say for @*INC'

    ~/.perl6/lib
    parrot_install/lib/2.1.0-devel/languages/perl6/lib
    .

The . entry may be removed from the default @*INC because it creates a
security risk, but it is needed in the medium term for the build process
to conveniently find F<Test.pm>.

Unlike the Perl 5 case, all matching candidate files in all applicable
directories will be considered, so in most cases the order of
directories in @*INC is not significant.  If copies of the same module
name with the same C<:auth> and C<:ver> name parts exist in the same or
even different directories, Rakudo may arbitrarily use any one of those
files and ignore the others.  The module installer utility should try to
prevent such duplication arising, but should tolerate it if it already
exists.

=head2 Room for wriggling on file names

If multiple instances of a module exist, they may be distributed among 
all the @*INC directories.  Folders correspond to packages (aka
namespaces) and they are not allowed to have :ver and :auth name parts.

In every directory, file name collisions are avoided by optionally
inserting a unique .infix in the name before the .pm extension.  The
following would all match a C<use Foo;> command:

    Foo.pm
    Foo.1.pm
    Foo.12345.pm

Currently only digits are being considered, but anything within reason
between the two dots should be allowed, and is under the control of the
module installer.  The infix characters are meaningless.  Only the code
inside the file specifies :ver and :auth.

=head1 Searches in C<use>, C<need> and C<require> commands

In commands such as C<use>, the :auth and :ver name parts are
independently optional.  Rakudo * will do only exact matches on :auth
and :ver name parts, because the alternative gives headaches...

Consider the example "use Foo::Bar:ver<1.2.3>:auth<baz>"

Rakudo * will look for files matching Foo/Bar.pm and Foo/Bar.*.pm from
every starting point listed in @*INC.  Rakudo will then open each file
in turn and partially (how?) parse the content to extract the first :ver
and :auth values, building a list of the results.  Cacheing will
probably be added soon after the initial implemention works, in order to
reduce the obvious overheads.

If the C<use> specified an C<:auth> or a C<:ver>, the values must match,
and non matching files are disqualified.

Rakudo will consider files in the user's local directories (. and
~/.perl6/lib) that omit :auth and :ver values.  Modules in the
parrot_install tree should all have :auth and :ver.

If the :ver is not specified, Rakudo must select the file containing the
highest :ver value.  Files without :ver are considered as having the
lowest possible :ver value.  Multiple files without :ver, or multiple
files with the same :ver, will result in an arbitrary selection.<commit_msg>[docs/S11] fixed a small typo<commit_after>=head1 S11-Modules proposal for Rakudo * implementation

The aim will be to implement it all in NQP if possible.

=head1 Overriding Principle

The source code must always be the absolute source of all information.
Everything else should act as a cache.  That means *.pir files and
databases of metadata may be automatically or manually derived from the
*.pm files, but they may not add information (from the command line, for
example).

If there is to be cached metadata in future, it should be stored in
files as close to the corresponding source files as possible.
If modules are precompiled to *.pir files, those files will be stored
in the same directories as their corresponding *.pm source files, with
identical names apart from the extension.

=head1 Restrictions and limitations

=head2 No C<package> keyword

Classes may contain other classes, which provides sufficient hierarchy.

Rakudo will implement C<module>, in order to contain sub definitions
that do not belong to any class.

=head2 Only simplest :ver and :auth implementation

There should be only one C<:ver> and C<:auth> name part per source code
file, in order to keep the implementation simple.  In order to keep
users sane, multiple C<:ver> or C<:auth> name parts in the same source
file will make the using program die with a NYI error.

The following is ok (loaded by "use Foo::Bar:ver<1.2.3>:auth<baz>"):

    # in Foo/Bar.pm
    class Foo::Bar:ver<1.2.3>:auth<baz> {       # or module, grammar etc
        ...
        class Foo::Bar::Baz {
            ...
        }
    }

The following (nested class declarations) will not be implemented in
Rakudo *:

    # in Foo.pm
    module Foo {                         # or class Foo, grammar Foo etc
        class Bar:ver<1.2.3>:auth<baz> {
            ...
        }
    }

=head2 No Unicode mangling in file names

If you want to use Unicode in your module names, your file system must
support Unicode as well.  If you want users without Unicode file names
to use your modules, don't use Unicode in your module names.

=head2 Retain @*INC to specify where searching begins

Rakudo effectively gives the following output to -e'.say for @*INC'

    ~/.perl6/lib
    parrot_install/lib/2.1.0-devel/languages/perl6/lib
    .

The . entry may be removed from the default @*INC because it creates a
security risk, but it is needed in the medium term for the build process
to conveniently find F<Test.pm>.

Unlike the Perl 5 case, all matching candidate files in all applicable
directories will be considered, so in most cases the order of
directories in @*INC is not significant.  If copies of the same module
name with the same C<:auth> and C<:ver> name parts exist in the same or
even different directories, Rakudo may arbitrarily use any one of those
files and ignore the others.  The module installer utility should try to
prevent such duplication arising, but should tolerate it if it already
exists.

=head2 Room for wriggling on file names

If multiple instances of a module exist, they may be distributed among 
all the @*INC directories.  Folders correspond to packages (aka
namespaces) and they are not allowed to have :ver and :auth name parts.

In every directory, file name collisions are avoided by optionally
inserting a unique .infix in the name before the .pm extension.  The
following would all match a C<use Foo;> command:

    Foo.pm
    Foo.1.pm
    Foo.12345.pm

Currently only digits are being considered, but anything within reason
between the two dots should be allowed, and is under the control of the
module installer.  The infix characters are meaningless.  Only the code
inside the file specifies :ver and :auth.

=head1 Searches in C<use>, C<need> and C<require> commands

In commands such as C<use>, the :auth and :ver name parts are
independently optional.  Rakudo * will do only exact matches on :auth
and :ver name parts, because the alternative gives headaches...

Consider the example "use Foo::Bar:ver<1.2.3>:auth<baz>"

Rakudo * will look for files matching Foo/Bar.pm and Foo/Bar.*.pm from
every starting point listed in @*INC.  Rakudo will then open each file
in turn and partially (how?) parse the content to extract the first :ver
and :auth values, building a list of the results.  Cacheing will
probably be added soon after the initial implemention works, in order to
reduce the obvious overheads.

If the C<use> specified an C<:auth> or a C<:ver>, the values must match,
and non-matching files are disqualified.

Rakudo will consider files in the user's local directories (. and
~/.perl6/lib) that omit :auth and :ver values.  Modules in the
parrot_install tree should all have :auth and :ver.

If the :ver is not specified, Rakudo must select the file containing the
highest :ver value.  Files without :ver are considered as having the
lowest possible :ver value.  Multiple files without :ver, or multiple
files with the same :ver, will result in an arbitrary selection.
{'fraction_non_alphanumeric': 0.05961732124874119, 'fraction_numerical': 0.0074521651560926485, 'mean_word_length': 3.7200570342205324, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4413769', 'n_tokens_mistral': 2815, 'n_tokens_neox': 2627, 'n_words': 1492}