<commit_before>Introduction
------------

Jltools is an extensible Java compiler toolkit designed for experimentation
with new language extensions.

The base jltools compiler, jlc, is a mostly-complete Java frontend; that is, it
parses and performs semantic checking on Java source code.  The compiler
outputs Java source code.  Thus, the base compiler implements the identity
translation.

Language extensions are implemented on top of the base compiler by extending
the concrete and abstract syntax and the type system.  Ideally, after type
checking the language extension, the abstract syntax tree (AST) is translated 
into a Java AST and the existing code is output into a Java source file which 
can then be compiled with javac.  For historical reasons, most extensions 
just override some portions of the Java output code to handle the extended 
syntax of the particular language extension being compiled.  We are 
currently reworkng the compiler to make it easier to rewrite a language 
extension's AST into a Java AST.


Architecture
------------

The jltools compiler is structured as a set of passes over source files
that ends with the output Java source code.  The passes parse the original
source language and create an AST, rewrite the AST to eliminate
any ambiguities, type check the AST, possibly rewrite the AST to another
AST, then output the AST as Java source code.

When the compiler is invoked (through jltools.main.Main.main()), it parses
the command line (with jltools.main.Options), then creates a compiler object
(an instance of jltools.frontend.Compiler) to manage the compilation process.
An important job of the command line parser is to identify the language
extension (specified on the command line with -ext L, where L is the name
of the extension), and load the extension (from jltools.ext.L.ExtensionInfo).
The compiler uses the extension to determine several important features of
the language, including its source file extension, AST node factory, type
system, and pass schedule.

Parsing is done with the Java CUP parser generator and a jltools extension
to it called JLgen.  JLgen allows CUP files to be selectively extended
to create parsers for extension languages by providing operations on
a CUP grammar, including adding, dropping, and renaming of productions.
The JFlex lexer generator is used to create a lexer for the source
language.  The semantic actions in the parser create an AST through
a NodeFactory, which is a class containing factory methods for
creating AST nodes.

After the AST has been created by the parser, a series of passes is
performed upon it.  The passes for a language extension, as well as the
order in which they should be run, are defined in the extension's
ExtensionInfo class.  The compiler object schedules the passes and runs
them on the AST.  Most passes are implemented using a modified version
of the Visitor design pattern (//FIXME: we should describe this somewhere).
The default set of passes is:

    * parsing, as described above
    * build-types (TypeBuilder):  Constructs the Type object representing
        each type in the source file and stores it in a Resolver
        associated with the source.
    * clean-super (AmbiguityRemover):  Removes any ambiguities found
        in the supertypes of a type.
    * clean-sigs (AmbiguityRemover):  Removes any ambiguities found in
        the members of a class or interface.
    * add-members (AddMemberVisitor):  Adds the members of a class or
        interface to its type object.
    * disambiguate (AmbiguityRemover):  Removes any ambiguities found
        in the bodies of methods.
    * constant folding (ConstantFolder):  Performs a simple constant
        folding optimization.
    * type checking (TypeChecker):  Performs semantic analysis for Java.
    * exception checking (ExceptionChecker):  Performs semantic analysis
        upon exception declaration and propagation.
    * dump (DumpAst):  Optional.  A debugging pass that outputs the AST.
    * serialization (ClassSerializer):  Optional.  Serializes information<commit_msg>Minor tweaks to descriptions of the passes.<commit_after>Introduction
------------

Jltools is an extensible Java compiler toolkit designed for experimentation
with new language extensions.

The base jltools compiler, jlc, is a mostly-complete Java frontend; that is, it
parses and performs semantic checking on Java source code.  The compiler
outputs Java source code.  Thus, the base compiler implements the identity
translation.

Language extensions are implemented on top of the base compiler by extending
the concrete and abstract syntax and the type system.  Ideally, after type
checking the language extension, the abstract syntax tree (AST) is translated 
into a Java AST and the existing code is output into a Java source file which 
can then be compiled with javac.  For historical reasons, most extensions 
just override some portions of the Java output code to handle the extended 
syntax of the particular language extension being compiled.  We are 
currently reworkng the compiler to make it easier to rewrite a language 
extension's AST into a Java AST.


Architecture
------------

The jltools compiler is structured as a set of passes over source files
that ends with the output Java source code.  The passes parse the original
source language and create an AST, rewrite the AST to eliminate
any ambiguities, type check the AST, possibly rewrite the AST to another
AST, then output the AST as Java source code.

When the compiler is invoked (through jltools.main.Main.main()), it parses
the command line (with jltools.main.Options), then creates a compiler object
(an instance of jltools.frontend.Compiler) to manage the compilation process.
An important job of the command line parser is to identify the language
extension (specified on the command line with -ext L, where L is the name
of the extension), and load the extension (from jltools.ext.L.ExtensionInfo).
The compiler uses the extension to determine several important features of
the language, including its source file extension, AST node factory, type
system, and pass schedule.

Parsing is done with the Java CUP parser generator and a jltools extension
to it called JLgen.  JLgen allows CUP files to be selectively extended
to create parsers for extension languages by providing operations on
a CUP grammar, including adding, dropping, and renaming of productions.
The JFlex lexer generator is used to create a lexer for the source
language.  The semantic actions in the parser create an AST through
a NodeFactory, which is a class containing factory methods for
creating AST nodes.

After the AST has been created by the parser, a series of passes is
performed upon it.  The passes for a language extension, as well as the
order in which they should be run, are defined in the extension's
ExtensionInfo class.  The compiler object schedules the passes and runs
them on the AST.  Most passes are implemented using a modified version
of the Visitor design pattern (//FIXME: we should describe this somewhere).
The default set of passes is:

    * parsing, as described above
    * build-types (TypeBuilder):  Constructs the Type object representing
        each type in the source file and stores it in a Resolver
        associated with the source.
    * clean-super (AmbiguityRemover):  Removes any ambiguities found
	in the declaration of the supertypes of a type (e.g., the "extends"
	clause).
    * clean-sigs (AmbiguityRemover):  Removes any ambiguities found in
        the signatures of class or interface members.
    * add-members (AddMemberVisitor):  Adds the members of a class or
        interface to its type object.
    * disambiguate (AmbiguityRemover):  Removes any ambiguities found
        in the bodies of methods, constructors, or initializers.
    * constant folding (ConstantFolder):  Performs a simple constant
        folding optimization.
    * type checking (TypeChecker):  Performs semantic analysis for Java.
    * exception checking (ExceptionChecker):  Performs semantic analysis
        upon exception declaration and propagation.
    * dump (DumpAst):  Optional.  A debugging pass that outputs the AST.
    * serialization (ClassSerializer):  Optional.  Serializes information
{'fraction_non_alphanumeric': 0.041364296081277216, 'fraction_numerical': 0.0, 'mean_word_length': 4.017597087378641, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3988144', 'n_tokens_mistral': 2024, 'n_tokens_neox': 1918, 'n_words': 1183}