<commit_before>\chapter{Python compiler package \label{compiler}}

\sectionauthor{<NAME>}{<EMAIL>}


The Python compiler package is a tool for analyzing Python source code
and generating Python bytecode.  The compiler contains libraries to
generate an abstract syntax tree from Python source code and to
generate Python bytecode from the tree.

The \refmodule{compiler} package is a Python source to bytecode
translator written in Python.  It uses the built-in parser and
standard \refmodule{parser} module to generated a concrete syntax
tree.  This tree is used to generate an abstract syntax tree (AST) and
then Python bytecode.

The full functionality of the package duplicates the builtin compiler
provided with the Python interpreter.  It is intended to match its
behavior almost exactly.  Why implement another compiler that does the
same thing?  The package is useful for a variety of purposes.  It can
be modified more easily than the builtin compiler.  The AST it
generates is useful for analyzing Python source code.

This chapter explains how the various components of the
\refmodule{compiler} package work.  It blends reference material with
a tutorial.

The following modules are part of the \refmodule{compiler} package:

\localmoduletable


\subsection{The basic interface}

\declaremodule{}{compiler}

The top-level of the package defines four functions.  If you import
\module{compiler}, you will get these functions and a collection of
modules contained in the package.

\begin{funcdesc}{parse}{buf}
Returns an abstract syntax tree for the Python source code in \var{buf}.
The function raises SyntaxError if there is an error in the source
code.  The return value is a \class{compiler.ast.Module} instance that
contains the tree.  
\end{funcdesc}

\begin{funcdesc}{parseFile}{path}
Return an abstract syntax tree for the Python source code in the file
specified by \var{path}.  It is equivalent to
\code{parse(open(\var{path}).read())}.
\end{funcdesc}

\begin{funcdesc}{walk}{ast, visitor\optional{, verbose}}
Do a pre-order walk over the abstract syntax tree \var{ast}.  Call the
appropriate method on the \var{visitor} instance for each node
encountered.
\end{funcdesc}

\begin{funcdesc}{compile}{path}
Compile the file \var{path} and generate the corresponding \file{.pyc}
file.
\end{funcdesc}

The \module{compiler} package contains the following modules:
\refmodule[compiler.ast]{ast}, \module{consts}, \module{future},
\module{misc}, \module{pyassem}, \module{pycodegen}, \module{symbols},
\module{transformer}, and \refmodule[compiler.visitor]{visitor}.

\subsection{Limitations}

There are some problems with the error checking of the compiler
package.  The interpreter detects syntax errors in two distinct
phases.  One set of errors is detected by the interpreter's parser,
the other set by the compiler.  The compiler package relies on the
interpreter's parser, so it get the first phases of error checking for
free.  It implements the second phase itself, and that implement is
incomplete.  For example, the compiler package does not raise an error
if a name appears more than once in an argument list: 
\code{def f(x, x): ...}

A future version of the compiler should fix these problems.

\section{Python Abstract Syntax}

The \module{compiler.ast} module defines an abstract syntax for
Python.  In the abstract syntax tree, each node represents a syntactic<commit_msg>Make sure there are no weird nesting behaviors; the PDF version will not format if there are (building the "bookmarks" for the reader breaks).<commit_after>\chapter{Python compiler package \label{compiler}}

\sectionauthor{<NAME>}{<EMAIL>}


The Python compiler package is a tool for analyzing Python source code
and generating Python bytecode.  The compiler contains libraries to
generate an abstract syntax tree from Python source code and to
generate Python bytecode from the tree.

The \refmodule{compiler} package is a Python source to bytecode
translator written in Python.  It uses the built-in parser and
standard \refmodule{parser} module to generated a concrete syntax
tree.  This tree is used to generate an abstract syntax tree (AST) and
then Python bytecode.

The full functionality of the package duplicates the builtin compiler
provided with the Python interpreter.  It is intended to match its
behavior almost exactly.  Why implement another compiler that does the
same thing?  The package is useful for a variety of purposes.  It can
be modified more easily than the builtin compiler.  The AST it
generates is useful for analyzing Python source code.

This chapter explains how the various components of the
\refmodule{compiler} package work.  It blends reference material with
a tutorial.

The following modules are part of the \refmodule{compiler} package:

\localmoduletable


\section{The basic interface}

\declaremodule{}{compiler}

The top-level of the package defines four functions.  If you import
\module{compiler}, you will get these functions and a collection of
modules contained in the package.

\begin{funcdesc}{parse}{buf}
Returns an abstract syntax tree for the Python source code in \var{buf}.
The function raises SyntaxError if there is an error in the source
code.  The return value is a \class{compiler.ast.Module} instance that
contains the tree.  
\end{funcdesc}

\begin{funcdesc}{parseFile}{path}
Return an abstract syntax tree for the Python source code in the file
specified by \var{path}.  It is equivalent to
\code{parse(open(\var{path}).read())}.
\end{funcdesc}

\begin{funcdesc}{walk}{ast, visitor\optional{, verbose}}
Do a pre-order walk over the abstract syntax tree \var{ast}.  Call the
appropriate method on the \var{visitor} instance for each node
encountered.
\end{funcdesc}

\begin{funcdesc}{compile}{path}
Compile the file \var{path} and generate the corresponding \file{.pyc}
file.
\end{funcdesc}

The \module{compiler} package contains the following modules:
\refmodule[compiler.ast]{ast}, \module{consts}, \module{future},
\module{misc}, \module{pyassem}, \module{pycodegen}, \module{symbols},
\module{transformer}, and \refmodule[compiler.visitor]{visitor}.

\section{Limitations}

There are some problems with the error checking of the compiler
package.  The interpreter detects syntax errors in two distinct
phases.  One set of errors is detected by the interpreter's parser,
the other set by the compiler.  The compiler package relies on the
interpreter's parser, so it get the first phases of error checking for
free.  It implements the second phase itself, and that implement is
incomplete.  For example, the compiler package does not raise an error
if a name appears more than once in an argument list: 
\code{def f(x, x): ...}

A future version of the compiler should fix these problems.

\section{Python Abstract Syntax}

The \module{compiler.ast} module defines an abstract syntax for
Python.  In the abstract syntax tree, each node represents a syntactic
{'fraction_non_alphanumeric': 0.06875631951466127, 'fraction_numerical': 0.0, 'mean_word_length': 4.862828111769686, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5036323', 'n_tokens_mistral': 1823, 'n_tokens_neox': 1791, 'n_words': 918}