<commit_before>GAST, daou naer!
================

A generic AST to represent Python2 and Python3's Abstract Syntax Tree(AST).

GAST provides a compatibility layer between the AST of various Python versions,
as produced by ``ast.parse`` from the standard ``ast`` module.

Basic Usage
-----------

.. code:: python

    >>> import ast, gast
    >>> code = open('file.py').read()
    >>> tree = ast.parse(code)
    >>> gtree = gast.ast_to_gast(tree)
    >>> ... # process gtree
    >>> tree = gast.gast_to_ast(gtree)
    >>> ... # do stuff specific to tree

API
---

``gast`` provides the same API as the ``ast`` module. All functions and classes
from the ``ast`` module are also available in the ``gast`` module, and work
accordingly on the ``gast`` tree.

Three notable exceptions:

1. ``gast.parse`` directly produces a GAST node. It's equivalent to running
       ``gast.ast_to_gast`` on the output of ``ast.parse``.

2. ``gast.dump`` dumps the ``gast`` common representation, not the original
       one.

3. ``gast.gast_to_ast`` and ``gast.ast_to_gast`` can be used to convert
       from one ast to the other, back and forth.

Version Compatibility
---------------------

GAST is tested using ``tox`` and Travis on the following Python versions:

- 2.7
- 3.0
- 3.1
- 3.2
- 3.3
- 3.4
- 3.5

AST Changes
-----------


Python3
*******

The AST used by GAST is the same as the one used in Python3.5, to the
notable exception of ``ast.arg`` being replaced by ``ast.Name`` with an
``ast.Param`` context. Addiitonnaly, ``ast.Name`` have an extra ``annotation``
field.

Python2
*******

To cope with Python3 features, several nodes from the Python2 AST are extended
with some new attributes/children.

- ``FunctionDef`` nodes have a ``returns`` attribute.

- ``ClassDef`` nodes have a ``keywords`` attribute.

- ``With``'s ``context_expr`` and ``optional_vars`` fields are hold in a
  ``withitem`` object.

- ``Raise``'s ``type``, ``inst`` and ``tback`` fields are hold in a single
  ``exc`` field, using the transformation ``raise E, V, T => raise E(V).with_traceback(T)``.

- ``TryExcept`` and ``TryFinally`` nodes are merged in the ``Try`` node.

- ``Name`` nodes have an ``annotation`` attribute.

- ``arguments`` nodes have a ``kwonlyargs`` and ``kw_defaults`` attributes.

- ``Call`` nodes loose their ``starargs`` attribute, replaced by an
  argument wrapped in a ``Starred`` node. They also loose their ``kwargs``
  attribute, wrapped in a ``keyword`` node with the identifier set to
  ``None``, as done in Python3.

ASDL
****

.. code::

    -- ASDL's six builtin types are identifier, int, string, bytes, object, singleton

    module Python
    {
        mod = Module(stmt* body)
            | Interactive(stmt* body)
            | Expression(expr body)
<commit_msg>Explicitly warn about NamedConstant vs. Name<commit_after>GAST, daou naer!
================

A generic AST to represent Python2 and Python3's Abstract Syntax Tree(AST).

GAST provides a compatibility layer between the AST of various Python versions,
as produced by ``ast.parse`` from the standard ``ast`` module.

Basic Usage
-----------

.. code:: python

    >>> import ast, gast
    >>> code = open('file.py').read()
    >>> tree = ast.parse(code)
    >>> gtree = gast.ast_to_gast(tree)
    >>> ... # process gtree
    >>> tree = gast.gast_to_ast(gtree)
    >>> ... # do stuff specific to tree

API
---

``gast`` provides the same API as the ``ast`` module. All functions and classes
from the ``ast`` module are also available in the ``gast`` module, and work
accordingly on the ``gast`` tree.

Three notable exceptions:

1. ``gast.parse`` directly produces a GAST node. It's equivalent to running
       ``gast.ast_to_gast`` on the output of ``ast.parse``.

2. ``gast.dump`` dumps the ``gast`` common representation, not the original
       one.

3. ``gast.gast_to_ast`` and ``gast.ast_to_gast`` can be used to convert
       from one ast to the other, back and forth.

Version Compatibility
---------------------

GAST is tested using ``tox`` and Travis on the following Python versions:

- 2.7
- 3.0
- 3.1
- 3.2
- 3.3
- 3.4
- 3.5

AST Changes
-----------


Python3
*******

The AST used by GAST is the same as the one used in Python3.5, to the
notable exception of ``ast.arg`` being replaced by ``ast.Name`` with an
``ast.Param`` context. Addiitonnaly, ``ast.Name`` have an extra ``annotation``
field.

Python2
*******

To cope with Python3 features, several nodes from the Python2 AST are extended
with some new attributes/children.

- ``FunctionDef`` nodes have a ``returns`` attribute.

- ``ClassDef`` nodes have a ``keywords`` attribute.

- ``With``'s ``context_expr`` and ``optional_vars`` fields are hold in a
  ``withitem`` object.

- ``Raise``'s ``type``, ``inst`` and ``tback`` fields are hold in a single
  ``exc`` field, using the transformation ``raise E, V, T => raise E(V).with_traceback(T)``.

- ``TryExcept`` and ``TryFinally`` nodes are merged in the ``Try`` node.

- ``Name`` nodes have an ``annotation`` attribute.

- ``arguments`` nodes have a ``kwonlyargs`` and ``kw_defaults`` attributes.

- ``Call`` nodes loose their ``starargs`` attribute, replaced by an
  argument wrapped in a ``Starred`` node. They also loose their ``kwargs``
  attribute, wrapped in a ``keyword`` node with the identifier set to
  ``None``, as done in Python3.

Pit Falls
*********

- In Python3, ``None``, ``True`` and ``False`` are parsed as ``NamedConstant``
  while they are parsed as regular ``Name`` in Python2. ``gast`` uses the same
  convention.

ASDL
****

.. code::

    -- ASDL's six builtin types are identifier, int, string, bytes, object, singleton

    module Python
    {
        mod = Module(stmt* body)
            | Interactive(stmt* body)
            | Expression(expr body)

{'fraction_non_alphanumeric': 0.16028417951828106, 'fraction_numerical': 0.009357130479986138, 'mean_word_length': 3.3268365817091454, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5669759', 'n_tokens_mistral': 1812, 'n_tokens_neox': 1744, 'n_words': 715}