<commit_before>Developer Tutorial
##################

This is a long tutorial to help new Pythran developer discover the Pythran
architecture. This is *not* a developer documentation, but it aims at giving a
good overview of Pythran capacity.

It requires you are comfortable with Python, and eventually with C++11. It also
assumes you have some compilation background, i.e. you know what an AST is and
you don't try to escape when hearing the words alias analysis, memory effect
computations and such.

Parsing Python Code
-------------------

Python ships a standard module, ``ast`` to turn Python code into an AST. For instance::

  >>> import ast
  >>> code = "a=1"
  >>> tree = ast.parse(code)  # turn the code into an AST
  >>> print ast.dump(tree)  # view it as a string
  Module(body=[Assign(targets=[Name(id='a', ctx=Store())], value=Num(n=1))])

Deciphering the above line, one learns that the single assignment is parsed as
a module containing a single statement, which is an assignment to a single
target, a ``ast.Name`` with the identifier ``a``, of the literal value ``1``.

Eventually, one needs to parse more complex codes, and things get a bit more cryptic, but you get the idea::

  >>> fib_src = """
  ... def fib(n):
  ...     return n if n< 2 else fib(n-1) + fib(n-2)"""
  >>> tree = ast.parse(fib_src)
  >>> print ast.dump(tree)
  Module(body=[FunctionDef(name='fib', args=arguments(args=[Name(id='n', ctx=Param())], vararg=None, kwarg=None, defaults=[]), body=[Return(value=IfExp(test=Compare(left=Name(id='n', ctx=Load()), ops=[Lt()], comparators=[Num(n=2)]), body=Name(id='n', ctx=Load()), orelse=BinOp(left=Call(func=Name(id='fib', ctx=Load()), args=[BinOp(left=Name(id='n', ctx=Load()), op=Sub(), right=Num(n=1))], keywords=[], starargs=None, kwargs=None), op=Add(), right=Call(func=Name(id='fib', ctx=Load()), args=[BinOp(left=Name(id='n', ctx=Load()), op=Sub(), right=Num(n=2))], keywords=[], starargs=None, kwargs=None))))], decorator_list=[])])

The idea remains the same. The whole Python syntax is described in
http://docs.python.org/2/library/ast.html and is worth a glance, otherwise
you'll be in serious trouble understanding the following.

Pythran Pass Manager
--------------------

A pass is a code transformation, i.e. a function that turns an AST node into a
new AST node with refined behavior. As a compiler infrastructure, Pythran
proposes a pass manager that (guess what?) manages pass scheduling, that is
the order in which pass is applied to achieve the ultimate goal, world
domination. Oooops, efficient C++11 code generation.

One first need to instantiate a pass manager with a module name::

  >>> from pythran import passmanager
  >>> pm = passmanager.PassManager("tutorial_module")

The pass manager has 3 methods and an attribute::

  >>> [x for x in dir(pm) if not x.startswith('_')]
  ['apply', 'dump', 'gather', 'module_name']

``apply``
    applies a code transformation

``dump``
    dumps a node using a dedicated backend

``gather``
    gathers information about the node

Pythran Backends
----------------

Pythran currently has two backends. The main one is used to dump Pythran AST (a
subset of Python AST) into a C++ AST::

  >>> from pythran import backend
  >>> cxx = pm.dump(backend.Cxx, tree)
  >>> str(cxx)
  '#include <pythonic/include/__builtin__/bool_.hpp>\n#include <pythonic/__builtin__/bool_.hpp>\nnamespace __pythran_tutorial_module\n{\n  ;\n  struct fib\n  {\n    typedef void callable;\n    typedef void pure;\n    template <typename argument_type0 >\n    struct type\n    {\n      typedef typename pythonic::returnable<typename std::remove_cv<typename std::remove_reference<argument_type0>::type>::type>::type result_type;\n    }  \n    ;\n    template <typename argument_type0 >\n    typename type<argument_type0>::result_type operator()(argument_type0 const & n) const\n    ;\n  }  ;\n  template <typename argument_type0 >\n  typename fib::type<argument_type0>::result_type fib::operator()(argument_type0 const & n) const\n  {\n    return (pythonic::__builtin__::functor::bool_{}((n < 2L)) ? n : (fib()((n - 1L)) + fib()((n - 2L))));\n  }\n}'

The above string is understandable by a C++11 compiler, but it quickly reaches the limit of our developer brain, so most of the time, we are more comfortable with the Python backend::

  >>> py = pm.dump(backend.Python, tree)
  >>> print py
  def fib(n):
      return (n if (n < 2) else (fib((n - 1)) + fib((n - 2))))

Passes
------

There are many code transformations in Pythran. Some of them are used to lower
the representation from Python AST to the simpler Pythran AST. For instance
there is no tuple unpacking in Pythran, so Pythran provides an adequate
transformation::

  >>> from pythran import transformations
  >>> tree = ast.parse("def foo(): a,b = 1,3.5")
  >>> _ = pm.apply(transformations.NormalizeTuples, tree)  # in-place
  >>> print pm.dump(backend.Python, tree)
  def foo():
      __tuple0 = (1, 3.5)
      a = __tuple0[0]
      b = __tuple0[1]

Pythran transforms the tuple unpacking into an intermediate tuple assignment.
This ensures that a single instruction from the input code maps to a single
instruction in the generated code. This property is enforced everywhere in
Pythran and is used to maintain OpenMP directive semantic without deeply
understanding it.

There are many small passes used iteratively to produce the Pythran AST. For instance the implicit return at the end of every function is made explicit::

  >>> tree = ast.parse('def foo():pass')
  >>> _ = pm.apply(transformations.NormalizeReturn, tree)
  >>> print pm.dump(backend.Python, tree)
  def foo():
      pass
      return __builtin__.None

There are many other passes in Pythran. For instance one can prevent clashes with C++ keywords::

  >>> tree = ast.parse('namespace_ = new = 1\nnamespace = namespace_ + new')
  >>> _ = pm.apply(transformations.NormalizeIdentifiers, tree)  # out is a renaming table
  >>> print pm.dump(backend.Python, tree)
  namespace_ = new_ = 1
  namespace__ = (namespace_ + new_)

More complex ones rely on introspection to implement constant folding::

  >>> code = [fib_src, 'def foo(): print __builtin__.map(fib, [1,2,3])']
  >>> fib_call = '\n'.join(code)
  >>> tree = ast.parse(fib_call)
  >>> from pythran import optimizations as optim
  >>> _ = pm.apply(optim.ConstantFolding, tree)
  >>> print pm.dump(backend.Python, tree)<commit_msg>Up-to-date tuple unpacking doc w/ OpenMP<commit_after>Developer Tutorial
##################

This is a long tutorial to help new Pythran developer discover the Pythran
architecture. This is *not* a developer documentation, but it aims at giving a
good overview of Pythran capacity.

It requires you are comfortable with Python, and eventually with C++11. It also
assumes you have some compilation background, i.e. you know what an AST is and
you don't try to escape when hearing the words alias analysis, memory effect
computations and such.

Parsing Python Code
-------------------

Python ships a standard module, ``ast`` to turn Python code into an AST. For instance::

  >>> import ast
  >>> code = "a=1"
  >>> tree = ast.parse(code)  # turn the code into an AST
  >>> print ast.dump(tree)  # view it as a string
  Module(body=[Assign(targets=[Name(id='a', ctx=Store())], value=Num(n=1))])

Deciphering the above line, one learns that the single assignment is parsed as
a module containing a single statement, which is an assignment to a single
target, a ``ast.Name`` with the identifier ``a``, of the literal value ``1``.

Eventually, one needs to parse more complex codes, and things get a bit more cryptic, but you get the idea::

  >>> fib_src = """
  ... def fib(n):
  ...     return n if n< 2 else fib(n-1) + fib(n-2)"""
  >>> tree = ast.parse(fib_src)
  >>> print ast.dump(tree)
  Module(body=[FunctionDef(name='fib', args=arguments(args=[Name(id='n', ctx=Param())], vararg=None, kwarg=None, defaults=[]), body=[Return(value=IfExp(test=Compare(left=Name(id='n', ctx=Load()), ops=[Lt()], comparators=[Num(n=2)]), body=Name(id='n', ctx=Load()), orelse=BinOp(left=Call(func=Name(id='fib', ctx=Load()), args=[BinOp(left=Name(id='n', ctx=Load()), op=Sub(), right=Num(n=1))], keywords=[], starargs=None, kwargs=None), op=Add(), right=Call(func=Name(id='fib', ctx=Load()), args=[BinOp(left=Name(id='n', ctx=Load()), op=Sub(), right=Num(n=2))], keywords=[], starargs=None, kwargs=None))))], decorator_list=[])])

The idea remains the same. The whole Python syntax is described in
http://docs.python.org/2/library/ast.html and is worth a glance, otherwise
you'll be in serious trouble understanding the following.

Pythran Pass Manager
--------------------

A pass is a code transformation, i.e. a function that turns an AST node into a
new AST node with refined behavior. As a compiler infrastructure, Pythran
proposes a pass manager that (guess what?) manages pass scheduling, that is
the order in which pass is applied to achieve the ultimate goal, world
domination. Oooops, efficient C++11 code generation.

One first need to instantiate a pass manager with a module name::

  >>> from pythran import passmanager
  >>> pm = passmanager.PassManager("tutorial_module")

The pass manager has 3 methods and an attribute::

  >>> [x for x in dir(pm) if not x.startswith('_')]
  ['apply', 'dump', 'gather', 'module_name']

``apply``
    applies a code transformation

``dump``
    dumps a node using a dedicated backend

``gather``
    gathers information about the node

Pythran Backends
----------------

Pythran currently has two backends. The main one is used to dump Pythran AST (a
subset of Python AST) into a C++ AST::

  >>> from pythran import backend
  >>> cxx = pm.dump(backend.Cxx, tree)
  >>> str(cxx)
  '#include <pythonic/include/__builtin__/bool_.hpp>\n#include <pythonic/__builtin__/bool_.hpp>\nnamespace __pythran_tutorial_module\n{\n  ;\n  struct fib\n  {\n    typedef void callable;\n    typedef void pure;\n    template <typename argument_type0 >\n    struct type\n    {\n      typedef typename pythonic::returnable<typename std::remove_cv<typename std::remove_reference<argument_type0>::type>::type>::type result_type;\n    }  \n    ;\n    template <typename argument_type0 >\n    typename type<argument_type0>::result_type operator()(argument_type0 const & n) const\n    ;\n  }  ;\n  template <typename argument_type0 >\n  typename fib::type<argument_type0>::result_type fib::operator()(argument_type0 const & n) const\n  {\n    return (pythonic::__builtin__::functor::bool_{}((n < 2L)) ? n : (fib()((n - 1L)) + fib()((n - 2L))));\n  }\n}'

The above string is understandable by a C++11 compiler, but it quickly reaches the limit of our developer brain, so most of the time, we are more comfortable with the Python backend::

  >>> py = pm.dump(backend.Python, tree)
  >>> print py
  def fib(n):
      return (n if (n < 2) else (fib((n - 1)) + fib((n - 2))))

Passes
------

There are many code transformations in Pythran. Some of them are used to lower
the representation from Python AST to the simpler Pythran AST. For instance
there is no tuple unpacking in Pythran, so Pythran provides an adequate
transformation::

  >>> from pythran import transformations
  >>> tree = ast.parse("def foo(): a,b = 1,3.5")
  >>> _ = pm.apply(transformations.NormalizeTuples, tree)  # in-place
  >>> print pm.dump(backend.Python, tree)
  def foo():
      __tuple0 = (1, 3.5)
      a = __tuple0[0]
      b = __tuple0[1]

Pythran transforms the tuple unpacking into an intermediate tuple assignment.
Note that if the unpacking statement is marked as critical using an OpenMP
statement, then a temporary variable is used to hold the left hand side
computation, if any::

  >>> from pythran import transformations
  >>> tree = ast.parse("""
  ... def foo(x):
  ...     #omp critical
  ...     a,b = bar(x)
  ...     return a + b""")
  >>> _ = pm.apply(transformations.NormalizeTuples, tree)  # in-place
  >>> print pm.dump(backend.Python, tree)
  def foo(x):
      __tuple0 = bar(x)
      a = __tuple0[0]
      b = __tuple0[1]
      return (a + b)

There are many small passes used iteratively to produce the Pythran AST. For instance the implicit return at the end of every function is made explicit::

  >>> tree = ast.parse('def foo():pass')
  >>> _ = pm.apply(transformations.NormalizeReturn, tree)
  >>> print pm.dump(backend.Python, tree)
  def foo():
      pass
      return __builtin__.None

There are many other passes in Pythran. For instance one can prevent clashes with C++ keywords::

  >>> tree = ast.parse('namespace_ = new = 1\nnamespace = namespace_ + new')
  >>> _ = pm.apply(transformations.NormalizeIdentifiers, tree)  # out is a renaming table
  >>> print pm.dump(backend.Python, tree)
  namespace_ = new_ = 1
  namespace__ = (namespace_ + new_)

More complex ones rely on introspection to implement constant folding::

  >>> code = [fib_src, 'def foo(): print __builtin__.map(fib, [1,2,3])']
  >>> fib_call = '\n'.join(code)
  >>> tree = ast.parse(fib_call)
  >>> from pythran import optimizations as optim
  >>> _ = pm.apply(optim.ConstantFolding, tree)
  >>> print pm.dump(backend.Python, tree)
{'fraction_non_alphanumeric': 0.13997102996111915, 'fraction_numerical': 0.007547457497903484, 'mean_word_length': 4.136256851996868, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 221, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7010042', 'n_tokens_mistral': 4092, 'n_tokens_neox': 3952, 'n_words': 1489}