<commit_before>:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

--------------------------------------------------------------------
Theano: A CPU and GPU Math Compiler in Python
--------------------------------------------------------------------

.. class:: abstract

    *Theano is a compiler for mathematical expressions in Python that
    combines the convenience of NumPy's syntax with the speed
    of optimized native machine language.
    The user composes mathematical expressions in a high-level
    description that mimics NumPy's syntax and semantics, while being statically
    typed and functional (as opposed to imperative).
    These expressions allow Theano to provide symbolic differentiation.
    Before performing computation, Theano optimizes the choice of expressions,
    translates them into C++ (or CUDA for GPU),
    compiles them into dynamically loaded Python modules, all automatically.
    Common machine learning algorithms implemented with Theano
    are from* :math:`$1.6\times$` *to* :math:`$7.5\times$` *faster
    than competitive alternatives (including those implemented with
    C/C++, NumPy/SciPy and Matlab) when compiled for the CPU
    and between* :math:`$6.5\times$` *and* :math:`$44\times$` *faster
    when compiled for the GPU.
    This paper illustrates how to use
    Theano, outlines the scope of the compiler, provides benchmarks
    on both CPU and GPU processors, and explains its overall design.*



Introduction
------------

Python is a powerful and flexible language for describing large-scale mathematical
calculations, but the Python interpreter is in many cases a poor engine for executing
them. One reason is that Python uses full-fledged Python objects on the heap to
represent simple numeric scalars.
To reduce the overhead in numeric calculations, it is important to use array
types such as NumPy's ``ndarray`` so that single Python objects on the heap can
stand for multidimensional arrays of numeric scalars, each stored efficiently in
the host processor's native format.

[NumPy]_ provides an N-dimensional array data type, and many functions
for indexing, reshaping, and performing elementary computations (``exp``, ``log``,
``sin``, etc.) on entire arrays at once. These functions are implemented in C for
use within Python programs. However, the composition of many such NumPy functions
can be unnecessarily slow when each call is dominated by the cost of transferring
memory rather than the cost of performing calculations [Alted]_.
[numexpr]_ goes one step further by providing a loop fusion optimization
that can glue several elementwise computations together.
Unfortunately, the syntax required
by numexpr is a bit unusual (the expression must be encoded as a string
within the code), and it is limited to elementwise computations.
[Cython]_ and [scipy.weave]_ address Python's performance issue by offering a simple way to
hand-write crucial segments of code into C. While this might bring about 
significant improvements, if the bottleneck of a program is a large 
mathematical expression, comprising hundreds of operations, manual
optimization of the math can be time-consuming and suboptimal compared to
automatic optimization.

Theano on the other hand works on a symbolic representation of the
mathematical expressions, provided by the user through NumPy-like syntax.
Having access to a computational graph, Theano can
provide symbolic differentiation of complex expressions, but more
importantly, it can perform local graph trasnformations that corrects
many unnecessary, slow or numerically unstable expression patterns.
Once optimized, the graph can be used to generate CPU as well as GPU 
implementations ( the latter using CUDA) without requireing changes to 
user code. 

It is similar to [SymPy]_, in that both libraries manipulate symbolic
mathematical graphs, but the two projects have a distinctly different focus.
While SymPy implements a richer set of mathematical operations of the kind
expected in a modern computer algebra system, Theano focuses on fast, efficient
evaluation of primarily array-valued expressions.

Theano is free open source software, licensed under the New (3-clause) BSD license.
It depends upon NumPy, and can optionally use SciPy, as well as custom C and CUDA code
generators which are able to specialize for particular types, sizes, and shapes of
inputs. It can be extended to use ``scipy.weave``, PyCUDA, Cython, and other
numerical libraries and compilation technologies. Theano has been actively and
continuously developed and used since January 2008.
It has been used in several scientific papers and it is used to teach machine
learning in graduate courses at the Université de Montréal.
Documentation and installation instructions can be found on Theano's website [theano]_.
All Theano users should subscribe to the
`announce <http://groups.google.com/group/theano-announce>`_ [#]_ mailing list
(low traffic). There are medium traffic mailing lists for
`developer discussion <http://groups.google.com/group/theano-dev>`_ [#]_
and `user support <http://groups.google.com/group/theano-users>`_ [#]_.

This paper is divided as follows:
Section `Case Study: Logistic Regression`_ shows how Theano can be used to solve
a simple problem in statistical prediction.
Section `Benchmarking Results`_ presents some results of performance
benchmarking on problems related to machine learning and expression evaluation.
Section `What's in Theano`_ gives an overview of the design of Theano.
Section `Limitations and Future Work`_ outlines current limitations
outlines planned future work.

.. [#] http://groups.google.com/group/theano-announce
.. [#] http://groups.google.com/group/theano-dev
.. [#] http://groups.google.com/group/theano-users

.. _example1:

.. _caseStudy:

Case Study: Logistic Regression
------------------------------------------

To get a sense of how Theano feels from a user's perspective,
we will look at how to solve a binary logistic regression problem.
Binary logistic regression is a classification model
parametrized by a weight matrix :math:`W` and
bias vector :math:`b`.
The model estimates the probability
:math:`$P(Y=1|x)$` (which we will denote with shorthand :math:`$p$`) that the input
`x` belongs to class `y=1` as:<commit_msg>typo end of intro<commit_after>:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

:author: <NAME>
:email: <EMAIL>
:institution: University of Montreal

--------------------------------------------------------------------
Theano: A CPU and GPU Math Compiler in Python
--------------------------------------------------------------------

.. class:: abstract

    *Theano is a compiler for mathematical expressions in Python that
    combines the convenience of NumPy's syntax with the speed
    of optimized native machine language.
    The user composes mathematical expressions in a high-level
    description that mimics NumPy's syntax and semantics, while being statically
    typed and functional (as opposed to imperative).
    These expressions allow Theano to provide symbolic differentiation.
    Before performing computation, Theano optimizes the choice of expressions,
    translates them into C++ (or CUDA for GPU),
    compiles them into dynamically loaded Python modules, all automatically.
    Common machine learning algorithms implemented with Theano
    are from* :math:`$1.6\times$` *to* :math:`$7.5\times$` *faster
    than competitive alternatives (including those implemented with
    C/C++, NumPy/SciPy and Matlab) when compiled for the CPU
    and between* :math:`$6.5\times$` *and* :math:`$44\times$` *faster
    when compiled for the GPU.
    This paper illustrates how to use
    Theano, outlines the scope of the compiler, provides benchmarks
    on both CPU and GPU processors, and explains its overall design.*



Introduction
------------

Python is a powerful and flexible language for describing large-scale mathematical
calculations, but the Python interpreter is in many cases a poor engine for executing
them. One reason is that Python uses full-fledged Python objects on the heap to
represent simple numeric scalars.
To reduce the overhead in numeric calculations, it is important to use array
types such as NumPy's ``ndarray`` so that single Python objects on the heap can
stand for multidimensional arrays of numeric scalars, each stored efficiently in
the host processor's native format.

[NumPy]_ provides an N-dimensional array data type, and many functions
for indexing, reshaping, and performing elementary computations (``exp``, ``log``,
``sin``, etc.) on entire arrays at once. These functions are implemented in C for
use within Python programs. However, the composition of many such NumPy functions
can be unnecessarily slow when each call is dominated by the cost of transferring
memory rather than the cost of performing calculations [Alted]_.
[numexpr]_ goes one step further by providing a loop fusion optimization
that can glue several elementwise computations together.
Unfortunately, the syntax required
by numexpr is a bit unusual (the expression must be encoded as a string
within the code), and it is limited to elementwise computations.
[Cython]_ and [scipy.weave]_ address Python's performance issue by offering a simple way to
hand-write crucial segments of code into C. While this might bring about 
significant improvements, if the bottleneck of a program is a large 
mathematical expression, comprising hundreds of operations, manual
optimization of the math can be time-consuming and suboptimal compared to
automatic optimization.

Theano on the other hand works on a symbolic representation of the
mathematical expressions, provided by the user through NumPy-like syntax.
Having access to a computational graph, Theano can
provide symbolic differentiation of complex expressions, but more
importantly, it can perform local graph trasnformations that corrects
many unnecessary, slow or numerically unstable expression patterns.
Once optimized, the graph can be used to generate CPU as well as GPU 
implementations ( the latter using CUDA) without requireing changes to 
user code. 

It is similar to [SymPy]_, in that both libraries manipulate symbolic
mathematical graphs, but the two projects have a distinctly different focus.
While SymPy implements a richer set of mathematical operations of the kind
expected in a modern computer algebra system, Theano focuses on fast, efficient
evaluation of primarily array-valued expressions.

Theano is free open source software, licensed under the New (3-clause) BSD license.
It depends upon NumPy, and can optionally use SciPy, as well as custom C and CUDA code
generators which are able to specialize for particular types, sizes, and shapes of
inputs. It can be extended to use ``scipy.weave``, PyCUDA, Cython, and other
numerical libraries and compilation technologies. Theano has been actively and
continuously developed and used since January 2008.
It has been used in several scientific papers and it is used to teach machine
learning in graduate courses at the Université de Montréal.
Documentation and installation instructions can be found on Theano's website [theano]_.
All Theano users should subscribe to the
`announce <http://groups.google.com/group/theano-announce>`_ [#]_ mailing list
(low traffic). There are medium traffic mailing lists for
`developer discussion <http://groups.google.com/group/theano-dev>`_ [#]_
and `user support <http://groups.google.com/group/theano-users>`_ [#]_.

This paper is divided as follows:
Section `Case Study: Logistic Regression`_ shows how Theano can be used to solve
a simple problem in statistical prediction.
Section `Benchmarking Results`_ presents some results of performance
benchmarking on problems related to machine learning and expression evaluation.
Section `What's in Theano`_ gives an overview of the design of Theano.
Section `Limitations and Future Work`_ outlines current limitations
and planned future work.

.. [#] http://groups.google.com/group/theano-announce
.. [#] http://groups.google.com/group/theano-dev
.. [#] http://groups.google.com/group/theano-users

.. _example1:

.. _caseStudy:

Case Study: Logistic Regression
------------------------------------------

To get a sense of how Theano feels from a user's perspective,
we will look at how to solve a binary logistic regression problem.
Binary logistic regression is a classification model
parametrized by a weight matrix :math:`W` and
bias vector :math:`b`.
The model estimates the probability
:math:`$P(Y=1|x)$` (which we will denote with shorthand :math:`$p$`) that the input
`x` belongs to class `y=1` as:
{'fraction_non_alphanumeric': 0.08981555733761026, 'fraction_numerical': 0.002332871619158708, 'mean_word_length': 4.212006079027356, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7359149', 'n_tokens_mistral': 3522, 'n_tokens_neox': 3309, 'n_words': 1786}