<commit_before>.. -*-rst-*-

*********************************************
Developer notes on the transition to Python 3
*********************************************

:date: 2009-12-05
:author: <NAME>
:author: <NAME>

General
=======

If you work on Py3 transition, please try to keep this document up-to-date.

Resources
---------

Information on porting to 3K:

- http://wiki.python.org/moin/cporting
- http://wiki.python.org/moin/PortingExtensionModulesToPy3k


Git trees
---------

- http://github.com/pv/numpy-work/commits/py3k
- http://github.com/cournape/numpy/commits/py3_bootstrap
- http://github.com/illume/numpy3k/commits/work


Prerequisites
-------------

The Nose test framework has currently (Nov 2009) no released Python 3
compatible version. Its 3K SVN branch, however, works quite well:

- http://python-nose.googlecode.com/svn/branches/py3k


Known semantic changes on Py2
=============================

As a side effect, the Py3 adaptation has caused the following semantic
changes that are visible on Py2.

* Objects (non-string, non-unicode) that implement the PEP 3118 array interface
  will behave as ndarrays in `array(...)` and `asarray(...)`; the same way
  as if they had ``__array_interface__`` defined.

* Otherwise, there are no known semantic changes.


Known semantic changes on Py3
=============================

The following semantic changes have been made on Py3:

* Division: integer division is by default true_divide, also for arrays.

* Dtype field names are Unicode.

* Only unicode dtype field titles are included in fields dict.

* :pep:`3118` buffer objects will behave differently from Py2 buffer objects
  when used as an argument to `array(...)`, `asarray(...)`.

  In Py2, they would cast to an object array.

  In Py3, they cast similarly as objects having an
  ``__array_interface__`` attribute, ie., they behave as if they were
  an ndarray view on the data.

.. todo::

   Check for any other changes ... This we want in the end to include
   in the release notes, and also in a "how to port" document.


Python code
===========


2to3 in setup.py
----------------

Currently, setup.py calls 2to3 automatically to convert Python sources
to Python 3 ones, and stores the results under::

    build/py3k

Only changed files will be re-converted when setup.py is called a second
time, making development much faster.

Currently, this seems to handle most (all?) of the necessary Python
code conversion.

Not all of the 2to3 transformations are appropriate for all files.
Especially, 2to3 seems to be quite trigger-happy in replacing e.g.
``unicode`` by ``str`` which causes problems in ``defchararray.py``.
For files that need special handling, add entries to
``tools/py3tool.py``.

.. todo::

   Should we be a good citizen and use ``lib2to3`` instead?

.. todo::

   Do we want to get rid of this hack in the long run?


numpy.compat.py3k
-----------------

There are some utility functions needed for 3K compatibility in
``numpy.compat.py3k`` -- they can be imported from ``numpy.compat``:

- bytes: bytes constructor
- asbytes: convert string to bytes (no-op on Py2)
- getexception: get current exception (see below)
- isfileobj: detect Python file objects

More can be added as needed.


Bytes vs. strings
-----------------

At many points in Numpy, bytes literals are needed. These can be created via
numpy.compat.asbytes and asbytes_nested.
<commit_msg>3K: doc: note that f2py hasn't been really ported yet<commit_after>.. -*-rst-*-

*********************************************
Developer notes on the transition to Python 3
*********************************************

:date: 2009-12-05
:author: <NAME>
:author: <NAME>

General
=======

If you work on Py3 transition, please try to keep this document up-to-date.

Resources
---------

Information on porting to 3K:

- http://wiki.python.org/moin/cporting
- http://wiki.python.org/moin/PortingExtensionModulesToPy3k


Git trees
---------

- http://github.com/pv/numpy-work/commits/py3k
- http://github.com/cournape/numpy/commits/py3_bootstrap
- http://github.com/illume/numpy3k/commits/work


Prerequisites
-------------

The Nose test framework has currently (Nov 2009) no released Python 3
compatible version. Its 3K SVN branch, however, works quite well:

- http://python-nose.googlecode.com/svn/branches/py3k


Known semantic changes on Py2
=============================

As a side effect, the Py3 adaptation has caused the following semantic
changes that are visible on Py2.

* Objects (non-string, non-unicode) that implement the PEP 3118 array interface
  will behave as ndarrays in `array(...)` and `asarray(...)`; the same way
  as if they had ``__array_interface__`` defined.

* Otherwise, there are no known semantic changes.


Known semantic changes on Py3
=============================

The following semantic changes have been made on Py3:

* Division: integer division is by default true_divide, also for arrays.

* Dtype field names are Unicode.

* Only unicode dtype field titles are included in fields dict.

* :pep:`3118` buffer objects will behave differently from Py2 buffer objects
  when used as an argument to `array(...)`, `asarray(...)`.

  In Py2, they would cast to an object array.

  In Py3, they cast similarly as objects having an
  ``__array_interface__`` attribute, ie., they behave as if they were
  an ndarray view on the data.

.. todo::

   Check for any other changes ... This we want in the end to include
   in the release notes, and also in a "how to port" document.


Python code
===========


2to3 in setup.py
----------------

Currently, setup.py calls 2to3 automatically to convert Python sources
to Python 3 ones, and stores the results under::

    build/py3k

Only changed files will be re-converted when setup.py is called a second
time, making development much faster.

Currently, this seems to handle most (all?) of the necessary Python
code conversion.

Not all of the 2to3 transformations are appropriate for all files.
Especially, 2to3 seems to be quite trigger-happy in replacing e.g.
``unicode`` by ``str`` which causes problems in ``defchararray.py``.
For files that need special handling, add entries to
``tools/py3tool.py``.

.. todo::

   Should we be a good citizen and use ``lib2to3`` instead?

.. todo::

   Do we want to get rid of this hack in the long run?


numpy.compat.py3k
-----------------

There are some utility functions needed for 3K compatibility in
``numpy.compat.py3k`` -- they can be imported from ``numpy.compat``:

- bytes: bytes constructor
- asbytes: convert string to bytes (no-op on Py2)
- getexception: get current exception (see below)
- isfileobj: detect Python file objects

More can be added as needed.


numpy.f2py
----------

F2py has not yet been ported to Py3.

It imports, but it's not tested.

.. todo::

   Port f2py to Py3


Bytes vs. strings
-----------------

At many points in Numpy, bytes literals are needed. These can be created via
numpy.compat.asbytes and asbytes_nested.

{'fraction_non_alphanumeric': 0.14935622317596567, 'fraction_numerical': 0.016738197424892704, 'mean_word_length': 3.5103225806451612, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1114429', 'n_tokens_mistral': 2105, 'n_tokens_neox': 1964, 'n_words': 877}