<commit_before>.. -*-rst-*-

*********************************************
Developer notes on the transition to Python 3
*********************************************

:date: 2010-07-11
:author: <NAME>
:author: <NAME>

General
=======

If you work on Py3 transition, please try to keep this document up-to-date.

Resources
---------

Information on porting to 3K:

- http://wiki.python.org/moin/cporting
- http://wiki.python.org/moin/PortingExtensionModulesToPy3k


Git trees
---------

- http://github.com/pv/numpy-work/commits/py3k
- http://github.com/cournape/numpy/commits/py3_bootstrap
- http://github.com/illume/numpy3k/commits/work


Prerequisites
-------------

The Nose test framework has currently (Nov 2009) no released Python 3
compatible version. Its 3K SVN branch, however, works quite well:

- http://python-nose.googlecode.com/svn/branches/py3k


Known semantic changes on Py2
=============================

As a side effect, the Py3 adaptation has caused the following semantic
changes that are visible on Py2.

* Objects (except bytes and str) that implement the PEP 3118 array interface
  will behave as ndarrays in `array(...)` and `asarray(...)`; the same way
  as if they had ``__array_interface__`` defined.

* Otherwise, there are no known semantic changes.


Known semantic changes on Py3
=============================

The following semantic changes have been made on Py3:

* Division: integer division is by default true_divide, also for arrays.

* Dtype field names are Unicode.

* Only unicode dtype field titles are included in fields dict.

* :pep:`3118` buffer objects will behave differently from Py2 buffer objects
  when used as an argument to `array(...)`, `asarray(...)`.

  In Py2, they would cast to an object array.

  In Py3, they cast similarly as objects having an
  ``__array_interface__`` attribute, ie., they behave as if they were
  an ndarray view on the data.



Python code
===========


2to3 in setup.py
----------------

Currently, setup.py calls 2to3 automatically to convert Python sources
to Python 3 ones, and stores the results under::

    build/py3k

Only changed files will be re-converted when setup.py is called a second
time, making development much faster.

Currently, this seems to handle all of the necessary Python code
conversion.

Not all of the 2to3 transformations are appropriate for all files.
Especially, 2to3 seems to be quite trigger-happy in replacing e.g.
``unicode`` by ``str`` which causes problems in ``defchararray.py``.
For files that need special handling, add entries to
``tools/py3tool.py``.



numpy.compat.py3k
-----------------

There are some utility functions needed for 3K compatibility in
``numpy.compat.py3k`` -- they can be imported from ``numpy.compat``:

- bytes, unicode: bytes and unicode constructors
- asbytes: convert string to bytes (no-op on Py2)
- asbytes_nested: convert strings in lists to Bytes
- asunicode: convert string to unicode
- asunicode_nested: convert strings in lists to Unicode
- asstr: convert item to the str type
- getexception: get current exception (see below)
- isfileobj: detect Python file objects
- strchar: character for Unicode (Py3) or Strings (Py2)
- open_latin1: open file in the latin1 text mode

More can be added as needed.


numpy.f2py
----------

F2py is ported to Py3.


Bytes vs. strings
-----------------

At many points in Numpy, bytes literals are needed. These can be created via
numpy.compat.asbytes and asbytes_nested.


Exception syntax
----------------

Syntax change: "except FooException, bar:" -> "except FooException as bar:"

This is taken care by 2to3, however.


Relative imports
----------------

The new relative import syntax,

    from . import foo

is not available on Py2.4, so we can't simply use it.

Using absolute imports everywhere is probably OK, if they just happen
to work.

2to3, however, converts the old syntax to new syntax, so as long as we
use the converter, it takes care of most parts.


Print
-----

The Print statement changed to a builtin function in Py3.

Also this is taken care of by 2to3.

types module
------------

The following items were removed from `types` module in Py3:

- StringType    (Py3: `bytes` is equivalent, to some degree)
- InstanceType  (Py3: ???)
- IntType       (Py3: no equivalent)
- LongType      (Py3: equivalent `long`)
- FloatType     (Py3: equivalent `float`)
- BooleanType   (Py3: equivalent `bool`)
- ComplexType   (Py3: equivalent `complex`)
- UnicodeType   (Py3: equivalent `str`)
- BufferType    (Py3: more-or-less equivalent `memoryview`)

In ``numerictypes.py``, the "common" types were replaced by their
plain equivalents, and `IntType` was dropped.


numpy.core.numerictypes
-----------------------

In numerictypes, types on Python 3 were changed so that:

===========   ============
Scalar type   Value
===========   ============
str_          This is the basic Unicode string type on Py3
bytes_        This is the basic Byte-string type on Py3
string_       bytes_ alias
unicode_      str_ alias
===========   ============


numpy.loadtxt et al
-------------------

These routines are difficult to duck-type to read both Unicode and
Bytes input.

I assumed they are meant for reading Bytes streams -- this is probably
the far more common use case with scientific data.<commit_msg>DOC: more updates to Py3K.txt<commit_after>.. -*-rst-*-

*********************************************
Developer notes on the transition to Python 3
*********************************************

:date: 2010-07-11
:author: <NAME>
:author: <NAME>

General
=======

Numpy has now been ported to Python 3.

Some glitches may still be present; however, we are not aware of any
significant ones, the test suite passes.


Resources
---------

Information on porting to 3K:

- http://wiki.python.org/moin/cporting
- http://wiki.python.org/moin/PortingExtensionModulesToPy3k


Prerequisites
-------------

The Nose test framework has currently (Nov 2009) no released Python 3
compatible version. Its 3K SVN branch, however, works quite well:

- http://python-nose.googlecode.com/svn/branches/py3k


Known semantic changes on Py2
=============================

As a side effect, the Py3 adaptation has caused the following semantic
changes that are visible on Py2.

* Objects (except bytes and str) that implement the PEP 3118 array interface
  will behave as ndarrays in `array(...)` and `asarray(...)`; the same way
  as if they had ``__array_interface__`` defined.

* Otherwise, there are no known semantic changes.


Known semantic changes on Py3
=============================

The following semantic changes have been made on Py3:

* Division: integer division is by default true_divide, also for arrays.

* Dtype field names are Unicode.

* Only unicode dtype field titles are included in fields dict.

* :pep:`3118` buffer objects will behave differently from Py2 buffer objects
  when used as an argument to `array(...)`, `asarray(...)`.

  In Py2, they would cast to an object array.

  In Py3, they cast similarly as objects having an
  ``__array_interface__`` attribute, ie., they behave as if they were
  an ndarray view on the data.



Python code
===========


2to3 in setup.py
----------------

Currently, setup.py calls 2to3 automatically to convert Python sources
to Python 3 ones, and stores the results under::

    build/py3k

Only changed files will be re-converted when setup.py is called a second
time, making development much faster.

Currently, this seems to handle all of the necessary Python code
conversion.

Not all of the 2to3 transformations are appropriate for all files.
Especially, 2to3 seems to be quite trigger-happy in replacing e.g.
``unicode`` by ``str`` which causes problems in ``defchararray.py``.
For files that need special handling, add entries to
``tools/py3tool.py``.



numpy.compat.py3k
-----------------

There are some utility functions needed for 3K compatibility in
``numpy.compat.py3k`` -- they can be imported from ``numpy.compat``:

- bytes, unicode: bytes and unicode constructors
- asbytes: convert string to bytes (no-op on Py2)
- asbytes_nested: convert strings in lists to Bytes
- asunicode: convert string to unicode
- asunicode_nested: convert strings in lists to Unicode
- asstr: convert item to the str type
- getexception: get current exception (see below)
- isfileobj: detect Python file objects
- strchar: character for Unicode (Py3) or Strings (Py2)
- open_latin1: open file in the latin1 text mode

More can be added as needed.


numpy.f2py
----------

F2py is ported to Py3.


Bytes vs. strings
-----------------

At many points in Numpy, bytes literals are needed. These can be created via
numpy.compat.asbytes and asbytes_nested.


Exception syntax
----------------

Syntax change: "except FooException, bar:" -> "except FooException as bar:"

This is taken care by 2to3, however.


Relative imports
----------------

The new relative import syntax,

    from . import foo

is not available on Py2.4, so we can't simply use it.

Using absolute imports everywhere is probably OK, if they just happen
to work.

2to3, however, converts the old syntax to new syntax, so as long as we
use the converter, it takes care of most parts.


Print
-----

The Print statement changed to a builtin function in Py3.

Also this is taken care of by 2to3.

types module
------------

The following items were removed from `types` module in Py3:

- StringType    (Py3: `bytes` is equivalent, to some degree)
- InstanceType  (Py3: ???)
- IntType       (Py3: no equivalent)
- LongType      (Py3: equivalent `long`)
- FloatType     (Py3: equivalent `float`)
- BooleanType   (Py3: equivalent `bool`)
- ComplexType   (Py3: equivalent `complex`)
- UnicodeType   (Py3: equivalent `str`)
- BufferType    (Py3: more-or-less equivalent `memoryview`)

In ``numerictypes.py``, the "common" types were replaced by their
plain equivalents, and `IntType` was dropped.


numpy.core.numerictypes
-----------------------

In numerictypes, types on Python 3 were changed so that:

===========   ============
Scalar type   Value
===========   ============
str_          This is the basic Unicode string type on Py3
bytes_        This is the basic Byte-string type on Py3
string_       bytes_ alias
unicode_      str_ alias
===========   ============


numpy.loadtxt et al
-------------------

These routines are difficult to duck-type to read both Unicode and
Bytes input.

I assumed they are meant for reading Bytes streams -- this is probably
the far more common use case with scientific data.


Cyclic imports
--------------

Python 3 is less forgiving about cyclic imports than Python 2.  Cycles
need to be broken to have the same code work both on Python 2 and 3.
{'fraction_non_alphanumeric': 0.14377862952229598, 'fraction_numerical': 0.01551836963634664, 'mean_word_length': 3.338199513381995, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5341882', 'n_tokens_mistral': 3138, 'n_tokens_neox': 2912, 'n_words': 1364}