<commit_before>.. doctest-skip-all

.. _whatsnew-5.0:

**************************
What's New in Astropy 5.0?
**************************

Overview
========

.. _whatsnew-5.0-cosmology-io:

Support for reading, writing, and converting ``Cosmology``
==========================================================

Four new methods -- ``read``, ``write``, ``to_format``, ``from_format`` -- have
been added to the ``Cosmology`` class, enabling reading from and writing to
files and converting between different python objects.
The methods use Astropy's Unified I/O registry so custom formats can be
registered. Details are discussed in an addition to the docs.

Currently no file formats are registered, but the syntax is as follows:

.. doctest-skip::

    >>> from astropy.cosmology import Planck18
    >>> Planck18.write('<file name>.<format>')
    >>>
    >>> from astropy.cosmology import Cosmology
    >>> cosmo = Cosmology.read('<file name>.<format>')
    >>> cosmo == Planck18
    True


The transformation between ``Cosmology`` and `dict` is pre-registered,
e.g. enabling::

    >>> from astropy.cosmology import Planck18
    >>> cm = Planck18.to_format("mapping")
    >>> cm
    {'cosmology': <class 'astropy.cosmology.flrw.FlatLambdaCDM'>,
     'name': 'Planck18',
     'H0': <Quantity 67.66 km / (Mpc s)>,
     'Om0': 0.30966,
     ...

    >>> from astropy.cosmology import Cosmology
    >>> cosmo = Cosmology.from_format(cm, format="mapping")
    >>> cosmo == Planck18
    True

.. _whatsnew-5.0-cosmology-units:

``Cosmology`` units module
==========================

A new module -- ``cosmology.units`` -- is added to the cosmology subpackage for
defining and collecting cosmological units and equivalencies.
The unit ``littleh`` and equivalency ``with_H0`` are deprecated from the main
``astropy.units`` subpackage and moved to ``cosmology.units``.
A new unit, ``redshift``, is added for tracking factors of cosmological redshift.
As this is a pseudo-unit an equivalency ``dimensionless_redshift`` is added
(and enabled by default) to allow for redshift - dimensionless conversions.
To convert between redshift and other cosmological distance measures, e.g.
CMB temperature, the equivalency ``with_redshift`` is also added.

    >>> import astropy.units as u
    >>> import astropy.cosmology.units as cu
    >>> z = 1100 * cu.redshift

    >>> z.to(u.dimensionless_unscaled)
    <Quantity 1100.>

    >>> from astropy.cosmology import WMAP9
    >>> equivalency = cu.with_redshift(WMAP9)  # construct equivalency
    >>> z.to(u.K, equivalency)
    <Quantity 3000.225 K>

    >>> z.to(u.km / u.s / u.Mpc, equivalency)
    <Quantity 1565637.40154275 km / (Mpc s)>

    >>> z.to(cu.littleh, equivalency)
    <Quantity 15656.37401543 littleh>

Further details are available in an addition to the docs.


.. _whatsnew-5.0-io-unified:

New Unified I/O architecture
============================

I/O registry submodule has switched to a class-based architecture, allowing for
the creation of custom registries. The three supported registry types are:

* read-only : ``astropy.io.registry.UnifiedInputRegistry``
* write-only : ``astropy.io.registry.UnifiedOutputRegistry``
* read/write : ``astropy.io.registry.UnifiedIORegistry``

For backward compatibility all the methods on the read/write have corresponding
module-level functions, which work with a default global read/write registry.


.. _whatsnew-5.0-modeling-new-models:

New Models
==========

The following models have now been added:

* :class:`~astropy.modeling.functional_models.Cosine1D`: a one-dimensional
  cosine model.
* :class:`~astropy.modeling.functional_models.Tangent1D`: a one-dimensional
  Tangent model.
* :class:`~astropy.modeling.functional_models.ArcSine1D`: a one-dimensional
  inverse sine model.
* :class:`~astropy.modeling.functional_models.ArcCosine1D`: a one-dimensional
  inverse cosine model.
* :class:`~astropy.modeling.functional_models.ArcTangent1D`: a one-dimensional
  inverse tangent model.

.. _whatsnew-5.0-dask-in-table:

Added support for dask arrays in tables
=======================================

`Dask arrays <https://docs.dask.org/en/stable/>`_ are now preserved instead
of being converted to Numpy arrays when added to tables:

.. doctest-requires:: dask

    >>> from astropy.table import Table
    >>> import dask.array as da
    >>> t = Table()
    >>> t['a'] = da.arange(1_000_000_000_000)
    >>> t
    <Table length=1000000000000>
         a
       int64
    ------------
               0
               1
               2
               3
               4
             ...
    999999999995
    999999999996
    999999999997
    999999999998
    999999999999
    >>> t['a'][100:200].compute()
    array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
           113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
           126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138,
           139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151,
           152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
           165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
           178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
           191, 192, 193, 194, 195, 196, 197, 198, 199])

.. _whatsnew-5.0-mixin-registry:

Added support for registering array-like objects as mixin columns
=================================================================

It is now possible to register functions (which we call mixin 'handlers') which
can be used by astropy tables to convert, on-the-fly, any array-like object into
a 'mixin' column that can be used in a table. As an example, this is used
internally to provide the seamless integration of dask arrays into tables, as
shown in :ref:`whatsnew-5.0-dask-in-table`.

For more information about how to write your own handlers and register them,
see :ref:`table_mixin_registry`.

Full change log
===============

To see a detailed list of all changes in version v5.0, including changes in
API, please see the :ref:`changelog`.


Renamed/removed functionality
=============================<commit_msg>Add Parquet support to 5.0 what's new.<commit_after>.. doctest-skip-all

.. _whatsnew-5.0:

**************************
What's New in Astropy 5.0?
**************************

Overview
========

.. _whatsnew-5.0-cosmology-io:

Support for reading, writing, and converting ``Cosmology``
==========================================================

Four new methods -- ``read``, ``write``, ``to_format``, ``from_format`` -- have
been added to the ``Cosmology`` class, enabling reading from and writing to
files and converting between different python objects.
The methods use Astropy's Unified I/O registry so custom formats can be
registered. Details are discussed in an addition to the docs.

Currently no file formats are registered, but the syntax is as follows:

.. doctest-skip::

    >>> from astropy.cosmology import Planck18
    >>> Planck18.write('<file name>.<format>')
    >>>
    >>> from astropy.cosmology import Cosmology
    >>> cosmo = Cosmology.read('<file name>.<format>')
    >>> cosmo == Planck18
    True


The transformation between ``Cosmology`` and `dict` is pre-registered,
e.g. enabling::

    >>> from astropy.cosmology import Planck18
    >>> cm = Planck18.to_format("mapping")
    >>> cm
    {'cosmology': <class 'astropy.cosmology.flrw.FlatLambdaCDM'>,
     'name': 'Planck18',
     'H0': <Quantity 67.66 km / (Mpc s)>,
     'Om0': 0.30966,
     ...

    >>> from astropy.cosmology import Cosmology
    >>> cosmo = Cosmology.from_format(cm, format="mapping")
    >>> cosmo == Planck18
    True

.. _whatsnew-5.0-cosmology-units:

``Cosmology`` units module
==========================

A new module -- ``cosmology.units`` -- is added to the cosmology subpackage for
defining and collecting cosmological units and equivalencies.
The unit ``littleh`` and equivalency ``with_H0`` are deprecated from the main
``astropy.units`` subpackage and moved to ``cosmology.units``.
A new unit, ``redshift``, is added for tracking factors of cosmological redshift.
As this is a pseudo-unit an equivalency ``dimensionless_redshift`` is added
(and enabled by default) to allow for redshift - dimensionless conversions.
To convert between redshift and other cosmological distance measures, e.g.
CMB temperature, the equivalency ``with_redshift`` is also added.

    >>> import astropy.units as u
    >>> import astropy.cosmology.units as cu
    >>> z = 1100 * cu.redshift

    >>> z.to(u.dimensionless_unscaled)
    <Quantity 1100.>

    >>> from astropy.cosmology import WMAP9
    >>> equivalency = cu.with_redshift(WMAP9)  # construct equivalency
    >>> z.to(u.K, equivalency)
    <Quantity 3000.225 K>

    >>> z.to(u.km / u.s / u.Mpc, equivalency)
    <Quantity 1565637.40154275 km / (Mpc s)>

    >>> z.to(cu.littleh, equivalency)
    <Quantity 15656.37401543 littleh>

Further details are available in an addition to the docs.


.. _whatsnew-5.0-io-unified:

New Unified I/O architecture
============================

I/O registry submodule has switched to a class-based architecture, allowing for
the creation of custom registries. The three supported registry types are:

* read-only : ``astropy.io.registry.UnifiedInputRegistry``
* write-only : ``astropy.io.registry.UnifiedOutputRegistry``
* read/write : ``astropy.io.registry.UnifiedIORegistry``

For backward compatibility all the methods on the read/write have corresponding
module-level functions, which work with a default global read/write registry.


.. _whatsnew-5.0-modeling-new-models:

New Models
==========

The following models have now been added:

* :class:`~astropy.modeling.functional_models.Cosine1D`: a one-dimensional
  cosine model.
* :class:`~astropy.modeling.functional_models.Tangent1D`: a one-dimensional
  Tangent model.
* :class:`~astropy.modeling.functional_models.ArcSine1D`: a one-dimensional
  inverse sine model.
* :class:`~astropy.modeling.functional_models.ArcCosine1D`: a one-dimensional
  inverse cosine model.
* :class:`~astropy.modeling.functional_models.ArcTangent1D`: a one-dimensional
  inverse tangent model.

.. _whatsnew-5.0-dask-in-table:

Added support for dask arrays in tables
=======================================

`Dask arrays <https://docs.dask.org/en/stable/>`_ are now preserved instead
of being converted to Numpy arrays when added to tables:

.. doctest-requires:: dask

    >>> from astropy.table import Table
    >>> import dask.array as da
    >>> t = Table()
    >>> t['a'] = da.arange(1_000_000_000_000)
    >>> t
    <Table length=1000000000000>
         a
       int64
    ------------
               0
               1
               2
               3
               4
             ...
    999999999995
    999999999996
    999999999997
    999999999998
    999999999999
    >>> t['a'][100:200].compute()
    array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
           113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
           126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138,
           139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151,
           152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
           165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
           178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
           191, 192, 193, 194, 195, 196, 197, 198, 199])

.. _whatsnew-5.0-mixin-registry:

Added support for registering array-like objects as mixin columns
=================================================================

It is now possible to register functions (which we call mixin 'handlers') which
can be used by astropy tables to convert, on-the-fly, any array-like object into
a 'mixin' column that can be used in a table. As an example, this is used
internally to provide the seamless integration of dask arrays into tables, as
shown in :ref:`whatsnew-5.0-dask-in-table`.

For more information about how to write your own handlers and register them,
see :ref:`table_mixin_registry`.

.. _whatsnew-5.0-parquet-tables:

Support for reading and writing tables to Parquet format
========================================================

.. _Parquet: https://parquet.apache.org/
.. _pyarrow: https://arrow.apache.org/docs/python/

The :ref:`table_io` now supports reading and writing files in the Parquet_ format if the pyarrow_ package is installed.
Apache Parquet is a columnar storage format related to the Hadoop ecosystem which supports a wide variety of data processing frameworks and programming languages.
A key benefit of Parquet files is that each column is stored independently, and thus reading a subset of columns is fast and efficient.
For more details see the :ref:`table_io_parquet` section.

Full change log
===============

To see a detailed list of all changes in version v5.0, including changes in
API, please see the :ref:`changelog`.


Renamed/removed functionality
=============================
{'fraction_non_alphanumeric': 0.1639960131871502, 'fraction_numerical': 0.0763627999693322, 'mean_word_length': 3.632102272727273, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 189, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '880584', 'n_tokens_mistral': 4862, 'n_tokens_neox': 3797, 'n_words': 1349}