<commit_before>.. _slep_018:

=======================================================
SLEP018: Pandas Output for Transformers with set_output
=======================================================

:Author: <NAME>
:Status: Draft
:Type: Standards Track
:Created: 2022-05-23

Abstract
--------

This SLEP proposes a ``set_output`` method to configure the output container of
scikit-learn transformers.

Detailed description
--------------------

Currently, scikit-learn transformers return NumPy ndarrays or SciPy sparse matrices.
This SLEP proposes adding a ``set_output`` method to configure a transformer to output
pandas DataFrames::

   scalar = StandardScalar().set_output(transform="pandas")
   scalar.fit(X_df)

   # X_trans_df is a pandas DataFrame
   X_trans_df = scalar.transform(X_df)

For a pipeline, calling ``set_output`` on the pipeline will configure
all steps in the pipeline::

   num_prep = make_pipeline(SimpleImputer(), StandardScalar(), PCA())
   num_preprocessor.set_output(transform="pandas")

   # X_trans_df is a pandas DataFrame
   X_trans_df = num_preprocessor.fit_transform(X_df)

By setting ``transform="pandas"`` calls to ``fit_transform`` will also return a
pandas DataFrame::

   num_prep = make_pipeline(
      SimpleImputer(),
      DependsOnPandasInputStandardScalar(),  # Depends on Pandas input to train
   )
   num_prep.set_output(transform="pandas")

   # Pipeline calls ``SimpleImputer.fit_transform`` returning a pandas DataFrame
   num_prep.fit(X_df)

Global Configuration
....................

This SLEP proposes a global configuration flag that sets the output for
all transformers::

   import sklearn
   sklearn.set_config(transform_output="pandas")

The global default configuration is ``"default"`` where the estimator determines
the output container.

Implementation
--------------

A prototype implementation was created to showcase different use cases for this SLEP,
which is seen in
`this rendered notebook <https://nbviewer.org/github/thomasjpfan/pandas-prototype-demo/blob/main/index.ipynb>`__
and
`this interactive notebook <https://colab.research.google.com/github/thomasjpfan/pandas-prototype-demo/blob/main/index.ipynb>`__.


Backward compatibility
----------------------

There are no backward compatibility concerns, because the ``set_output`` method
is a new API. Third party estimators can opt-in to the API by defining
``set_output``. The scikit-learn sparse container is backward compatible because
it is a subclass of SciPy's sparse matrix.

Alternatives
------------

Alternatives to this SLEP includes:

1. `SLEP014 <https://github.com/scikit-learn/enhancement_proposals/pull/37>`__
   proposes that if the input is a DataFrame than the output is a DataFrame.
2. :ref:`SLEP012 <slep_012>` proposes a custom scikit-learn container
   for dense and sparse data that contains feature names. This SLEP
   also proposes a custom container for sparse data, but pandas for dense data.
3. Prototype `#20100 <https://github.com/scikit-learn/scikit-learn/pull/20100>`__
   showcases ``array_out="pandas"`` in `transform`. This API
   is limited because does not directly support fitting on a pipeline where the
   steps requires data frames input.

Discussion
----------

A list of issues discussing Pandas output are:
`#14315 <https://github.com/scikit-learn/scikit-learn/pull/14315>`__,
`#20100 <https://github.com/scikit-learn/scikit-learn/pull/20100>`__, and
`#23001 <https://github.com/scikit-learn/scikit-learn/issueas/23001>`__.

Future Extensions
-----------------

Sparse Data
...........
<commit_msg>DOC Be more explicit about behavior<commit_after>.. _slep_018:

=======================================================
SLEP018: Pandas Output for Transformers with set_output
=======================================================

:Author: <NAME>
:Status: Draft
:Type: Standards Track
:Created: 2022-05-23

Abstract
--------

This SLEP proposes a ``set_output`` method to configure the output container of
scikit-learn transformers.

Detailed description
--------------------

Currently, scikit-learn transformers return NumPy ndarrays or SciPy sparse matrices.
This SLEP proposes adding a ``set_output`` method to configure a transformer to output
pandas DataFrames::

   scalar = StandardScalar().set_output(transform="pandas")
   scalar.fit(X_df)

   # X_trans_df is a pandas DataFrame
   X_trans_df = scalar.transform(X_df)

The index of the output DataFrame must match the index of the input.

For a pipeline, calling ``set_output`` on the pipeline will configure all steps in the
pipeline::

   num_prep = make_pipeline(SimpleImputer(), StandardScalar(), PCA())
   num_preprocessor.set_output(transform="pandas")

   # X_trans_df is a pandas DataFrame
   X_trans_df = num_preprocessor.fit_transform(X_df)

Meta-estimators that support ``set_output`` are required to configure all estimators
by calling ``set_output``.

Global Configuration
....................

This SLEP proposes a global configuration flag that sets the output for
all transformers::

   import sklearn
   sklearn.set_config(transform_output="pandas")

The global default configuration is ``"default"`` where the estimator determines
the output container.

Implementation
--------------

A prototype implementation was created to showcase different use cases for this SLEP,
which is seen in
`this rendered notebook <https://nbviewer.org/github/thomasjpfan/pandas-prototype-demo/blob/main/index.ipynb>`__
and
`this interactive notebook <https://colab.research.google.com/github/thomasjpfan/pandas-prototype-demo/blob/main/index.ipynb>`__.


Backward compatibility
----------------------

There are no backward compatibility concerns, because the ``set_output`` method
is a new API. Third party estimators can opt-in to the API by defining
``set_output``. Meta-estimators that define ``set_output`` to configure
it's inner estimators with ``set_output`` should error if any of the inner
estimators do not define ``set_output``.

Alternatives
------------

Alternatives to this SLEP includes:

1. `SLEP014 <https://github.com/scikit-learn/enhancement_proposals/pull/37>`__
   proposes that if the input is a DataFrame than the output is a DataFrame.
2. :ref:`SLEP012 <slep_012>` proposes a custom scikit-learn container
   for dense and sparse data that contains feature names. This SLEP
   also proposes a custom container for sparse data, but pandas for dense data.
3. Prototype `#20100 <https://github.com/scikit-learn/scikit-learn/pull/20100>`__
   showcases ``array_out="pandas"`` in `transform`. This API
   is limited because does not directly support fitting on a pipeline where the
   steps requires data frames input.

Discussion
----------

A list of issues discussing Pandas output are:
`#14315 <https://github.com/scikit-learn/scikit-learn/pull/14315>`__,
`#20100 <https://github.com/scikit-learn/scikit-learn/pull/20100>`__, and
`#23001 <https://github.com/scikit-learn/scikit-learn/issueas/23001>`__.

Future Extensions
-----------------

Sparse Data
...........

{'fraction_non_alphanumeric': 0.14971509971509972, 'fraction_numerical': 0.019373219373219373, 'mean_word_length': 4.121079504011671, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3473385', 'n_tokens_mistral': 2182, 'n_tokens_neox': 1983, 'n_words': 673}