<commit_before>..  Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

..    http://www.apache.org/licenses/LICENSE-2.0

..  Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.


====================
Options and settings
====================
.. currentmodule:: pyspark.pandas

Pandas API on Spark has an options system that lets you customize some aspects of its behaviour,
display-related options being those the user is most likely to adjust.

Options have a full "dotted-style", case-insensitive name (e.g. ``display.max_rows``).
You can get/set options directly as attributes of the top-level ``options`` attribute:


.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.options.display.max_rows
   1000
   >>> ps.options.display.max_rows = 10
   >>> ps.options.display.max_rows
   10

The API is composed of 3 relevant functions, available directly from the ``pandas_on_spark``
namespace:

* :func:`get_option` / :func:`set_option` - get/set the value of a single option.
* :func:`reset_option` - reset one or more options to their default value.

**Note:** Developers can check out `pyspark.pandas/config.py <https://github.com/apache/spark/blob/master/python/pyspark/pandas/config.py>`_ for more information.

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option("display.max_rows")
   1000
   >>> ps.set_option("display.max_rows", 101)
   >>> ps.get_option("display.max_rows")
   101


Getting and setting options
---------------------------

As described above, :func:`get_option` and :func:`set_option`
are available from the ``pandas_on_spark`` namespace.  To change an option, call
``set_option('option name', new_value)``.

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option('compute.max_rows')
   1000
   >>> ps.set_option('compute.max_rows', 2000)
   >>> ps.get_option('compute.max_rows')
   2000

All options also have a default value, and you can use ``reset_option`` to do just that:

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.reset_option("display.max_rows")

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option("display.max_rows")
   1000
   >>> ps.set_option("display.max_rows", 999)
   >>> ps.get_option("display.max_rows")
   999
   >>> ps.reset_option("display.max_rows")
   >>> ps.get_option("display.max_rows")
   1000

``option_context`` context manager has been exposed through
the top-level API, allowing you to execute code with given option values. Option values
are restored automatically when you exit the `with` block:

.. code-block:: python

   >>> with ps.option_context("display.max_rows", 10, "compute.max_rows", 5):
   ...    print(ps.get_option("display.max_rows"))
   ...    print(ps.get_option("compute.max_rows"))
   10
   5
   >>> print(ps.get_option("display.max_rows"))
   >>> print(ps.get_option("compute.max_rows"))
   1000
   1000


Operations on different DataFrames
----------------------------------

Pandas API on Spark disallows the operations on different DataFrames (or Series) by default to prevent expensive
operations. It internally performs a join operation which can be expensive in general.

This can be enabled by setting `compute.ops_on_diff_frames` to `True` to allow such cases.
See the examples below.

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.ops_on_diff_frames', True)
    >>> psdf1 = ps.range(5)
    >>> psdf2 = ps.DataFrame({'id': [5, 4, 3]})
    >>> (psdf1 - psdf2).sort_index()
        id
    0 -5.0
    1 -3.0
    2 -1.0
    3  NaN
    4  NaN
    >>> ps.reset_option('compute.ops_on_diff_frames')

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.ops_on_diff_frames', True)
    >>> psdf = ps.range(5)
    >>> psser_a = ps.Series([1, 2, 3, 4])
    >>> # 'psser_a' is not from 'psdf' DataFrame. So it is considered as a Series not from 'psdf'.
    >>> psdf['new_col'] = psser_a
    >>> psdf
       id  new_col
    0   0      1.0
    1   1      2.0
    3   3      4.0
    2   2      3.0
    4   4      NaN
    >>> ps.reset_option('compute.ops_on_diff_frames')


Default Index type
------------------

In pandas API on Spark, the default index is used in several cases, for instance,
when Spark DataFrame is converted into pandas-on-Spark DataFrame. In this case, internally pandas API on Spark attaches a
default index into pandas-on-Spark DataFrame.

There are several types of the default index that can be configured by `compute.default_index_type` as below:

**sequence**: It implements a sequence that increases one by one, by PySpark's Window function without
specifying partition. Therefore, it can end up with whole partition in single node.
This index type should be avoided when the data is large. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.default_index_type', 'sequence')
    >>> psdf = ps.range(3)
    >>> ps.reset_option('compute.default_index_type')
    >>> psdf.index
    Int64Index([0, 1, 2], dtype='int64')

This is conceptually equivalent to the PySpark example as below:

.. code-block:: python

    >>> from pyspark.sql import functions as F, Window
    >>> import pyspark.pandas as ps
    >>> spark_df = ps.range(3).to_spark()
    >>> sequential_index = F.row_number().over(
    ...    Window.orderBy(F.monotonically_increasing_id().asc())) - 1
    >>> spark_df.select(sequential_index).rdd.map(lambda r: r[0]).collect()
    [0, 1, 2]

**distributed-sequence** (default): It implements a sequence that increases one by one, by group-by and
group-map approach in a distributed manner. It still generates the sequential index globally.
If the default index must be the sequence in a large dataset, this
index has to be used. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.default_index_type', 'distributed-sequence')
    >>> psdf = ps.range(3)
    >>> ps.reset_option('compute.default_index_type')
    >>> psdf.index
    Int64Index([0, 1, 2], dtype='int64')

This is conceptually equivalent to the PySpark example as below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> spark_df = ps.range(3).to_spark()
    >>> spark_df.rdd.zipWithIndex().map(lambda p: p[1]).collect()
    [0, 1, 2]

**distributed**: It implements a monotonically increasing sequence simply by using
PySpark's `monotonically_increasing_id` function in a fully distributed manner. The
values are indeterministic. If the index does not have to be a sequence that increases
one by one, this index should be used. Performance-wise, this index almost does not
have any penalty comparing to other index types. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps<commit_msg>Revert "[SPARK-34827][PYTHON][DOC] Remove outdated statements on distributed-sequence default index"<commit_after>..  Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

..    http://www.apache.org/licenses/LICENSE-2.0

..  Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.


====================
Options and settings
====================
.. currentmodule:: pyspark.pandas

Pandas API on Spark has an options system that lets you customize some aspects of its behaviour,
display-related options being those the user is most likely to adjust.

Options have a full "dotted-style", case-insensitive name (e.g. ``display.max_rows``).
You can get/set options directly as attributes of the top-level ``options`` attribute:


.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.options.display.max_rows
   1000
   >>> ps.options.display.max_rows = 10
   >>> ps.options.display.max_rows
   10

The API is composed of 3 relevant functions, available directly from the ``pandas_on_spark``
namespace:

* :func:`get_option` / :func:`set_option` - get/set the value of a single option.
* :func:`reset_option` - reset one or more options to their default value.

**Note:** Developers can check out `pyspark.pandas/config.py <https://github.com/apache/spark/blob/master/python/pyspark/pandas/config.py>`_ for more information.

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option("display.max_rows")
   1000
   >>> ps.set_option("display.max_rows", 101)
   >>> ps.get_option("display.max_rows")
   101


Getting and setting options
---------------------------

As described above, :func:`get_option` and :func:`set_option`
are available from the ``pandas_on_spark`` namespace.  To change an option, call
``set_option('option name', new_value)``.

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option('compute.max_rows')
   1000
   >>> ps.set_option('compute.max_rows', 2000)
   >>> ps.get_option('compute.max_rows')
   2000

All options also have a default value, and you can use ``reset_option`` to do just that:

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.reset_option("display.max_rows")

.. code-block:: python

   >>> import pyspark.pandas as ps
   >>> ps.get_option("display.max_rows")
   1000
   >>> ps.set_option("display.max_rows", 999)
   >>> ps.get_option("display.max_rows")
   999
   >>> ps.reset_option("display.max_rows")
   >>> ps.get_option("display.max_rows")
   1000

``option_context`` context manager has been exposed through
the top-level API, allowing you to execute code with given option values. Option values
are restored automatically when you exit the `with` block:

.. code-block:: python

   >>> with ps.option_context("display.max_rows", 10, "compute.max_rows", 5):
   ...    print(ps.get_option("display.max_rows"))
   ...    print(ps.get_option("compute.max_rows"))
   10
   5
   >>> print(ps.get_option("display.max_rows"))
   >>> print(ps.get_option("compute.max_rows"))
   1000
   1000


Operations on different DataFrames
----------------------------------

Pandas API on Spark disallows the operations on different DataFrames (or Series) by default to prevent expensive
operations. It internally performs a join operation which can be expensive in general.

This can be enabled by setting `compute.ops_on_diff_frames` to `True` to allow such cases.
See the examples below.

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.ops_on_diff_frames', True)
    >>> psdf1 = ps.range(5)
    >>> psdf2 = ps.DataFrame({'id': [5, 4, 3]})
    >>> (psdf1 - psdf2).sort_index()
        id
    0 -5.0
    1 -3.0
    2 -1.0
    3  NaN
    4  NaN
    >>> ps.reset_option('compute.ops_on_diff_frames')

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.ops_on_diff_frames', True)
    >>> psdf = ps.range(5)
    >>> psser_a = ps.Series([1, 2, 3, 4])
    >>> # 'psser_a' is not from 'psdf' DataFrame. So it is considered as a Series not from 'psdf'.
    >>> psdf['new_col'] = psser_a
    >>> psdf
       id  new_col
    0   0      1.0
    1   1      2.0
    3   3      4.0
    2   2      3.0
    4   4      NaN
    >>> ps.reset_option('compute.ops_on_diff_frames')


Default Index type
------------------

In pandas API on Spark, the default index is used in several cases, for instance,
when Spark DataFrame is converted into pandas-on-Spark DataFrame. In this case, internally pandas API on Spark attaches a
default index into pandas-on-Spark DataFrame.

There are several types of the default index that can be configured by `compute.default_index_type` as below:

**sequence**: It implements a sequence that increases one by one, by PySpark's Window function without
specifying partition. Therefore, it can end up with whole partition in single node.
This index type should be avoided when the data is large. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.default_index_type', 'sequence')
    >>> psdf = ps.range(3)
    >>> ps.reset_option('compute.default_index_type')
    >>> psdf.index
    Int64Index([0, 1, 2], dtype='int64')

This is conceptually equivalent to the PySpark example as below:

.. code-block:: python

    >>> from pyspark.sql import functions as F, Window
    >>> import pyspark.pandas as ps
    >>> spark_df = ps.range(3).to_spark()
    >>> sequential_index = F.row_number().over(
    ...    Window.orderBy(F.monotonically_increasing_id().asc())) - 1
    >>> spark_df.select(sequential_index).rdd.map(lambda r: r[0]).collect()
    [0, 1, 2]

**distributed-sequence** (default): It implements a sequence that increases one by one, by group-by and
group-map approach in a distributed manner. It still generates the sequential index globally.
If the default index must be the sequence in a large dataset, this
index has to be used.
Note that if more data are added to the data source after creating this index,
then it does not guarantee the sequential index. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> ps.set_option('compute.default_index_type', 'distributed-sequence')
    >>> psdf = ps.range(3)
    >>> ps.reset_option('compute.default_index_type')
    >>> psdf.index
    Int64Index([0, 1, 2], dtype='int64')

This is conceptually equivalent to the PySpark example as below:

.. code-block:: python

    >>> import pyspark.pandas as ps
    >>> spark_df = ps.range(3).to_spark()
    >>> spark_df.rdd.zipWithIndex().map(lambda p: p[1]).collect()
    [0, 1, 2]

**distributed**: It implements a monotonically increasing sequence simply by using
PySpark's `monotonically_increasing_id` function in a fully distributed manner. The
values are indeterministic. If the index does not have to be a sequence that increases
one by one, this index should be used. Performance-wise, this index almost does not
have any penalty comparing to other index types. See the example below:

.. code-block:: python

    >>> import pyspark.pandas as ps
{'fraction_non_alphanumeric': 0.12556762092793683, 'fraction_numerical': 0.017703191839420863, 'mean_word_length': 3.447175885279485, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 341, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6644362', 'n_tokens_mistral': 4759, 'n_tokens_neox': 4512, 'n_words': 1612}