<commit_before>.. _distributional_expectations:

================================================================================
Distributional Expectations
================================================================================

Distributional expectations help identify when new datasets or samples may be different than expected, and can help ensure that assumptions developed during exploratory analysis still hold as new data becomes available. You should use distributional expectations in the same way as other expectations: to provide a rough analogue to unit testing for software development, where the expectations can help to accelerate identification of risks as diverse as changes in a system being modeled or disruptions to a complex upstream data feed.

Great Expectations' Philosophy of Distributional Expectations
--------------------------------------------------------------------------------

Great Expectations attempts to provide a simple, expressive framework for describing distributional expectations. The framework generally adopts a nonparametric approach, although it is possible to build expectations from parameterized distributions.

The design is motivated by the following assumptions:

* Encoding expectations into a simple object that allows for portable data pipeline testing is the top priority. In many circumstances the loss of precision associated with "compressing" data into an expectation may be beneficial because of its intentional simplicity as well as because it adds a very light layer of obfuscation over the data which may align with privacy preservation goals.
* While it should be possible to easily extend the framework with more rigorous statistical tests, great expectations should provide simple, reasonable defaults. Care should be taken in cases where robust statistical guarantees are expected.


Distributional Expectations' Core Datatypes
--------------------------------------------------------------------------------

Distributional expectations currently focus exclusively on unidimensional (columnar) data. The core constructs of a great expectations distributional expectation are the partition and associated weights.

For continuous data:

* A partition is defined by an ordered list of points that define intervals on the real number line. Note that partition intervals do not need to be uniform. Currently, -inf and +inf are not explicitly supported as endpoints, so the term partition is used somewhat loosely.
* Partition weights define the probability of the associated interval. Note that this effectively applies a "piecewise uniform" distribution to the data for the purpose of statistical tests.

Example continuous partition object:
.. code-block:: python
  {
    "partition": [ 0, 1, 2, 10],
    "weights": [0.3, 0.3, 0.4]
  }

For discrete/categorical data:

* A partition defines the categorical values present in the data.
* Partition weights define the probability of the associated categorical value.

Example discrete partition object:
.. code-block:: python
  {
    "partition": [ "cat", "dog", "fish"],
    "weights": [0.3, 0.3, 0.4]
  }


Constructing Partition Objects
--------------------------------------------------------------------------------
Three convenience functions are available to easily construct partition objects from existing data:

* :func:`categorical_partition_data <great_expectations.dataset.util.categorical_partition_data>`
* :func:`kde_smooth_data <great_expectations.dataset.util.kde_smooth_data>`
* :func:`partition_data <great_expectations.dataset.util.partition_data>`

A convenience function s provided to validate that an object is a valid partition density object:

* :func:`is_valid_partition_object <great_expectations.dataset.util.is_valid_partition_object>`

Distributional Expectations Core Tests<commit_msg>Minor formatting tweak to distributional_expectations.rst<commit_after>.. _distributional_expectations:

================================================================================
Distributional Expectations
================================================================================

Distributional expectations help identify when new datasets or samples may be different than expected, and can help ensure that assumptions developed during exploratory analysis still hold as new data becomes available. You should use distributional expectations in the same way as other expectations: to provide a rough analogue to unit testing for software development, where the expectations can help to accelerate identification of risks as diverse as changes in a system being modeled or disruptions to a complex upstream data feed.

Great Expectations' Philosophy of Distributional Expectations
--------------------------------------------------------------------------------

Great Expectations attempts to provide a simple, expressive framework for describing distributional expectations. The framework generally adopts a nonparametric approach, although it is possible to build expectations from parameterized distributions.

The design is motivated by the following assumptions:

* Encoding expectations into a simple object that allows for portable data pipeline testing is the top priority. In many circumstances the loss of precision associated with "compressing" data into an expectation may be beneficial because of its intentional simplicity as well as because it adds a very light layer of obfuscation over the data which may align with privacy preservation goals.
* While it should be possible to easily extend the framework with more rigorous statistical tests, great expectations should provide simple, reasonable defaults. Care should be taken in cases where robust statistical guarantees are expected.


Distributional Expectations' Core Datatypes
--------------------------------------------------------------------------------

Distributional expectations currently focus exclusively on unidimensional (columnar) data. The core constructs of a great expectations distributional expectation are the partition and associated weights.

For continuous data:

* A partition is defined by an ordered list of points that define intervals on the real number line. Note that partition intervals do not need to be uniform. Currently, -inf and +inf are not explicitly supported as endpoints, so the term partition is used somewhat loosely.
* Partition weights define the probability of the associated interval. Note that this effectively applies a "piecewise uniform" distribution to the data for the purpose of statistical tests.

Example continuous partition object:

.. code-block:: python
  {
    "partition": [ 0, 1, 2, 10],
    "weights": [0.3, 0.3, 0.4]
  }

For discrete/categorical data:

* A partition defines the categorical values present in the data.
* Partition weights define the probability of the associated categorical value.

Example discrete partition object:

.. code-block:: python
  {
    "partition": [ "cat", "dog", "fish"],
    "weights": [0.3, 0.3, 0.4]
  }


Constructing Partition Objects
--------------------------------------------------------------------------------
Three convenience functions are available to easily construct partition objects from existing data:

* :func:`categorical_partition_data <great_expectations.dataset.util.categorical_partition_data>`
* :func:`kde_smooth_data <great_expectations.dataset.util.kde_smooth_data>`
* :func:`partition_data <great_expectations.dataset.util.partition_data>`

A convenience function s provided to validate that an object is a valid partition density object:

* :func:`is_valid_partition_object <great_expectations.dataset.util.is_valid_partition_object>`

Distributional Expectations Core Tests
{'fraction_non_alphanumeric': 0.14306022227965884, 'fraction_numerical': 0.004393900232618248, 'mean_word_length': 3.960897435897436, 'pattern_counts': {'":': 8, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '663548', 'n_tokens_mistral': 1627, 'n_tokens_neox': 1511, 'n_words': 851}