<commit_before>==========
Statistics
==========

Common descriptive and aggregate statistics are included with the core agate library. For additional statistical methods beyond the scope of agate consider using the `agate-stats <http://agate-stats.rtfd.org/>`_ extension or integrating with `scipy <http://www.scipy.org/>`_.

Descriptive statistics
======================

agate includes a full set of standard descriptive statistics that can be applied to any column containing :class:`.Number` data.

.. code-block:: python

    table.aggregate(Sum('salary'))
    table.aggregate(Min('salary'))
    table.aggregate(Max('salary'))
    table.aggregate(Mean('salary'))
    table.aggregate(Median('salary'))
    table.aggregate(Mode('salary'))
    table.aggregate(Variance('salary'))
    table.aggregate(StDev('salary'))
    table.aggregate(MAD('salary'))

Or, get several at once:

.. code-block:: python

    table.aggregate([
        Min('salary'),
        Mean('salary'),
        Max('salary')
    ])

Aggregate statistics
====================

You can also generate aggregate statistics for subsets of data (sometimes referred to as "rolling up"):

.. code-block:: python

    doctors = patients.group_by('doctor')
    patient_ages = doctors.aggregate([
        ('patient_count', agate.Count())
        ('age_mean', agate.Mean('age')),
        ('age_median', agate.Median('age'))
    ])

The resulting table will have four columns: ``doctor``, ``patient_count``, ``age_mean`` and ``age_median``.


Distribution by count (frequency)
=================================

Counting the number of each unique value in a column can be accomplished with the :meth:`.Table.pivot` method:

.. code-block:: python

    # Counts of a single column's values
    table.pivot('doctor')

    # Counts of all combinations of more than one column's values
    table.pivot(['doctor', 'hospital'])

The resulting tables will have a column for each key column and another :code:`Count` column counting the number of instances of each value.

Distribution by percent
=======================

:meth:`.Table.pivot` can also be used to calculate the distribution of values as a percentage of the total number:

.. code-block:: python

    # Percents of a single column's values
    table.pivot('doctor', computation=Percent('Count'))

    # Percents of all combinations of more than one column's values
    table.pivot(['doctor', 'hospital'], computation=Percent('Count'))

The output table will be the same format as the previous example, except the value column will be named :code:`Percent`.

Identify outliers
=================

The `agate-stats <http://agate-stats.rtfd.org/>`_ extension adds methods for finding outliers.

.. code-block:: python

    import agatestats
<commit_msg>Added an 'agate.' prefix to examples for consistency<commit_after>==========
Statistics
==========

Common descriptive and aggregate statistics are included with the core agate library. For additional statistical methods beyond the scope of agate consider using the `agate-stats <http://agate-stats.rtfd.org/>`_ extension or integrating with `scipy <http://www.scipy.org/>`_.

Descriptive statistics
======================

agate includes a full set of standard descriptive statistics that can be applied to any column containing :class:`.Number` data.

.. code-block:: python

    table.aggregate(agate.Sum('salary'))
    table.aggregate(agate.Min('salary'))
    table.aggregate(agate.Max('salary'))
    table.aggregate(agate.Mean('salary'))
    table.aggregate(agate.Median('salary'))
    table.aggregate(agate.Mode('salary'))
    table.aggregate(agate.Variance('salary'))
    table.aggregate(agate.StDev('salary'))
    table.aggregate(agate.MAD('salary'))

Or, get several at once:

.. code-block:: python

    table.aggregate([
        agate.Min('salary'),
        agate.Mean('salary'),
        agate.Max('salary')
    ])

Aggregate statistics
====================

You can also generate aggregate statistics for subsets of data (sometimes referred to as "rolling up"):

.. code-block:: python

    doctors = patients.group_by('doctor')
    patient_ages = doctors.aggregate([
        ('patient_count', agate.Count())
        ('age_mean', agate.Mean('age')),
        ('age_median', agate.Median('age'))
    ])

The resulting table will have four columns: ``doctor``, ``patient_count``, ``age_mean`` and ``age_median``.


Distribution by count (frequency)
=================================

Counting the number of each unique value in a column can be accomplished with the :meth:`.Table.pivot` method:

.. code-block:: python

    # Counts of a single column's values
    table.pivot('doctor')

    # Counts of all combinations of more than one column's values
    table.pivot(['doctor', 'hospital'])

The resulting tables will have a column for each key column and another :code:`Count` column counting the number of instances of each value.

Distribution by percent
=======================

:meth:`.Table.pivot` can also be used to calculate the distribution of values as a percentage of the total number:

.. code-block:: python

    # Percents of a single column's values
    table.pivot('doctor', computation=agate.Percent('Count'))

    # Percents of all combinations of more than one column's values
    table.pivot(['doctor', 'hospital'], computation=agate.Percent('Count'))

The output table will be the same format as the previous example, except the value column will be named :code:`Percent`.

Identify outliers
=================

The `agate-stats <http://agate-stats.rtfd.org/>`_ extension adds methods for finding outliers.

.. code-block:: python

    import agatestats

{'fraction_non_alphanumeric': 0.15802512829587684, 'fraction_numerical': 0.0, 'mean_word_length': 4.307042253521127, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2857338', 'n_tokens_mistral': 1640, 'n_tokens_neox': 1516, 'n_words': 533}