<commit_before>Segmentation Evaluation using SegEval
=====================================

Text segmentation is the task of splitting up any amount of text into segments by placing boundaries between some atomic unit (e.g., morphemes, words, lines, sentences, paragraphs, sections, etc.).

E.g., if we were to perform both manual and automatic `syllabification <http://en.wikipedia.org/wiki/Syllabification>`_ of words, one may need a way to compare how close the automatic solution is to the manual one.  For this, we can use **Boundary Edit Distance** and **Boundary Similarity**.   `Evaluating a hypothetical automatic syllabifier <http://nbviewer.ipython.org/urls/raw.github.com/cfournie/segmentation.evaluation/master/examples/syllabification_evaluation.ipynb>`_, we can obtain the results shown below.

+--------------+-----------------+--------------------+------------------------+---------------------+
| Word         | Manual Solution | Automatic Solution | Boundary Edit Distance | Boundary Similarity |
+==============+=================+====================+========================+=====================+
| automatic    | au·to·ma·tic    | au·tom·a·tic       | 2 matches, 1 near      | 0.83                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| segmentation | seg·men·ta·tion | seg·ment·ation     | 1 match, 1 near, 1 miss| 0.50                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| is           | is              | is                 | No edits               | 1.00                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| fun          | fun             | f·un               | 1 miss                 | 0.00                |
+--------------+-----------------+--------------------+------------------------+---------------------+

This package is a collection of metrics and for comparing text segmentations and evaluating automatic text segmenters.  Both new (**Boundary Similarity**, **Segmentation Similarity**) and traditional (**WindowDiff**, **Pk**) are included, as well as inter-coder agreement coefficients and confusion matrices based upon a boundary edit distance.

:Release: |release|
:Date: |today|

Feature Support
---------------
A variety of segmentation comparison metrics are implemented, including:

* Boundary Edit Distance (BED; [Fournier2013]_)
* Boundary Similarity (B; [Fournier2013]_)
* BED-based confusion matrices (and precision/recall/F1; [Fournier2013]_)
* Segmentation Similarity (S; [FournierInkpen2012]_)
* WindowDiff [PevznerHearst2002]_
* Pk [BeefermanBerger1999]_

Additionally, B-based inter-coder agreement coefficients for segmentation that are suitable for 2 or more coders are provided, including:

* Fleiss' :math:`\pi` [Fleiss1971]_ (i.e., Siegel and Castellan's :math:`K` [SiegelCastellan1988]_)
* Fleiss' :math:`\kappa` [DaviesFleiss1982]_


User Guide
----------

This part of the documentation, which is mostly prose, begins with some
background information about Requests, then focuses on step-by-step
instructions for getting the most out of Requests.

.. toctree::<commit_msg>Added a link to the poetry examples.<commit_after>Segmentation Evaluation using SegEval
=====================================

Text segmentation is the task of splitting up any amount of text into segments by placing boundaries between some atomic unit (e.g., morphemes, words, lines, sentences, paragraphs, sections, etc.).

E.g., if we were to perform both manual and automatic `syllabification <http://en.wikipedia.org/wiki/Syllabification>`_ of words, one may need a way to compare how close the automatic solution is to the manual one.  For this, we can use **Boundary Edit Distance** and **Boundary Similarity**.   `Evaluating a hypothetical automatic syllabifier <http://nbviewer.ipython.org/urls/raw.github.com/cfournie/segmentation.evaluation/master/examples/syllabification_evaluation.ipynb>`_, we can obtain the results shown below.

+--------------+-----------------+--------------------+------------------------+---------------------+
| Word         | Manual Solution | Automatic Solution | Boundary Edit Distance | Boundary Similarity |
+==============+=================+====================+========================+=====================+
| automatic    | au·to·ma·tic    | au·tom·a·tic       | 2 matches, 1 near      | 0.83                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| segmentation | seg·men·ta·tion | seg·ment·ation     | 1 match, 1 near, 1 miss| 0.50                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| is           | is              | is                 | No edits               | 1.00                |
+--------------+-----------------+--------------------+------------------------+---------------------+
| fun          | fun             | f·un               | 1 miss                 | 0.00                |
+--------------+-----------------+--------------------+------------------------+---------------------+

This package is a collection of metrics and for comparing text segmentations and evaluating automatic text segmenters.  Both new (**Boundary Similarity**, **Segmentation Similarity**) and traditional (**WindowDiff**, **Pk**) are included, as well as inter-coder agreement coefficients and confusion matrices based upon a boundary edit distance.

For more example usage of ``segeval``, see "`An initial study of topical poetry segmentation <http://nbviewer.ipython.org/urls/raw.github.com/cfournie/initial.poetry.segmentation/master/khan_segmentation.ipynb>`_".

:Release: |release|
:Date: |today|

Feature Support
---------------
A variety of segmentation comparison metrics are implemented, including:

* Boundary Edit Distance (BED; [Fournier2013]_)
* Boundary Similarity (B; [Fournier2013]_)
* BED-based confusion matrices (and precision/recall/F1; [Fournier2013]_)
* Segmentation Similarity (S; [FournierInkpen2012]_)
* WindowDiff [PevznerHearst2002]_
* Pk [BeefermanBerger1999]_

Additionally, B-based inter-coder agreement coefficients for segmentation that are suitable for 2 or more coders are provided, including:

* Fleiss' :math:`\pi` [Fleiss1971]_ (i.e., Siegel and Castellan's :math:`K` [SiegelCastellan1988]_)
* Fleiss' :math:`\kappa` [DaviesFleiss1982]_


User Guide
----------

This part of the documentation, which is mostly prose, begins with some
background information about Requests, then focuses on step-by-step
instructions for getting the most out of Requests.

.. toctree::
{'fraction_non_alphanumeric': 0.27267311988086373, 'fraction_numerical': 0.016679076693968727, 'mean_word_length': 2.044424297370807, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2075488', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1530, 'n_words': 562}