<commit_before>"""
Function Reference
------------------

These functions cut hierarchical clusterings into flat clusterings
or find the roots of the forest formed by a cut by providing the flat
cluster ids of each observation.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|fcluster          |forms flat clusters from hierarchical clusters.  |
+------------------+-------------------------------------------------+
|fclusterdata      |forms flat clusters directly from data.          |
+------------------+-------------------------------------------------+
|leaders           |singleton root nodes for flat cluster.           |
+------------------+-------------------------------------------------+

These are routines for agglomerative clustering.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|linkage           |agglomeratively clusters original observations.  |
+------------------+-------------------------------------------------+
|single            |the single/min/nearest algorithm. (alias)        |
+------------------+-------------------------------------------------+
|complete          |the complete/max/farthest algorithm. (alias)     |
+------------------+-------------------------------------------------+
|average           |the average/UPGMA algorithm. (alias)             |
+------------------+-------------------------------------------------+
|weighted          |the weighted/WPGMA algorithm. (alias)            |
+------------------+-------------------------------------------------+
|centroid          |the centroid/UPGMC algorithm. (alias)            |
+------------------+-------------------------------------------------+
|median            |the median/WPGMC algorithm. (alias)              |
+------------------+-------------------------------------------------+
|ward              |the Ward/incremental algorithm. (alias)          |
+------------------+-------------------------------------------------+

These routines compute statistics on hierarchies.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|cophenet          |computes the cophenetic distance between leaves. |
+------------------+-------------------------------------------------+
|from_mlab_linkage |converts a linkage produced by MATLAB(TM).       |
+------------------+-------------------------------------------------+
|inconsistent      |the inconsistency coefficients for cluster.      |
+------------------+-------------------------------------------------+
|maxinconsts       |the maximum inconsistency coefficient for each   |
|                  |cluster.                                         |
+------------------+-------------------------------------------------+
|maxdists          |the maximum distance for each cluster.           |
+------------------+-------------------------------------------------+
|maxRstat          |the maximum specific statistic for each cluster. |
+------------------+-------------------------------------------------+
|to_mlab_linkage   |converts a linkage to one MATLAB(TM) can         |
|                  |understand.                                      |
+------------------+-------------------------------------------------+

Routines for visualizing flat clusters.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|dendrogram        |visualizes linkages (requires matplotlib).       |
+------------------+-------------------------------------------------+

These are data structures and routines for representing hierarchies as
tree objects.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|ClusterNode       |represents cluster nodes in a cluster hierarchy. |
+------------------+-------------------------------------------------+
|leaves_list       |a left-to-right traversal of the leaves.         |
+------------------+-------------------------------------------------+
|to_tree           |represents a linkage matrix as a tree object.    |
+------------------+-------------------------------------------------+

These are predicates for checking the validity of linkage and
inconsistency matrices, both condensed and redundant.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|is_valid_im       |checks for a valid inconsistency matrix.         |
+------------------+-------------------------------------------------+
|is_valid_linkage  |checks for a valid hierarchical clustering.      |
+------------------+-------------------------------------------------+
|is_isomorphic     |checks if two flat clusterings are isomorphic.   |
+------------------+-------------------------------------------------+
|is_monotonic      |checks if a linkage is monotonic.                |
+------------------+-------------------------------------------------+
|Z_y_correspond    |checks for validity of distance matrix given a   |
|                  |linkage.                                         |
+------------------+-------------------------------------------------+


* MATLAB and MathWorks are registered trademarks of The MathWorks, Inc.

* Mathematica is a registered trademark of The Wolfram Research, Inc.


References
----------

.. [Sta07] "Statistics toolbox." API Reference Documentation. The MathWorks.
   http://www.mathworks.com/access/helpdesk/help/toolbox/stats/.
   Accessed October 1, 2007.

.. [Mti07] "Hierarchical clustering." API Reference Documentation.
   The Wolfram Research, Inc.
   http://reference.wolfram.com/mathematica/HierarchicalClustering/tutorial/HierarchicalClustering.html.
   Accessed October 1, 2007.

.. [Gow69] Gower, JC and Ross, GJS. "Minimum Spanning Trees and Single Linkage
   Cluster Analysis." Applied Statistics. 18(1): pp. 54--64. 1969.

.. [War63] <NAME>, JH. "Hierarchical grouping to optimize an objective
   function." Journal of the American Statistical Association. 58(301):
   pp. 236--44. 1963.<commit_msg>Fixed heading in hierarchy documentation index. Added additional function in the index that didn't appear before.<commit_after>"""
Function Reference
------------------

These functions cut hierarchical clusterings into flat clusterings
or find the roots of the forest formed by a cut by providing the flat
cluster ids of each observation.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|fcluster          |forms flat clusters from hierarchical clusters.  |
+------------------+-------------------------------------------------+
|fclusterdata      |forms flat clusters directly from data.          |
+------------------+-------------------------------------------------+
|leaders           |singleton root nodes for flat cluster.           |
+------------------+-------------------------------------------------+

These are routines for agglomerative clustering.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|linkage           |agglomeratively clusters original observations.  |
+------------------+-------------------------------------------------+
|single            |the single/min/nearest algorithm. (alias)        |
+------------------+-------------------------------------------------+
|complete          |the complete/max/farthest algorithm. (alias)     |
+------------------+-------------------------------------------------+
|average           |the average/UPGMA algorithm. (alias)             |
+------------------+-------------------------------------------------+
|weighted          |the weighted/WPGMA algorithm. (alias)            |
+------------------+-------------------------------------------------+
|centroid          |the centroid/UPGMC algorithm. (alias)            |
+------------------+-------------------------------------------------+
|median            |the median/WPGMC algorithm. (alias)              |
+------------------+-------------------------------------------------+
|ward              |the Ward/incremental algorithm. (alias)          |
+------------------+-------------------------------------------------+

These routines compute statistics on hierarchies.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|cophenet          |computes the cophenetic distance between leaves. |
+------------------+-------------------------------------------------+
|from_mlab_linkage |converts a linkage produced by MATLAB(TM).       |
+------------------+-------------------------------------------------+
|inconsistent      |the inconsistency coefficients for cluster.      |
+------------------+-------------------------------------------------+
|maxinconsts       |the maximum inconsistency coefficient for each   |
|                  |cluster.                                         |
+------------------+-------------------------------------------------+
|maxdists          |the maximum distance for each cluster.           |
+------------------+-------------------------------------------------+
|maxRstat          |the maximum specific statistic for each cluster. |
+------------------+-------------------------------------------------+
|to_mlab_linkage   |converts a linkage to one MATLAB(TM) can         |
|                  |understand.                                      |
+------------------+-------------------------------------------------+

Routines for visualizing flat clusters.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|dendrogram        |visualizes linkages (requires matplotlib).       |
+------------------+-------------------------------------------------+

These are data structures and routines for representing hierarchies as
tree objects.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|ClusterNode       |represents cluster nodes in a cluster hierarchy. |
+------------------+-------------------------------------------------+
|leaves_list       |a left-to-right traversal of the leaves.         |
+------------------+-------------------------------------------------+
|to_tree           |represents a linkage matrix as a tree object.    |
+------------------+-------------------------------------------------+

These are predicates for checking the validity of linkage and
inconsistency matrices as well as for checking isomorphism of two
flat cluster assignments.

+------------------+-------------------------------------------------+
|*Function*        | *Description*                                   |
+------------------+-------------------------------------------------+
|is_valid_im       |checks for a valid inconsistency matrix.         |
+------------------+-------------------------------------------------+
|is_valid_linkage  |checks for a valid hierarchical clustering.      |
+------------------+-------------------------------------------------+
|is_isomorphic     |checks if two flat clusterings are isomorphic.   |
+------------------+-------------------------------------------------+
|is_monotonic      |checks if a linkage is monotonic.                |
+------------------+-------------------------------------------------+
|Z_y_correspond    |checks for validity of distance matrix given a   |
|                  |linkage.                                         |
+------------------+-------------------------------------------------+
|numobs_linkage    |the number of observations corresponding to a    |
|                  |linkage matrix.                                  |
+------------------+-------------------------------------------------+


* MATLAB and MathWorks are registered trademarks of The MathWorks, Inc.

* Mathematica is a registered trademark of The Wolfram Research, Inc.


References
----------

.. [Sta07] "Statistics toolbox." API Reference Documentation. The MathWorks.
   http://www.mathworks.com/access/helpdesk/help/toolbox/stats/.
   Accessed October 1, 2007.

.. [Mti07] "Hierarchical clustering." API Reference Documentation.
   The Wolfram Research, Inc.
   http://reference.wolfram.com/mathematica/HierarchicalClustering/tutorial/HierarchicalClustering.html.
   Accessed October 1, 2007.

.. [Gow69] Gower, JC and Ross, GJS. "Minimum Spanning Trees and Single Linkage
   Cluster Analysis." Applied Statistics. 18(1): pp. 54--64. 1969.

.. [War63] <NAME>, JH. "Hierarchical grouping to optimize an objective
   function." Journal of the American Statistical Association. 58(301):
   pp. 236--44. 1963.
{'fraction_non_alphanumeric': 0.4431305095678349, 'fraction_numerical': 0.006163549057550347, 'mean_word_length': 0.7052425760723451, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4091005', 'n_tokens_mistral': 2593, 'n_tokens_neox': 2269, 'n_words': 747}