<commit_before><commit_msg>adding notes from <NAME><commit_after>Hierarchical Clustering
=======================

This module adds new hierarchical clustering functionality to the
cluster package. Its interface is similar to the hierarchical
clustering functions provided in MATLAB(TM)'s Statistics Toolbox to
facilitate easier migration to the NumPy/SciPy framework. Linkage
methods implemented include single, complete, average, weighted,
centroid, median, and ward. Several functions are provided for
computing statistics on clusters including inconsistency statistics,
cophenetic distance, and maximum distance of descendants. The fcluster
and fclusterdata functions take hierarchical tree clusterings
generated by these algorithms, cuts the tree, and labels the flat
clusters. The leaders function finds the root of each flat cluster
given a hierarchical clustering and labellings of its leaves. Finally, a
matplotlib extension is provided for plotting dendrograms, which
may be outputted to postscript or any other supported format.

Spatial Package
===============

The new scipy.spatial package provides routines for distance computation
and kd-tree manipulation.

Distance Module
----------------

Provides a multitude of common distance functions useful for spatial
statistics, clustering, and kd-trees. Distance and dissimilarity
functions provided include Bray-Curtis, Canberra, Chebyshev, City
Block, Cosine, Dice, Euclidean, Hamming, Jaccard, Kulsinski,
Mahalanobis, Matching, Minkowski, Rogers-Tanimoto, Russell-Rao,
Squared Euclidean, Standardized Euclidean, Sokal-Michener,
Sokal-Sneath, and Yule. Two functions are provided for computing
distances between collections of vectors: pdist and cdist. pdist
is similar to the MATLAB(TM) function and computes pairwise distances
between a collection of vectors. cdist computes distances between
vectors in two sets of vectors. squareform is useful for converting
a square distance matrix to a condensed matrix and vice versa.

{'fraction_non_alphanumeric': 0.06237424547283702, 'fraction_numerical': 0.0, 'mean_word_length': 5.78839590443686, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6483905', 'n_tokens_mistral': 513, 'n_tokens_neox': 455, 'n_words': 245}