<jupyter_start><jupyter_text>Evaluating clustering algorithmsThe present library contains utilities for evaluating different clustering algorithms(with or without ground truth labels). On top of the evaluation utilities there are classes forperforming parameters sweeps and model selection. Here we give an overview of the most important functionality Before running the notebookInstall the library and its dependencies with, if you haven't done so already```pip install -e .```from the root directory. You can also execute this command directly in the notebook but will need to reload thekernel afterwards<jupyter_code># Note - this cell should be executed only once per session
%load_ext autoreload
%autoreload 2

import sys, os

# in order to get the config, it is not part of the library
os.chdir("..")
sys.path.append(os.path.abspath("."))
import numpy as np
import os
from pprint import pprint
from sklearn.cluster import DBSCAN
import seaborn as sns
import geopandas as gp
import matplotlib.pyplot as plt
import logging

from sensai.geoanalytics.coordinate_clustering import SkLearnCoordinateClustering
from sensai.hyperopt import GridSearch
from sensai.evaluation.evaluator_clustering import ClusteringModelSupervisedEvaluator, \
    ClusteringModelUnsupervisedEvaluator
from sensai.evaluation.eval_stats import ClusteringUnsupervisedEvalStats, ClusteringSupervisedEvalStats, \
    AdjustedMutualInfoScore
from sensai.geoanalytics.coordinate_clustering_ground_truth import PolygonAnnotatedCoordinates

from config import get_config

logging.basicConfig(level=logging.INFO)
# loading data and config
c  = get_config(reload=True)
sampleFile = c.datafile_path("sample", stage=c.RAW) # this can point to a directory or a shp/geojson file
coordinatesDF = gp.read_file(sampleFile)<jupyter_output><empty_output><jupyter_text>Evaluating a single modelFor a single model that was already fitted, evaluation statistics can be extracted with `ClusteringEvalStats`, see theexample below (the eval_stats object can also be used to retrieve evaluation results one by one)<jupyter_code>dbscan = SkLearnCoordinateClustering(DBSCAN(eps=150, min_samples=20))
dbscan.fit(coordinatesDF)
evalStats = ClusteringUnsupervisedEvalStats.fromModel(dbscan)
pprint(evalStats.getAll())
plt.hist(evalStats.clusterSizeDistribution)
plt.show()<jupyter_output><empty_output><jupyter_text>Model selectionFor model selection we need to compare different (or differently parametrized) models that weretrained on the same dataset. The `ClusteringEvaluator` abstraction was designed with this goal in mind.The evaluator can be used to obtain evaluation statistics for different models that are guaranteedto be comparable with each other (always computed by the same object in the same way). Here an example evaluatinga dbscan performance on metrics that don't necessitate ground truth labels<jupyter_code>modelEvaluator = ClusteringModelUnsupervisedEvaluator(coordinatesDF)

dbscanEvalStats = modelEvaluator.evalModel(dbscan, fit=False)  # dbscan was already fitted on this data
print("dbscan_performance: \n")
pprint(dbscanEvalStats.getAll())<jupyter_output><empty_output><jupyter_text>One of the main purposes of evaluators is to be used within classes that perform a parameter sweep, e.g.a `GridSearch`. All such objects return a data frame and (optionally but recommended!) persist all evaluation resultsin a csv.<jupyter_code>parameterOptions = {
    "min_samples": [10, 20],
    "eps": [50, 150]
}

# for running the grid search in multiple processes, all objects need to be picklable.
# Therefore we pass a named function as model factory instead of a lambda
def dbscanFactory(**kwargs):
    return SkLearnCoordinateClustering(DBSCAN(**kwargs))

dbscanGridSearch = GridSearch(dbscanFactory, parameterOptions, csvResultsPath=os.path.join(c.temp, "dbscanGridSearchCsv"))
# the results of the grid-search are saved as csv under the path provided above
resultDf = dbscanGridSearch.run(modelEvaluator, sortColumnName="numClusters", ascending=False)
resultDf.head()<jupyter_output><empty_output><jupyter_text>The resulting data frame can be used to visualize the results through standard techniques,e.g. pivoting and heatmaps<jupyter_code>print("calinskiHarabaszScores")
chScoreHeatmap = resultDf.pivot(index="min_samples", columns="eps", values="CalinskiHarabaszScore")
sns.heatmap(chScoreHeatmap, annot=True)
plt.show()
print("daviesBouldinScores")
chScoreHeatmap = resultDf.pivot(index="min_samples", columns="eps", values="DaviesBouldinScore")
sns.heatmap(chScoreHeatmap, cmap=sns.cm.rocket_r, annot=True)
plt.show()
print("numClusters")
numClustersHeatmap = resultDf.pivot(index="min_samples", columns="eps", values="numClusters").astype(int)
sns.heatmap(numClustersHeatmap, annot=True)  # something goes wrong with the datatype here, maybe b/c of zero clusters
plt.show()<jupyter_output><empty_output><jupyter_text>Dealing with ground truth labelsThe evaluation classes can take ground truth labels for all coordinates and use them for calculating related metrics.However, such labels are typically hard to come by, especially if the coordinates cover a large area. Therefore thelibrary includes utilities for extracting labels from ground truth provided in form of __cluster polygons in a selectedregion__. The central class for dealing with this kind of data is `ground_truth.PolygonAnnotatedCoordinates`,see examples below.<jupyter_code># The polygons can be read directly from a file, see the documentation for more details
groundTruthClusters = PolygonAnnotatedCoordinates(coordinatesDF, c.datafile_path("sample", stage=c.GROUND_TRUTH))<jupyter_output><empty_output><jupyter_text>As usual, the object has methods for plotting and exporting to geodata frames.These can be very useful for inspecting the provided data<jupyter_code>groundTruthClusters.plot(markersize=0.2, cmap="plasma")
plt.show()

groundTruthClusters.toGeoDF().head()<jupyter_output><empty_output><jupyter_text>We can extract the coordinates and labels for the annotated region and use them in evaluation. In the followingwe will train our own adaption of DBSCAN, namely `boundedDBSCAN` on datapoints in the ground truth region andevaluate the results against the true labels<jupyter_code>boundedDbscan = SkLearnCoordinateClustering(DBSCAN(eps=150, min_samples=20), minClusterSize=100)
groundTruthCoordinates, groundTruthLabels = groundTruthClusters.getCoordinatesLabels()
supervisedEvaluator = ClusteringModelSupervisedEvaluator(groundTruthCoordinates, trueLabels=groundTruthLabels)
supervisedEvalStats = supervisedEvaluator.evalModel(boundedDbscan)

print("Supervised evaluation metrics of bounded dbscan:")
pprint(supervisedEvalStats.getAll())
print("Unsupervised evaluation metrics of bounded dbscan:")
pprint(ClusteringUnsupervisedEvalStats(groundTruthCoordinates, groundTruthLabels).getAll())
print("")
print("Unsupervised evaluation metrics of annotated data")
pprint(ClusteringUnsupervisedEvalStats.fromModel(boundedDbscan).getAll())<jupyter_output><empty_output><jupyter_text>The bounded dbscan is performing quite OK with the given parameters, although we see that it segregates clusters toomuch and has a general tendency towards smaller clusters. These tendencies can be seen visually by comparing the groundtruth and the bounded dbscan cluster plots<jupyter_code>groundTruthClusters.plot(markersize=0.2, cmap="plasma", includeNoise=False)

boundedDbscan.plot(markersize=0.2, includeNoise=False)<jupyter_output><empty_output><jupyter_text>Supervised parameter estimationWe can now bring everything together by running a grid search and evaluating against ground truth. Very little codeis needed for that, so we will write it entirely in the cell below<jupyter_code>parameterOptions = {
    "min_samples": [19, 20, 21],
    "eps": [140, 150, 160]
}

supervisedGridSearch = GridSearch(dbscanFactory, parameterOptions,
                                           csvResultsPath=os.path.join(c.temp, "bounded_dbscan_grid_search.csv"))
# we will sort the results by mutual information store
supervisedResultDf = supervisedGridSearch.run(supervisedEvaluator, sortColumnName=AdjustedMutualInfoScore.name,
                                              ascending=False)
supervisedResultDf<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06858594411515664, 'fraction_numerical': 0.005322366033627676, 'mean_word_length': 7.011627906976744, 'pattern_counts': {'":': 4, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '516292', 'license': 'MIT', 'n_tokens_mistral': 2393, 'n_tokens_neox': 2242, 'n_words': 760, 'path': 'notebooks/Clustering Evaluation.ipynb', 'repo_name': 'schroedk/sensAI', 'size': '8267'}