<jupyter_start><jupyter_text>River reliability![CI](https://github.com/MaximLippeveld/riverreliability/workflows/CI/badge.svg)The riverreliability package provides tools for assessing [classifier calibration](https://en.wikipedia.org/wiki/Calibration_(statistics)In_classification).The package provides two calibration visualizations:1. the newly proposed **river reliability diagram**, and2. the confidence-reliability diagram.It also provides two calibration metrics:1. the Expected Calibration Error (ECE), and2. the **Posterior Expected balanced Accuracy-based Calibration Error (PEACE)**, which is derived from the river reliability diagram.This package accompanies a publication, which is currently under review.  Install Install the package with:`pip install riverreliability`  How to useBelow, we show some basic funtionality of the package. Please look at the notebooks for more examples and documentation.<jupyter_code>np.random.seed(42)<jupyter_output><empty_output><jupyter_text>We start of by generating a fake dataset for classification and splitting it in a train and test set.<jupyter_code>X, y = sklearn.datasets.make_classification(n_samples=5000, n_features=12, n_informative=3, n_classes=3)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.2, shuffle=True)<jupyter_output><empty_output><jupyter_text>For this example we use an SVM. We fit it on the training data and generate probabilities for the test set.<jupyter_code>logreg = sklearn.svm.SVC(probability=True)
logreg.fit(X_train, y_train)
y_probs = logreg.predict_proba(X_test)<jupyter_output><empty_output><jupyter_text>As a sanity check we compute some performance metrics.<jupyter_code>print(f"Accuracy: {sklearn.metrics.accuracy_score(y_test, y_probs.argmax(axis=1))}")
print(f"Balanced accuracy: {sklearn.metrics.balanced_accuracy_score(y_test, y_probs.argmax(axis=1))}")<jupyter_output>Accuracy: 0.808
Balanced accuracy: 0.8084048918146675<jupyter_text>To get an insight into calibration we can look at the posterior reliability diagrams and the PEACE metric.We can plot the diagrams aggregated over all classes:<jupyter_code>ax = riverreliability.plots.river_reliability_diagram(y_probs.max(axis=1), y_probs.argmax(axis=1), y_test, bins="fd")

peace_metric = riverreliability.metrics.peace(y_probs.max(axis=1), y_probs.argmax(axis=1), y_test)
ax.set_title(f"PEACE: {peace_metric:.4f}")

_ = ax.legend()<jupyter_output><empty_output><jupyter_text>Or class-wise to spot miscalibrations for particular classes:<jupyter_code>import matplotlib.pyplot as plt
axes = riverreliability.plots.class_wise_river_reliability_diagram(y_probs, y_probs.argmax(axis=1), y_test, bins=15)

peace_metric = riverreliability.metrics.class_wise_error(y_probs, y_probs.argmax(axis=1), y_test, base_error=riverreliability.metrics.peace)
_ = plt.suptitle(f"PEACE: {peace_metric:.4f}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09849004804392587, 'fraction_numerical': 0.016815374056280027, 'mean_word_length': 8.342948717948717, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '452343', 'license': 'Apache-2.0', 'n_tokens_mistral': 1014, 'n_tokens_neox': 944, 'n_words': 221, 'path': 'notebooks/index.ipynb', 'repo_name': 'MaximLippeveld/riverreliability', 'size': '2914'}