<jupyter_start><jupyter_text>Simple Classification<jupyter_code>%load_ext autoreload
%autoreload 2
import numpy as np 
import matplotlib.pyplot as plt 
from matplotlib import cm
import seaborn as sns 

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV

from macest.classification import models as clmod
from macest.classification import plots as clplot

from plotting_utils import plot_prediction_conf_surface, plot_macest_sklearn_comparison_surface
sns.set_context("notebook")
sns.set_style('darkgrid')<jupyter_output><empty_output><jupyter_text>Plotting Functions  Generate some data and split it into 4 groups<jupyter_code>X,y = datasets.make_circles(n_samples= 10**4, noise = 0.4, factor =10**-1)

X_pp_train, X_conf_train, y_pp_train, y_conf_train  = train_test_split(X, y, test_size=0.66, random_state=10)
X_conf_train, X_cal, y_conf_train, y_cal = train_test_split(X_conf_train, y_conf_train,
                                                            test_size=0.5, random_state=0)
X_cal, X_test, y_cal,  y_test, = train_test_split(X_cal, y_cal, test_size=0.5, random_state=0)
plt.scatter(X_conf_train[:,0],X_conf_train[:,1], c=y_conf_train, cmap = cm.seismic)
plt.xlabel('x1', fontsize = 20)
plt.ylabel('x2', fontsize = 20)<jupyter_output><empty_output><jupyter_text>Train a point prediction model<jupyter_code>model = RandomForestClassifier(random_state =0,
                               n_estimators =800,
                               n_jobs =-1)

model.fit(X_pp_train, y_pp_train)
param_bounds = clmod.SearchBounds(alpha_bounds = (0, 500), k_bounds = (5,15))
neighbour_search_params = clmod.HnswGraphArgs(query_args = dict(ef = 1100))
macest_model = clmod.ModelWithConfidence(model,
                                       X_conf_train,
                                       y_conf_train)

macest_model.fit(X_cal, y_cal)<jupyter_output><empty_output><jupyter_text>Let's have a look at the performance of the model on unseen data, we will compare to the random forest confidence predictions<jupyter_code>preds = model.predict(X_test)
conf_preds = macest_model.predict_proba(X_test)
rf_conf_preds = model.predict_proba(X_test)
macest_point_prediction_conf = macest_model.predict_confidence_of_point_prediction(X_test) 
                              
rf_point_prediction_conf = np.amax(rf_conf_preds, axis=1)
clplot.plot_calibration_curve([rf_point_prediction_conf,
                               macest_point_prediction_conf], 
                              ['Random Forest', 'MACE'],
                              preds,
                              y_test)<jupyter_output><empty_output><jupyter_text>Let's compare calibration and forecast metrics<jupyter_code>clplot.plot_calibration_metrics([rf_point_prediction_conf, 
                                macest_point_prediction_conf], 
                              [ 'RF','MACE'], preds, y_test)
clplot.plot_forecast_metrics([rf_point_prediction_conf, 
                                macest_point_prediction_conf], 
                              [ 'RF','MACE'], preds, y_test)<jupyter_output><empty_output><jupyter_text>What does the surface learnt by the random forest  look like ?  As expected it is able to partition the space around the circles, it then extrapolates this surface and predicts anything outside of the inner circle as blue no matter how far we are from the data, this is probably what we want to happen. If we now look at the confidence plot we see that apart from the boundary, between the two circles the random forest is >90% confident. This high confident  is also extrapolated very far from the data, this is probably bad as we have no relevant data 20 standard deviations away from the data<jupyter_code>plot_prediction_conf_surface(3, 25, model, X_pp_train=X_pp_train, plot_training_data=False)<jupyter_output><empty_output><jupyter_text>Let's now compare those confidence estimates with MACEst  MACEst also learns to be very confident close to the data, however as we move further away from the data we see that the confidence drops off. At a certain point MACEst will return close an approximately uniform distribution, this suggests that we do not have the relevant data to be confident about the prediction<jupyter_code>plot_macest_sklearn_comparison_surface(3, 25, macest_model, model, X_pp_train=X_pp_train, plot_training_data=True)
plot_macest_sklearn_comparison_surface(3, 25, macest_model, model, X_pp_train=X_pp_train, plot_training_data=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06868905742145179, 'fraction_numerical': 0.012784398699891657, 'mean_word_length': 3.9795037756202807, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '235556', 'license': 'UPL-1.0', 'n_tokens_mistral': 1484, 'n_tokens_neox': 1408, 'n_words': 370, 'path': 'notebooks/classification/basic_classification.ipynb', 'repo_name': 'LaudateCorpus1/macest', 'size': '4615'}