<jupyter_start><jupyter_text>In this demo, we will show how to use uq360 metrics for hyperparameters optimization using Lale and sklearn gridsearchCV.<jupyter_code>%matplotlib inline
import sys
sys.path.append("../../")
!pip install lale
!pip install 'liac-arff>=2.4.0'
from uq360.utils.misc import make_sklearn_compatible_scorer<jupyter_output><empty_output><jupyter_text>The uq360 metrics can be converted to a sklearn scorer usign `make_sklearn_compatible_scorer` utility function.  We will now show to how to convert ECE and AURRRC metric in uq360 into scorer that can be used with LALE framework for hyperparameter optimization.<jupyter_code>sklearn_aurrrc = make_sklearn_compatible_scorer(task_type="classification", metric="aurrrc", greater_is_better=False)
sklearn_ece = make_sklearn_compatible_scorer(task_type="classification", metric="ece", greater_is_better=False)
import lale
from lale.lib.lale import Hyperopt
lale.wrap_imported_operators()
from sklearn import datasets
X, y = datasets.load_breast_cancer(return_X_y=True)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
from lale.lib.sklearn import AdaBoostClassifier as Model
clf_ece = Hyperopt(estimator=Model, cv=3, max_evals=20, scoring=sklearn_ece, verbose=True)
clf_aurrrc = Hyperopt(estimator=Model, cv=3, max_evals=20, scoring=sklearn_aurrrc, verbose=True)
trained_with_ece = clf_ece.fit(X_train, y_train)
trained_with_aurrrc = clf_aurrrc.fit(X_train, y_train)
clf_accuracy = Hyperopt(estimator=Model, cv=3, max_evals=20, verbose=True)
trained_with_accuracy = clf_accuracy.fit(X_train, y_train)
from sklearn.metrics import classification_report
print(classification_report(y_test, trained_with_accuracy.predict(X_test)))
print(classification_report(y_test, trained_with_ece.predict(X_test)))
print(classification_report(y_test, trained_with_aurrrc.predict(X_test)))<jupyter_output>precision    recall  f1-score   support

           0       0.96      0.97      0.96        67
           1       0.98      0.98      0.98       121

    accuracy                           0.97       188
   macro avg       0.97      0.97      0.97       188
weighted avg       0.97      0.97      0.97       188<jupyter_text>Lets us now see how to use `uq360` models with sklearn's `GridsearchCV` We use the boston housing dataset with `QuantileRegression` model while optimizing the `picp` metric.<jupyter_code>from sklearn.model_selection import GridSearchCV
import pandas as pd
house_prices_dataset = datasets.load_boston()
house_prices_df = pd.DataFrame(house_prices_dataset['data'])
house_prices_df.columns = house_prices_dataset['feature_names']

all_features = ['RM','CRIM','PTRATIO', 'DIS']
X = house_prices_df[all_features].values
y = house_prices_dataset['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
base_config = {
    "alpha":0.95,
    "n_estimators":20,
    "max_depth":3,
    "learning_rate":0.1,
    "min_samples_leaf":20,
    "min_samples_split":20
}

configs  = {"config": []}
for num_estimators in [1, 2, 5, 10, 20, 30, 40, 50]:
    config = base_config.copy()
    config["n_estimators"] = num_estimators
    configs["config"].append(config)
from uq360.algorithms.quantile_regression import QuantileRegression
sklearn_picp = make_sklearn_compatible_scorer(task_type="regression", metric="picp", greater_is_better=True)
clf = GridSearchCV(QuantileRegression(config=base_config), configs, scoring=sklearn_picp)
clf.fit(X_train, y_train)
df = pd.DataFrame(clf.cv_results_)
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09660502346121998, 'fraction_numerical': 0.0328457079768148, 'mean_word_length': 5.517985611510792, 'pattern_counts': {'":': 7, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '494937', 'license': 'Apache-2.0', 'n_tokens_mistral': 1346, 'n_tokens_neox': 1259, 'n_words': 270, 'path': 'examples/autoai/demo_autoai.ipynb', 'repo_name': 'ZainNasrullah/UQ360', 'size': '3623'}