<jupyter_start><jupyter_text>Example notebookThis notebook will cover a regression case using scikit-learn's *California Housing* dataset.<jupyter_code>from sklearn.datasets import fetch_california_housing
import pandas as pd

X, y = fetch_california_housing(data_home='miraiml_local', return_X_y=True)
data = pd.DataFrame(X)
data['target'] = y<jupyter_output>Downloading Cal. housing from https://ndownloader.figshare.com/files/5976036 to miraiml_local<jupyter_text>Let's split the data into training and testing data. In a real case scenario, we'd only have labels for training data.<jupyter_code>from sklearn.model_selection import train_test_split

train_data, test_data = train_test_split(data, test_size=0.2)<jupyter_output><empty_output><jupyter_text>Building the search spacesLet's compare (and ensemble) a `KNeighborsRegressor` and a pipeline composed by `StandardScaler` and a `LinearRegression`.<jupyter_code>from sklearn.neighbors import KNeighborsRegressor
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler

from miraiml import SearchSpace
from miraiml.pipeline import compose

Pipeline = compose(
    [('scaler', StandardScaler), ('lin_reg', LinearRegression)]
)

search_spaces = [
    SearchSpace(
        id='k-NN Regressor',
        model_class=KNeighborsRegressor,
        parameters_values=dict(
            n_neighbors=range(2, 9),
            weights=['uniform', 'distance'],
            p=range(2, 5)
        )
    ),
    SearchSpace(
        id='Pipeline',
        model_class=Pipeline,
        parameters_values=dict(
            scaler__with_mean=[True, False],
            scaler__with_std=[True, False],
            lin_reg__fit_intercept=[True, False]
        )
    )
]<jupyter_output><empty_output><jupyter_text>Configuring the EngineFor this demonstration, let's use `r2_score` to evaluate our modeling.<jupyter_code>from sklearn.metrics import r2_score

from miraiml import Config

config = Config(
    local_dir='miraiml_local',
    problem_type='regression',
    score_function=r2_score,
    search_spaces=search_spaces,
    ensemble_id='Ensemble'
)<jupyter_output><empty_output><jupyter_text>Triggering the EngineLet's also print the scores everytime the Engine finds a better solution.<jupyter_code>from miraiml import Engine

def on_improvement(status):
    scores = status.scores
    for key in sorted(scores.keys()):
        print('{}: {}'.format(key, round(scores[key], 3)), end='; ')
    print()

engine = Engine(config=config, on_improvement=on_improvement)<jupyter_output><empty_output><jupyter_text>Now we're ready to load the data.<jupyter_code>engine.load_train_data(train_data, 'target')
engine.load_test_data(test_data)<jupyter_output><empty_output><jupyter_text>Let's leave it running for 2 minutes, shuffle the train data, let it run for 2 more minutes and then interrupt it.<jupyter_code>from time import sleep

engine.restart()

sleep(120)

print('\nShuffling train data')
engine.shuffle_train_data(restart=True)

sleep(120)

engine.interrupt()<jupyter_output>Ensemble: 0.118; Pipeline: -3.214; k-NN Regressor: 0.118; 
Ensemble: 0.142; Pipeline: -3.214; k-NN Regressor: 0.142; 
Ensemble: 0.143; Pipeline: 0.467; k-NN Regressor: 0.142; 
Ensemble: 0.474; Pipeline: 0.467; k-NN Regressor: 0.142; 
Ensemble: 0.473; Pipeline: 0.467; k-NN Regressor: 0.172; 
Ensemble: 0.509; Pipeline: 0.503; k-NN Regressor: 0.172; 
Ensemble: 0.509; Pipeline: 0.503; k-NN Regressor: 0.172; 
Ensemble: 0.525; Pipeline: 0.503; k-NN Regressor: 0.321; 
Ensemble: 0.539; Pipeline: 0.503; k-NN Regressor: 0.321; 
Ensemble: 0.552; Pipeline: 0.503; k-NN Regressor: 0.521; 
Ensemble: 0.566; Pipeline: 0.503; k-NN Regressor: 0.521; 
Ensemble: 0.565; Pipeline: 0.503; k-NN Regressor: 0.538; 
Ensemble: 0.566; Pipeline: 0.503; k-NN Regressor: 0.538; 
Ensemble: 0.566; Pipeline: 0.503; k-NN Regressor: 0.538; 
Ensemble: 0.566; Pipeline: 0.512; k-NN Regressor: 0.538; 
Ensemble: 0.566; Pipeline: 0.512; k-NN Regressor: 0.538; 
Ensemble: 0.566; Pipeline: 0.512; k-NN Regressor: 0.538; 
Ensemble: 0.[...]<jupyter_text>Engine's status analysis<jupyter_code>status = engine.request_status()<jupyter_output><empty_output><jupyter_text>Let's see the status report.<jupyter_code>print(status.build_report(include_features=True))<jupyter_output>########################
best id: Ensemble
best score: 0.7583702712570008
########################
ensemble weights:
    k-NN Regressor: 0.4325346249356786
    Pipeline: 0.06615069839850787
########################
all scores:
    Ensemble: 0.7583702712570008
    k-NN Regressor: 0.7545806614607227
    Pipeline: 0.5963819838101254
########################
id: Pipeline
model class: MiraiPipeline
n features: 8
parameters:
    lin_reg__fit_intercept: True
    scaler__with_mean: False
    scaler__with_std: False
features: 0, 1, 2, 3, 4, 5, 6, 7
########################
id: k-NN Regressor
model class: KNeighborsRegressor
n features: 6
parameters:
    n_neighbors: 6
    p: 2
    weights: distance
features: 0, 2, 3, 5, 6, 7<jupyter_text>k-NN Regressor's historyHow does the k-NN Regressor's scores change with `n_neighbors`, on average?<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

knn_history = status.histories['k-NN Regressor']

knn_history[['n_neighbors__(hyperparameter)', 'score']]\
    .groupby('n_neighbors__(hyperparameter)').mean()\
    .plot.bar()

plt.show()<jupyter_output><empty_output><jupyter_text>We can also see how the presence of features (0 or 1) correlate with the score. These results can work as some sort of feature importance.<jupyter_code>knn_history[[col for col in knn_history if col.endswith('(feature)')] + ['score']]\
    .corr()['score'][:-1]\
    .sort_values()\
    .plot.bar(label='Correlation')

plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Theoretical performance in productionAgain, in practice we wouldn't have labels for `test_data`. But since we do have labels here, how would MiraiML perform on the test dataset?<jupyter_code>r2_score(test_data['target'], status.test_predictions['Ensemble'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13128171550171877, 'fraction_numerical': 0.0581109837944017, 'mean_word_length': 5.260245901639344, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '120061', 'license': 'MIT', 'n_tokens_mistral': 2429, 'n_tokens_neox': 2103, 'n_words': 510, 'path': 'docs/example.ipynb', 'repo_name': 'arthurpaulino/mirai_ml', 'size': '6109'}