<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  AutoML 05 : Blacklisting models, Early termination and handling missing dataIn this example we use the scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) to showcase how you can use AutoML for handling missing values in data. We also provide a stopping metric indicating a target for the primary metric so that AutoML can terminate the run without necessarly going through all the iterations. Finally, if you want to avoid a certain pipeline, we allow you to specify a black list of algos that AutoML will ignore for this run.Make sure you have executed the [00.configuration](00.configuration.ipynb) before running this notebook.In this notebook you would see1. Creating an Experiment using an existing Workspace2. Instantiating AutoMLConfig4. Training the Model5. Exploring the results6. Testing the fitted modelIn addition this notebook showcases the following features- **Blacklist** certain pipelines- Specify a **target metrics** to indicate stopping criteria- Handling **Missing Data** in the input  Create ExperimentAs part of the setup you have already created a Workspace. For AutoML you would need to create an Experiment. An Experiment is a named object in a Workspace, which is used to run experiments.<jupyter_code>import logging
import os
import random

from matplotlib import pyplot as plt
from matplotlib.pyplot import imshow
import numpy as np
import pandas as pd
from sklearn import datasets

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.train.automl import AutoMLConfig
from azureml.train.automl.run import AutoMLRun
ws = Workspace.from_config()

# choose a name for the experiment
experiment_name = 'automl-local-missing-data'
# project folder
project_folder = './sample_projects/automl-local-missing-data'

experiment=Experiment(ws, experiment_name)

output = {}
output['SDK version'] = azureml.core.VERSION
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Project Directory'] = project_folder
output['Experiment Name'] = experiment.name
pd.set_option('display.max_colwidth', -1)
pd.DataFrame(data=output, index=['']).T<jupyter_output><empty_output><jupyter_text>DiagnosticsOpt-in diagnostics for better experience, quality, and security of future releases<jupyter_code>from azureml.telemetry import set_diagnostics_collection
set_diagnostics_collection(send_diagnostics=True)<jupyter_output><empty_output><jupyter_text>Creating Missing Data<jupyter_code>from scipy import sparse

digits = datasets.load_digits()
X_digits = digits.data[10:,:]
y_digits = digits.target[10:]

# Add missing values in 75% of the lines
missing_rate = 0.75
n_missing_samples = int(np.floor(X_digits.shape[0] * missing_rate))
missing_samples = np.hstack((np.zeros(X_digits.shape[0] - n_missing_samples, dtype=np.bool), np.ones(n_missing_samples, dtype=np.bool)))
rng = np.random.RandomState(0)
rng.shuffle(missing_samples)
missing_features = rng.randint(0, X_digits.shape[1], n_missing_samples)
X_digits[np.where(missing_samples)[0], missing_features] = np.nan
df = pd.DataFrame(data=X_digits)
df['Label'] = pd.Series(y_digits, index=df.index)
df.head()<jupyter_output><empty_output><jupyter_text>Instantiate Auto ML ConfigThis defines the settings and data used to run the experiment.|Property|Description||-|-||**task**|classification or regression||**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics accuracyAUC_weightedbalanced_accuracyaverage_precision_score_weightedprecision_score_weighted||**max_time_sec**|Time limit in seconds for each iteration||**iterations**|Number of iterations. In each iteration Auto ML trains the data with a specific pipeline||**n_cross_validations**|Number of cross validation splits||**preprocess**| *True/False* Setting this to *True* enables Auto ML to perform preprocessing on the input to handle *missing data*, and perform some common *feature extraction*||**exit_score**|*double* value indicating the target for *primary_metric*.  Once the target is surpassed the run terminates||**blacklist_algos**|*Array* of *strings* indicating pipelines to ignore for Auto ML. Allowed values for **Classification**LogisticRegressionSGDClassifierWrapperNBWrapperBernoulliNBSVCWrapperLinearSVMWrapperKNeighborsClassifierDecisionTreeClassifierRandomForestClassifierExtraTreesClassifierLightGBMClassifierAllowed values for **Regression**ElasticNetGradientBoostingRegressorDecisionTreeRegressorKNeighborsRegressorLassoLarsSGDRegressorRandomForestRegressorExtraTreesRegressor||**X**|(sparse) array-like, shape = [n_samples, n_features]||**y**|(sparse) array-like, shape = [n_samples, ], [n_samples, n_classes]Multi-class targets. An indicator matrix turns on multilabel classification.  This should be an array of integers. ||**path**|Relative path to the project folder.  AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder. |<jupyter_code>automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             primary_metric = 'AUC_weighted',
                             max_time_sec = 3600,
                             iterations = 20,
                             n_cross_validations = 5,
                             preprocess = True,
                             exit_score = 0.994,
                             blacklist_algos = ['KNeighborsClassifier','LinearSVMWrapper'],
                             verbosity = logging.INFO,
                             X = X_digits, 
                             y = y_digits,
                             path=project_folder)<jupyter_output><empty_output><jupyter_text>Training the ModelYou can call the submit method on the experiment object and pass the run configuration. For Local runs the execution is synchronous. Depending on the data and number of iterations this can run for while.You will see the currently running iterations printing to the console.<jupyter_code>local_run = experiment.submit(automl_config, show_output=True)<jupyter_output><empty_output><jupyter_text>Exploring the results  Widget for monitoring runsThe widget will sit on "loading" until the first iteration completed, then you will see an auto-updating graph and table show up. It refreshed once per minute, so you should see the graph update as child runs complete.NOTE: The widget will display a link at the bottom. This will not currently work, but will eventually link to a web-ui to explore the individual run details.<jupyter_code>from azureml.train.widgets import RunDetails
RunDetails(local_run).show()<jupyter_output><empty_output><jupyter_text>Retrieve All Child RunsYou can also use sdk methods to fetch all the child runs and see individual metrics that we log.<jupyter_code>children = list(local_run.get_children())
metricslist = {}
for run in children:
    properties = run.get_properties()
    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    
    metricslist[int(properties['iteration'])] = metrics

rundata = pd.DataFrame(metricslist).sort_index(1)
rundata<jupyter_output><empty_output><jupyter_text>Retrieve the Best ModelBelow we select the best pipeline from our iterations. Each pipeline is a tuple of three elements. The first element is the score for the pipeline the second element is the string description of the pipeline and the last element are the pipeline objects used for each fold in the cross-validation.<jupyter_code>best_run, fitted_model = local_run.get_output()<jupyter_output><empty_output><jupyter_text>Best Model based on any other metric<jupyter_code># lookup_metric = "accuracy"
# best_run, fitted_model = local_run.get_output(metric=lookup_metric)<jupyter_output><empty_output><jupyter_text>Model from a specific iteration<jupyter_code># iteration = 3
# best_run, fitted_model = local_run.get_output(iteration=iteration)<jupyter_output><empty_output><jupyter_text>Register fitted model for deployment<jupyter_code>description = 'AutoML Model'
tags = None
local_run.register_model(description=description, tags=tags)
local_run.model_id # Use this id to deploy the model as a web service in Azure<jupyter_output><empty_output><jupyter_text>Testing the Fitted Model<jupyter_code>digits = datasets.load_digits()
X_digits = digits.data[:10, :]
y_digits = digits.target[:10]
images = digits.images[:10]

#Randomly select digits and test
for index in np.random.choice(len(y_digits), 2):
    print(index)
    predicted = fitted_model.predict(X_digits[index:index + 1])[0]
    label = y_digits[index]
    title = "Label value = %d  Predicted value = %d " % ( label,predicted)
    fig = plt.figure(1, figsize=(3,3))
    ax1 = fig.add_axes((0,0,.8,.8))
    ax1.set_title(title)
    plt.imshow(images[index], cmap=plt.cm.gray_r, interpolation='nearest')
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08290719181808291, 'fraction_numerical': 0.006310521162006311, 'mean_word_length': 5.21501014198783, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '101632', 'license': 'MIT', 'n_tokens_mistral': 2672, 'n_tokens_neox': 2567, 'n_words': 826, 'path': 'automl/05.auto-ml-missing-data-Blacklist-Early-Termination.ipynb', 'repo_name': 'dem108/MachineLearningNotebooks', 'size': '9191'}