<jupyter_start><jupyter_text>Automated Machine Learning_**Regression with Aml Compute**_ Contents1. [Introduction](Introduction)1. [Setup](Setup)1. [Data](Data)1. [Train](Train)1. [Results](Results)1. [Test](Test)  IntroductionIn this example we use the Hardware Performance Dataset to showcase how you can use AutoML for a simple regression problem. The Regression goal is to predict the performance of certain combinations of hardware parts.After training AutoML models for this regression data set, we show how you can compute model explanations on your remote compute using a sample explainer script.If you are using an Azure Machine Learning Compute Instance, you are all set.  Otherwise, go through the [configuration](../../../configuration.ipynb)  notebook first if you haven't already to establish your connection to the AzureML Workspace. In this notebook you will learn how to:1. Create an `Experiment` in an existing `Workspace`.2. Instantiating AutoMLConfig with FeaturizationConfig for customization3. Train the model using remote compute.4. Explore the results and featurization transparency options5. Setup remote compute for computing the model explanations for a given AutoML model.6. Start an AzureML experiment on your remote compute to compute explanations for an AutoML model.7. Download the feature importance for engineered features and visualize the explanations for engineered features on azure portal. 8. Download the feature importance for raw features and visualize the explanations for raw features on azure portal.   SetupAs part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments.<jupyter_code>import logging

from matplotlib import pyplot as plt
import numpy as np
import pandas as pd

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace

from azureml.automl.core.featurization import FeaturizationConfig
from azureml.train.automl import AutoMLConfig
from azureml.core.dataset import Dataset<jupyter_output><empty_output><jupyter_text>This sample notebook may use features that are not available in previous versions of the Azure ML SDK.<jupyter_code>ws = Workspace.from_config()

# Choose a name for the experiment.
experiment_name = "automl-regression-hardware-explain"
experiment = Experiment(ws, experiment_name)

output = {}
output["Subscription ID"] = ws.subscription_id
output["Workspace Name"] = ws.name
output["Resource Group"] = ws.resource_group
output["Location"] = ws.location
output["Experiment Name"] = experiment.name
pd.set_option("display.max_colwidth", -1)
outputDf = pd.DataFrame(data=output, index=[""])
outputDf.T<jupyter_output><empty_output><jupyter_text>Create or Attach existing AmlComputeYou will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecturecompute-target) for your AutoML run. In this tutorial, you create `AmlCompute` as your training compute resource.> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.<jupyter_code>from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# Choose a name for your cluster.
amlcompute_cluster_name = "hardware-cluster"

# Verify that cluster does not exist already
try:
    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)
    print("Found existing cluster, use it.")
except ComputeTargetException:
    compute_config = AmlCompute.provisioning_configuration(
        vm_size="STANDARD_DS12_V2", max_nodes=4
    )
    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)
compute_target.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Setup Training and Test Data for AutoML experimentLoad the hardware dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model.  We also register the datasets in your workspace using a name so that these datasets may be accessed from the remote compute.<jupyter_code>data = "https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/machineData.csv"

dataset = Dataset.Tabular.from_delimited_files(data)

# Split the dataset into train and test datasets
train_data, test_data = dataset.random_split(percentage=0.8, seed=223)


# Register the train dataset with your workspace
train_data.register(
    workspace=ws,
    name="machineData_train_dataset",
    description="hardware performance training data",
    create_new_version=True,
)

# Register the test dataset with your workspace
test_data.register(
    workspace=ws,
    name="machineData_test_dataset",
    description="hardware performance test data",
    create_new_version=True,
)

label = "ERP"

train_data.to_pandas_dataframe().head()<jupyter_output><empty_output><jupyter_text>TrainInstantiate an `AutoMLConfig` object to specify the settings and data used to run the experiment.|Property|Description||-|-||**task**|classification, regression or forecasting||**primary_metric**|This is the metric that you want to optimize. Regression supports the following primary metrics: spearman_correlationnormalized_root_mean_squared_errorr2_scorenormalized_mean_absolute_error||**experiment_timeout_hours**| Maximum amount of time in hours that all iterations combined can take before the experiment terminates.||**enable_early_stopping**| Flag to enble early termination if the score is not improving in the short term.||**featurization**| 'auto' / 'off' / FeaturizationConfig Indicator for whether featurization step should be done automatically or not, or whether customized featurization should be used. Setting this enables AutoML to perform featurization on the input to handle *missing data*, and to perform some common *feature extraction*. Note: If the input data is sparse, featurization cannot be turned on.||**n_cross_validations**|Number of cross validation splits.||**training_data**|(sparse) array-like, shape = [n_samples, n_features]||**label_column_name**|(sparse) array-like, shape = [n_samples, ], targets values.|  CustomizationSupported customization includes:1. Column purpose update: Override feature type for the specified column.2. Transformer parameter update: Update parameters for the specified transformer. Currently supports Imputer and HashOneHotEncoder.3. Drop columns: Columns to drop from being featurized.4. Block transformers: Allow/Block transformers to be used on featurization process. Create FeaturizationConfig object using API calls<jupyter_code>featurization_config = FeaturizationConfig()
featurization_config.blocked_transformers = ["LabelEncoder"]
# featurization_config.drop_columns = ['MMIN']
featurization_config.add_column_purpose("MYCT", "Numeric")
featurization_config.add_column_purpose("VendorName", "CategoricalHash")
# default strategy mean, add transformer param for for 3 columns
featurization_config.add_transformer_params("Imputer", ["CACH"], {"strategy": "median"})
featurization_config.add_transformer_params(
    "Imputer", ["CHMIN"], {"strategy": "median"}
)
featurization_config.add_transformer_params(
    "Imputer", ["PRP"], {"strategy": "most_frequent"}
)
# featurization_config.add_transformer_params('HashOneHotEncoder', [], {"number_of_bits": 3})
automl_settings = {
    "enable_early_stopping": True,
    "experiment_timeout_hours": 0.25,
    "max_concurrent_iterations": 4,
    "max_cores_per_iteration": -1,
    "n_cross_validations": 5,
    "primary_metric": "normalized_root_mean_squared_error",
    "verbosity": logging.INFO,
}

automl_config = AutoMLConfig(
    task="regression",
    debug_log="automl_errors.log",
    compute_target=compute_target,
    featurization=featurization_config,
    training_data=train_data,
    label_column_name=label,
    **automl_settings,
)<jupyter_output><empty_output><jupyter_text>Call the `submit` method on the experiment object and pass the run configuration. Execution of local runs is synchronous. Depending on the data and the number of iterations this can run for a while.In this example, we specify `show_output = True` to print currently running iterations to the console.<jupyter_code>remote_run = experiment.submit(automl_config, show_output=False)<jupyter_output><empty_output><jupyter_text>Run the following cell to access previous runs. Uncomment the cell below and update the run_id.<jupyter_code># from azureml.train.automl.run import AutoMLRun
# remote_run = AutoMLRun(experiment=experiment, run_id='<run_ID_goes_here')
# remote_run
remote_run.wait_for_completion()
best_run, fitted_model = remote_run.get_output()
best_run_customized, fitted_model_customized = remote_run.get_output()<jupyter_output><empty_output><jupyter_text>TransparencyView updated featurization summary<jupyter_code>custom_featurizer = fitted_model_customized.named_steps["datatransformer"]
custom_featurizer.get_featurization_summary()<jupyter_output><empty_output><jupyter_text>is_user_friendly=False allows for more detailed summary for transforms being applied<jupyter_code>custom_featurizer.get_featurization_summary(is_user_friendly=False)
custom_featurizer.get_stats_feature_type_summary()<jupyter_output><empty_output><jupyter_text>Results  Widget for Monitoring RunsThe widget will first report a "loading" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details<jupyter_code>from azureml.widgets import RunDetails

RunDetails(remote_run).show()<jupyter_output><empty_output><jupyter_text>ExplanationsThis section will walk you through the workflow to compute model explanations for an AutoML model on your remote compute. Retrieve any AutoML Model for explanationsBelow we select an AutoML pipeline from our iterations. The `get_output` method returns the a AutoML run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for any logged `metric` or for a particular `iteration`.<jupyter_code># automl_run, fitted_model = remote_run.get_output(metric='r2_score')
automl_run, fitted_model = remote_run.get_output(iteration=2)<jupyter_output><empty_output><jupyter_text>Setup model explanation run on the remote computeThe following section provides details on how to setup an AzureML experiment to run model explanations for an AutoML model on your remote compute.  Sample script used for computing explanationsView the sample script for computing the model explanations for your AutoML model on remote compute.<jupyter_code>with open("train_explainer.py", "r") as cefr:
    print(cefr.read())<jupyter_output><empty_output><jupyter_text>Substitute values in your sample scriptThe following cell shows how you change the values in the sample script so that you can change the sample script according to your experiment and dataset.<jupyter_code>import shutil
import os

# create script folder
script_folder = "./sample_projects/automl-regression-hardware"
if not os.path.exists(script_folder):
    os.makedirs(script_folder)

# Copy the sample script to script folder.
shutil.copy("train_explainer.py", script_folder)

# Create the explainer script that will run on the remote compute.
script_file_name = script_folder + "/train_explainer.py"

# Open the sample script for modification
with open(script_file_name, "r") as cefr:
    content = cefr.read()

# Replace the values in train_explainer.py file with the appropriate values
content = content.replace(
    "<<experiment_name>>", automl_run.experiment.name
)  # your experiment name.
content = content.replace(
    "<<run_id>>", automl_run.id
)  # Run-id of the AutoML run for which you want to explain the model.
content = content.replace("<<target_column_name>>", "ERP")  # Your target column name
content = content.replace("<<task>>", "regression")  # Training task type
# Name of your training dataset register with your workspace
content = content.replace("<<train_dataset_name>>", "machineData_train_dataset")
# Name of your test dataset register with your workspace
content = content.replace("<<test_dataset_name>>", "machineData_test_dataset")

# Write sample file into your script folder.
with open(script_file_name, "w") as cefw:
    cefw.write(content)<jupyter_output><empty_output><jupyter_text>Create conda configuration for model explanations experiment from automl_run object<jupyter_code>from azureml.core.runconfig import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies
import pkg_resources

# create a new RunConfig object
conda_run_config = RunConfiguration(framework="python")

# Set compute target to AmlCompute
conda_run_config.target = compute_target
conda_run_config.environment.docker.enabled = True

# specify CondaDependencies obj
conda_run_config.environment.python.conda_dependencies = (
    automl_run.get_environment().python.conda_dependencies
)<jupyter_output><empty_output><jupyter_text>Submit the experiment for model explanationsSubmit the experiment with the above `run_config` and the sample script for computing explanations.<jupyter_code># Now submit a run on AmlCompute for model explanations
from azureml.core.script_run_config import ScriptRunConfig

script_run_config = ScriptRunConfig(
    source_directory=script_folder,
    script="train_explainer.py",
    run_config=conda_run_config,
)

run = experiment.submit(script_run_config)

# Show run details
run
%%time
# Shows output of the run on stdout.
run.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Feature importance  and  visualizing explanation dashboardIn this section we describe how you can download the explanation results from the explanations experiment and visualize the feature importance for your AutoML model on the azure portal.  Download engineered feature importance from artifact storeYou can use *ExplanationClient* to download the engineered feature explanations from the artifact store of the *automl_run*. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features.<jupyter_code>from azureml.interpret import ExplanationClient

client = ExplanationClient.from_run(automl_run)
engineered_explanations = client.download_model_explanation(
    raw=False, comment="engineered explanations"
)
print(engineered_explanations.get_feature_importance_dict())
print(
    "You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n"
    + automl_run.get_portal_url()
)<jupyter_output><empty_output><jupyter_text>Download raw feature importance from artifact storeYou can use *ExplanationClient* to download the raw feature explanations from the artifact store of the *automl_run*. You can also use azure portal url to view the dash board visualization of the feature importance values of the raw features.<jupyter_code>raw_explanations = client.download_model_explanation(
    raw=True, comment="raw explanations"
)
print(raw_explanations.get_feature_importance_dict())
print(
    "You can visualize the raw explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n"
    + automl_run.get_portal_url()
)<jupyter_output><empty_output><jupyter_text>OperationalizeIn this section we will show how you can operationalize an AutoML model and the explainer which was used to compute the explanations in the previous section. Register the AutoML model and the scoring explainerWe use the *TreeScoringExplainer* from *azureml-interpret* package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service.<jupyter_code># Register trained automl model present in the 'outputs' folder in the artifacts
original_model = automl_run.register_model(
    model_name="automl_model", model_path="outputs/model.pkl"
)
scoring_explainer_model = automl_run.register_model(
    model_name="scoring_explainer", model_path="outputs/scoring_explainer.pkl"
)<jupyter_output><empty_output><jupyter_text>Create the conda dependencies for setting up the serviceWe need to create the conda dependencies comprising of the *azureml* packages using the training environment from the *automl_run*.<jupyter_code>conda_dep = automl_run.get_environment().python.conda_dependencies

with open("myenv.yml", "w") as f:
    f.write(conda_dep.serialize_to_string())
with open("myenv.yml", "r") as f:
    print(f.read())<jupyter_output><empty_output><jupyter_text>View your scoring file<jupyter_code>with open("score_explain.py", "r") as f:
    print(f.read())<jupyter_output><empty_output><jupyter_text>Deploy the serviceIn the cell below, we deploy the service using the conda file and the scoring file from the previous steps.<jupyter_code>from azureml.core.webservice import Webservice
from azureml.core.model import InferenceConfig
from azureml.core.webservice import AciWebservice
from azureml.core.model import Model
from azureml.core.environment import Environment

aciconfig = AciWebservice.deploy_configuration(
    cpu_cores=2,
    memory_gb=2,
    tags={"data": "Machine Data", "method": "local_explanation"},
    description="Get local explanations for Machine test data",
)

myenv = Environment.from_conda_specification(name="myenv", file_path="myenv.yml")
inference_config = InferenceConfig(entry_script="score_explain.py", environment=myenv)

# Use configs and models generated above
service = Model.deploy(
    ws,
    "model-scoring",
    [scoring_explainer_model, original_model],
    inference_config,
    aciconfig,
)
service.wait_for_deployment(show_output=True)<jupyter_output><empty_output><jupyter_text>View the service logs<jupyter_code>service.get_logs()<jupyter_output><empty_output><jupyter_text>Inference using some test dataInference using some test data to see the predicted value from autml model, view the engineered feature importance for the predicted value and raw feature importance for the predicted value.<jupyter_code>if service.state == "Healthy":
    X_test = test_data.drop_columns([label]).to_pandas_dataframe()
    # Serialize the first row of the test data into json
    X_test_json = X_test[:1].to_json(orient="records")
    print(X_test_json)
    # Call the service to get the predictions and the engineered and raw explanations
    output = service.run(X_test_json)
    # Print the predicted value
    print(output["predictions"])
    # Print the engineered feature importances for the predicted value
    print(output["engineered_local_importance_values"])
    # Print the raw feature importances for the predicted value
    print(output["raw_local_importance_values"])<jupyter_output><empty_output><jupyter_text>Delete the serviceDelete the service once you have finished inferencing.<jupyter_code>service.delete()<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># preview the first 3 rows of the dataset

test_data = test_data.to_pandas_dataframe()
y_test = test_data["ERP"].fillna(0)
test_data = test_data.drop("ERP", 1)
test_data = test_data.fillna(0)


train_data = train_data.to_pandas_dataframe()
y_train = train_data["ERP"].fillna(0)
train_data = train_data.drop("ERP", 1)
train_data = train_data.fillna(0)
y_pred_train = fitted_model.predict(train_data)
y_residual_train = y_train - y_pred_train

y_pred_test = fitted_model.predict(test_data)
y_residual_test = y_test - y_pred_test
%matplotlib inline
from sklearn.metrics import mean_squared_error, r2_score

# Set up a multi-plot chart.
f, (a0, a1) = plt.subplots(
    1, 2, gridspec_kw={"width_ratios": [1, 1], "wspace": 0, "hspace": 0}
)
f.suptitle("Regression Residual Values", fontsize=18)
f.set_figheight(6)
f.set_figwidth(16)

# Plot residual values of training set.
a0.axis([0, 360, -100, 100])
a0.plot(y_residual_train, "bo", alpha=0.5)
a0.plot([-10, 360], [0, 0], "r-", lw=3)
a0.text(
    16,
    170,
    "RMSE = {0:.2f}".format(np.sqrt(mean_squared_error(y_train, y_pred_train))),
    fontsize=12,
)
a0.text(
    16, 140, "R2 score = {0:.2f}".format(r2_score(y_train, y_pred_train)), fontsize=12
)
a0.set_xlabel("Training samples", fontsize=12)
a0.set_ylabel("Residual Values", fontsize=12)

# Plot residual values of test set.
a1.axis([0, 90, -100, 100])
a1.plot(y_residual_test, "bo", alpha=0.5)
a1.plot([-10, 360], [0, 0], "r-", lw=3)
a1.text(
    5,
    170,
    "RMSE = {0:.2f}".format(np.sqrt(mean_squared_error(y_test, y_pred_test))),
    fontsize=12,
)
a1.text(5, 140, "R2 score = {0:.2f}".format(r2_score(y_test, y_pred_test)), fontsize=12)
a1.set_xlabel("Test samples", fontsize=12)
a1.set_yticklabels([])

plt.show()
%matplotlib inline
test_pred = plt.scatter(y_test, y_pred_test, color="")
test_test = plt.scatter(y_test, y_test, color="g")
plt.legend(
    (test_pred, test_test), ("prediction", "truth"), loc="upper left", fontsize=8
)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08223386951021146, 'fraction_numerical': 0.007319718055304536, 'mean_word_length': 6.37029637029637, 'pattern_counts': {'":': 17, '<': 114, '<?xml version=': 0, '>': 114, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '22430', 'license': 'MIT', 'n_tokens_mistral': 6599, 'n_tokens_neox': 6362, 'n_words': 2079, 'path': 'python-sdk/tutorials/automl-with-azureml/regression-explanation-featurization/auto-ml-regression-explanation-featurization.ipynb', 'repo_name': 'msftcoderdjw/azureml-examples', 'size': '22132'}