<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/responsible-ai/model-analysis/regression/azureml-model-analysis-regression.png)  Model analysis for regression scenarios**This notebook will demonstrate on how to compute Responsible AI insights like explanations, counterfactual examples, causal effects and error analysis on remote compute for a regression model.** Contents1. [Prerequisites](Prerequisites)1. [Dataset](Dataset)1. [Create or attach existing AmlCompute cluster](AmlCompute)1. [Train model on remote compute](Train)1. [Generate RAI insights](Generate)1. [Responsible AI dashboard](Dashboard)  Prerequisites Install azureml-responsibleai Please make sure that you have the latest pypi version of `azureml-responsibleai` installed in you environment. Otherwise you can execute `pip install --upgrade azureml-responsibleai` before running this notebook.<jupyter_code># !pip install azureml-responsibleai
import azureml.core
from azureml.core import Workspace, Experiment, Run
from azureml.core import Model
from azureml.core.dataset import Dataset
from azureml.core import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies

from responsibleai import ModelAnalysis
from azureml.responsibleai.common.pickle_model_loader import PickleModelLoader
from azureml.responsibleai.tools.model_analysis.model_analysis_config import ModelAnalysisConfig
from azureml.responsibleai.tools.model_analysis.model_analysis_run import ModelAnalysisRun
from azureml.responsibleai.tools.model_analysis.explain_config import ExplainConfig
from azureml.responsibleai.tools.model_analysis.causal_config import CausalConfig
from azureml.responsibleai.tools.model_analysis.counterfactual_config import CounterfactualConfig
from azureml.responsibleai.tools.model_analysis.error_analysis_config import ErrorAnalysisConfig<jupyter_output><empty_output><jupyter_text>This sample notebook may use features that are not available in previous versions of the Azure ML SDK.<jupyter_code>print("This notebook was created using version AZUREML-SDK-VERSION of the Azure ML SDK")
print("You are currently using version", azureml.core.VERSION, "of the Azure ML SDK")<jupyter_output><empty_output><jupyter_text>Link an AzureML workspaceTo use this notebook, an Azure Machine Learning workspace is required.Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.<jupyter_code>user_workspace = Workspace.from_config()
print('Workspace name: ' + user_workspace.name, 
      'Azure region: ' + user_workspace.location, 
      'Subscription id: ' + user_workspace.subscription_id, 
      'Resource group: ' + user_workspace.resource_group, sep = '\n')<jupyter_output><empty_output><jupyter_text>DatasetThis notebook uses the Boston housing dataset. Below we load the Boston Hosuing dataset and split the dataset into train and test sets.<jupyter_code>import sklearn
import pandas as pd
from sklearn.model_selection import train_test_split

data = sklearn.datasets.load_boston()
target_feature = 'y'
continuous_features = data.feature_names
data_df = pd.DataFrame(data.data, columns=data.feature_names)

X_train, X_test, y_train, y_test = train_test_split(data_df, data.target, test_size=0.2, random_state=7)

train_data = X_train.copy()
test_data = X_test.copy()
train_data[target_feature] = y_train
test_data[target_feature] = y_test<jupyter_output><empty_output><jupyter_text>Upload the train and test dataset to datastoreIn the cell below, we upload the train and test datasets to the default datastore and register the train data and test data as azureml datasets.<jupyter_code>from azureml.core import Dataset
datastore = user_workspace.get_default_datastore()

# Upload train data to datastore
train_name = 'boston_train'
train_datastore_path = (datastore, train_name)
train_dataset = Dataset.Tabular.register_pandas_dataframe(
    train_data, train_datastore_path, train_name)


# Upload test data to datastore
test_name = 'boston_test'
test_datastore_path = (datastore, test_name)
test_dataset = Dataset.Tabular.register_pandas_dataframe(
    test_data, test_datastore_path, test_name)

label = 'y'<jupyter_output><empty_output><jupyter_text>Identify continous and categorical featuresBelow we identify continous and categorical features in the above dataset.<jupyter_code>continuous_features = X_train.columns
categorical_features = []
feature_names = X_train.columns
continuous_features<jupyter_output><empty_output><jupyter_text>Create or attach existing AmlCompute clusterYou will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecturecompute-target) for training your model and computing RAI insights for the trained model. In this tutorial, you create `AmlCompute` as your training compute resource.> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota. Create cimpute cluster<jupyter_code>from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# Choose a name for your CPU cluster
cpu_cluster_name = "rai-cluster"

# Verify that cluster does not exist already
try:
    compute_target = ComputeTarget(workspace=user_workspace, name=cpu_cluster_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',
                                                           max_nodes=6)
    compute_target = ComputeTarget.create(user_workspace, cpu_cluster_name, compute_config)

compute_target.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Train model on remote computeIn this section, we train a simple regression model on the remote compute.Add `azureml-responsibleai` as a pip dependency in the run configuration.<jupyter_code>run_config = RunConfiguration(framework="python")
conda_dependencies = CondaDependencies.create()
run_config.environment.python.conda_dependencies = conda_dependencies
run_config.environment.python.conda_dependencies.add_pip_package("azureml-responsibleai=={}".format(azureml.core.VERSION))
run_config.target = compute_target
run_config<jupyter_output><empty_output><jupyter_text>Copy the train script into the script directory. This train script will be used to train the model and register the model on remote compute.<jupyter_code>import shutil
import os

# create script folder
script_folder = './sample_projects/regression-boston'
if not os.path.exists(script_folder):
    os.makedirs(script_folder)

# Copy the sample script to script folder.
shutil.copy('train.py', script_folder)

# Create the explainer script that will run on the remote compute.
script_file_name = script_folder + '/train.py'<jupyter_output><empty_output><jupyter_text>Submit the train script via `ScriptRunConfig` to train the model on remote compute.<jupyter_code># Now submit a run on AmlCompute for model explanations
from azureml.core.script_run_config import ScriptRunConfig

exp_name = "RAI-Regression-Boston"
experiment = Experiment(user_workspace, exp_name)


script_run_config = ScriptRunConfig(source_directory=script_folder,
                                    script='train.py',
                                    run_config=run_config)

run = experiment.submit(script_run_config)

# Show run details
run<jupyter_output><empty_output><jupyter_text>Wait for the above model training run to complete.<jupyter_code>run.wait_for_completion(raise_on_error=True, wait_post_processing=True)<jupyter_output><empty_output><jupyter_text>Generate RAI insightsThis section will walk you through the workflow to compute Responsible AI insights like model explanations, counterfactual examples, causal effects and error analysis using model analysis workflow on your remote compute for the model trained in the previous section.  Configure model analysis and submit RAI insight computation runsIn this section, we will demonstrate how to configure model analysis, submit the model analysis run and submit the individual RAI computations for explanations, counterfactual examples, error analysis and causal effects for your trained model Create ModelAnalysis configurationCreate `ModelAnalysisConfig` for computing the RAI insights for the trained model. The `ModelAnalysisConfig` requires the following:-1. The registered model which was registered during the model training.2. The train and test datasets.3. `confidential_datastore_name`which is the name of the datastore where the analyses will be uploaded.4. List of the feature column names by dropping the name of the label column from the list of all column names.5. List of categorical features.6. Azureml run configuration whcih was setup in the previous section.<jupyter_code>registered_model = Model.list(user_workspace, 'boston')[0]
model_loader = PickleModelLoader('boston.pkl')

train_dataset = Dataset.get_by_name(workspace=user_workspace, name='boston_train')
test_dataset = Dataset.get_by_name(workspace=user_workspace, name='boston_test')

ma = ModelAnalysisConfig(
    title="RAI Regression Boston", # The name to assign to this model analysis
    model=registered_model, # The register model in AzureML to analyse
    model_type='regression', # Type of model it is, 'classification' or 'regression'
    model_loader=model_loader, # The model loader module for loading the model. Specify 'mlflow' to load using mlflow.
    train_dataset=train_dataset, # The training dataset to use for this analysis.
    test_dataset=test_dataset, # The test dataset to use for this analysis.
    X_column_names=feature_names, # The names of the columns in the train dataset.
    target_column_name=label, # The name of the target column.
    confidential_datastore_name=user_workspace.get_default_datastore().name, # The name of the confidential datastore where the analyses will be uploaded.
    run_configuration=run_config, # The RunConfiguration specifying the compute on which this analysis will be computed.
    categorical_column_names=categorical_features # List of all categorical columns in the dataset.
)<jupyter_output><empty_output><jupyter_text>Submit Model Analysis runThe model analysis run takes a snapshot of the data in preparation for model explanation, error analysis, causal and counterfactual.The model analysis run is the parent run for the model explanation, error analysis, causal and counterfactual runs.<jupyter_code>experiment = Experiment(user_workspace, exp_name)

model_analysis_run = experiment.submit(ma)
model_analysis_run.wait_for_completion(raise_on_error=True,
                                       wait_post_processing=True)
model_analysis_run<jupyter_output><empty_output><jupyter_text>Submit run for explanationsRun model explanation based on the model analysis.The explanation run is a child run of the model analysis run.In the future, the `add_request` method will allow extra parameters to configure the explanation generated.<jupyter_code>ec = ExplainConfig(model_analysis_run, run_config)
ec.add_request(
    comment="Compute Explanations" # Comment to identify the explain configuration
)
explain_run = model_analysis_run.submit_child(ec)<jupyter_output><empty_output><jupyter_text>Submit run for error analysisRun error analysis based on the model analysis.The error analysis run is a child run of the model analysis run.<jupyter_code>ec = ErrorAnalysisConfig(model_analysis_run, run_config)
ec.add_request(
    max_depth=3, # The maximum depth of the error analysis tree
    comment="Compute ErrorAnalysis" # Comment to identify the error analysis configuration
)
error_analysis_run = model_analysis_run.submit_child(ec)<jupyter_output><empty_output><jupyter_text>Submit run for counterfactual examplesGenerate counterfactuals for all the samples in the `test_dataset` based on the model analysis.The counterfactual run is a child run of the model analysis run.You may use the `add_request` method that allows you to specify extra parameters to configure the counterfactual examples to be generated.<jupyter_code>cf_config = CounterfactualConfig(model_analysis_run, run_config)
cf_config.add_request(
    total_CFs=10, # Total number of counterfactuals required
    desired_range=[10, 300], # For regression problems, contains the outcome range to generate counterfactuals in
    feature_importance=True # Flag to compute feature importance using dice-ml.
)
cf_run = model_analysis_run.submit_child(cf_config)<jupyter_output><empty_output><jupyter_text>Submit run for causal effectsCompute causal effects based on the model analysis.The causal run is a child run of the model analysis run.You may use the `add_request` method that allows you to specify extra parameters to configure the causal effects to be generated.<jupyter_code>causal_config = CausalConfig(model_analysis_run, run_config)
causal_config.add_request(
    treatment_features=['ZN', 'NOX'], # Treatment feature names
    nuisance_model='linear' # Model type to use for nuisance estimation
)
causal_run = model_analysis_run.submit_child(causal_config)<jupyter_output><empty_output><jupyter_text>Download and inspect RAI insightsIn this section, we will demonstrate how to download the RAI insights computed in previous section and look at different aspects of your trained model. Download explanations and view global feature importanceBefore downloading the explanations, make sure that the `explain_run` has completed.The `explanation_manager.list` method below returns a list of metadata dictionaries for each explain run.  In this case, there is a single explain run.  So, the list contains a single dictionary. You can then download the computed explanations using the `download_by_id` method in the `explanation_manager` and look at the feature importance.<jupyter_code>explain_run.wait_for_completion(raise_on_error=True, wait_post_processing=True)
explanations_meta = model_analysis_run.explanation_manager.list()
explanation = model_analysis_run.explanation_manager.download_by_id(explanations_meta[0]['id'])
explanation.get_feature_importance_dict()<jupyter_output><empty_output><jupyter_text>Download error analysis reportBefore downloading the error analysis report, make sure that the `error_analysis_run` has completed.The `error_analysis_manager.list` method below returns a list of metadata dictionaries for each error analysis run.  In this case, there is a single error analysis run.  So, the list contains a single dictionary. You can then download the computed error analysis report using the `download_by_id` method in the `error_analysis_manager` and inspect the error analysis report.<jupyter_code>error_analysis_run.wait_for_completion(raise_on_error=True, wait_post_processing=True)
erroranalysis_meta = model_analysis_run.error_analysis_manager.list()
erroranalysis_report = model_analysis_run.error_analysis_manager.download_by_id(erroranalysis_meta[0]['id'])<jupyter_output><empty_output><jupyter_text>You can view the json tree and heatmap representations on the error analysis report directly, without the visualization widget or uploading it to AzureML<jupyter_code>erroranalysis_report.tree<jupyter_output><empty_output><jupyter_text>Download counterfactuals examplesBefore downloading the counterfactual examples, make sure that the `cf_run` has completed.The `counterfactual_manager.list` method below returns a list of metadata dictionaries for each counterfactual run.  In this case, there is a single counterfactual run.  So, the list contains a single dictionary.The `download_by_id()` method available in the `counterfactual_manager` can be used to download the counterfactual example.<jupyter_code>cf_run.wait_for_completion(raise_on_error=True, wait_post_processing=True)
cf_meta = model_analysis_run.counterfactual_manager.list()
cf_meta
counterfactual_object = model_analysis_run.counterfactual_manager.download_by_id(cf_meta[0]['id'])<jupyter_output><empty_output><jupyter_text>You can use `visualize_as_dataframe()` method to view the generated counterfactual examples for the samples in `test_dataset`.<jupyter_code>counterfactual_object.visualize_as_dataframe()<jupyter_output><empty_output><jupyter_text>You can use `summary_importance` property to see the feature importance which is computed when generating counterfactual examples.<jupyter_code>counterfactual_object.summary_importance<jupyter_output><empty_output><jupyter_text>Download causal effectsBefore downloading the causal effects, make sure that the `causal_run` has completed.The `causal_manager.list` method below returns a list of metadata dictionaries for each causal effects run.  In this case, there is a single causal effects run.  So, the list contains a single dictionary. You can then download the computed causal effects using the `download_by_id` method in the `causal_manager` and inspect the downloaded causal effects.<jupyter_code>causal_run.wait_for_completion(raise_on_error=True, wait_post_processing=True)
causal_meta = model_analysis_run.causal_manager.list()
causal_object = model_analysis_run.causal_manager.download_by_id(causal_meta[0]['id'])
causal_object['global_effects']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06219458018658374, 'fraction_numerical': 0.0026099511328298536, 'mean_word_length': 6.618020304568528, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 98, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '323832', 'license': 'MIT', 'n_tokens_mistral': 5007, 'n_tokens_neox': 4839, 'n_words': 1626, 'path': 'notebooks/model-analysis/regression/azureml-model-analysis-regression.ipynb', 'repo_name': 'Azure/Azureml-ResponsibleAI-Preview', 'size': '18008'}