<jupyter_start><jupyter_text>Debug housing price predictions This notebook demonstrates the use of the AzureML RAI components to assess a classification model trained on Kaggle's apartments dataset (https://www.kaggle.com/alphaepsilon/housing-prices-dataset). The model predicts if the house sells for more than median price or not. It is a reimplementation of the [notebook of the same name](https://github.com/microsoft/responsible-ai-toolbox/blob/main/notebooks/responsibleaidashboard/responsibleaidashboard-housing-classification-model-debugging.ipynb) in the [Responsible AI toolbox repo](https://github.com/microsoft/responsible-ai-toolbox).First, we need to specify the version of the RAI components which are available in the workspace. This was specified when the components were uploaded, and will have defaulted to '1':<jupyter_code>version_string = '1'<jupyter_output><empty_output><jupyter_text>We also need to give the name of the compute cluster we want to use in AzureML. Later in this notebook, we will create it if it does not already exist:<jupyter_code>compute_name = "cpucluster"<jupyter_output><empty_output><jupyter_text>Finally, we need to specify a version for the data and components we will create while running this notebook. This should be unique for the workspace, but the specific value doesn't matter:<jupyter_code>rai_housing_example_version_string = '4'<jupyter_output><empty_output><jupyter_text>Accessing the DataThe following section examines the code necessary to create datasets and a model using components in AzureML.  Fetching the data<jupyter_code>import shap
import sklearn
import pandas as pd

from sklearn.model_selection import train_test_split

import zipfile<jupyter_output><empty_output><jupyter_text>First, we load the data from the blob store, do some basic data cleaning, and split in to training and test datasets:<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

def split_label(dataset, target_feature):
    X = dataset.drop([target_feature], axis=1)
    y = dataset[[target_feature]]
    return X, y

target_feature = 'Sold_HigherThan_Median'
categorical_features = []

outdirname = 'responsibleai.12.28.21'
try:
    from urllib import urlretrieve
except ImportError:
    from urllib.request import urlretrieve
zipfilename = outdirname + '.zip'
urlretrieve('https://publictestdatasets.blob.core.windows.net/data/' + zipfilename, zipfilename)
with zipfile.ZipFile(zipfilename, 'r') as unzip:
    unzip.extractall('.')

all_data = pd.read_csv('apartments-train.csv')
all_data = all_data.drop(['SalePrice','SalePriceK'], axis=1)
X, y = split_label(all_data, target_feature)


X_train_original, X_test_original, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=7, stratify=y)

train_data = X_train_original.copy()
train_data[target_feature] = y_train

test_data = X_test_original.copy()
test_data[target_feature] = y_test<jupyter_output><empty_output><jupyter_text>Get the Data to AzureMLWith the data now split into 'train' and 'test' DataFrames, we save them out to files in preparation for upload into AzureML:<jupyter_code>print("Saving to files")
train_data.to_parquet("housing_train.parquet", index=False)
test_data.to_parquet("housing_test.parquet", index=False)<jupyter_output><empty_output><jupyter_text>We are going to create two Datasets in AzureML, one for the train and one for the test datasets. The first step is to create an `MLClient` to perform the upload. The method we use assumes that there is a `config.json` file (downloadable from the Azure or AzureML portals) present in the same directory as this notebook file:<jupyter_code>from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
ml_client = MLClient.from_config(credential=DefaultAzureCredential(exclude_shared_token_cache_credential=True),
                     logging_enable=True)<jupyter_output><empty_output><jupyter_text>We can then define the Datasets, and create them in AzureML. This will also upload the Parquet files:<jupyter_code>from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

input_train_data = "housing_train_pq"
input_test_data = "housing_test_pq"

train_data = Data(
    path=f"housing_train.parquet",
    type=AssetTypes.URI_FILE,
    description="RAI housing example training data",
    name=input_train_data,
    version=rai_housing_example_version_string,
)
ml_client.data.create_or_update(train_data)

test_data = Data(
    path=f"housing_test.parquet",
    type=AssetTypes.URI_FILE,
    description="RAI housing example test data",
    name=input_test_data,
    version=rai_housing_example_version_string,
)
ml_client.data.create_or_update(test_data)<jupyter_output><empty_output><jupyter_text>A model training pipelineTo simplify the model creation process, we're going to use a pipeline. This will have two stages:1. The actual training component1. A model registration componentWe have to register the model in AzureML in order for our RAI insights components to use it. The Training ComponentThe training component is for this particular model. In this case, we are going to train an `LGBMClassifier` on the input data and save it using MLFlow. We need command line arguments to specify the location of the input data, the location where MLFlow should write the output model, and the name of the target column in the dataset.We start by creating a directory to hold the component source:<jupyter_code>import os

os.makedirs('component_src', exist_ok=True)<jupyter_output><empty_output><jupyter_text>Now, the component source code itself:<jupyter_code>%%writefile component_src/housing_training_script.py

import argparse
import os
import shutil
import tempfile


from azureml.core import Run

import mlflow
import mlflow.sklearn

import pandas as pd
from lightgbm import LGBMClassifier

def parse_args():
    # setup arg parser
    parser = argparse.ArgumentParser()

    # add arguments
    parser.add_argument("--training_data", type=str, help="Path to training data")
    parser.add_argument("--target_column_name", type=str, help="Name of target column")
    parser.add_argument("--model_output", type=str, help="Path of output model")

    # parse args
    args = parser.parse_args()

    # return args
    return args


def main(args):
    current_experiment = Run.get_context().experiment
    tracking_uri = current_experiment.workspace.get_mlflow_tracking_uri()
    print("tracking_uri: {0}".format(tracking_uri))
    mlflow.set_tracking_uri(tracking_uri)
    mlflow.set_experiment(current_experiment.name)

    # Read in data
    print("Reading data")
    all_data = pd.read_parquet(args.training_data)

    print("Extracting X_train, y_train")
    print("all_data cols: {0}".format(all_data.columns))
    y_train = all_data[args.target_column_name]
    X_train = all_data.drop(labels=args.target_column_name, axis="columns")
    print("X_train cols: {0}".format(X_train.columns))

    print("Training model")
    # The estimator can be changed to suit
    model = LGBMClassifier(n_estimators=5)
    model.fit(X_train, y_train)

    # Saving model with mlflow - leave this section unchanged
    with tempfile.TemporaryDirectory() as td:
        print("Saving model with MLFlow to temporary directory")
        tmp_output_dir = os.path.join(td, "my_model_dir")
        mlflow.sklearn.save_model(sk_model=model, path=tmp_output_dir)

        print("Copying MLFlow model to output path")
        for file_name in os.listdir(tmp_output_dir):
            print("  Copying: ", file_name)
            # As of Python 3.8, copytree will acquire dirs_exist_ok as
            # an option, removing the need for listdir
            shutil.copy2(src=os.path.join(tmp_output_dir, file_name), dst=os.path.join(args.model_output, file_name))


# run script
if __name__ == "__main__":
    # add space in logs
    print("*" * 60)
    print("\n\n")

    # parse args
    args = parse_args()

    # run main function
    main(args)

    # add space in logs
    print("*" * 60)
    print("\n\n")<jupyter_output><empty_output><jupyter_text>Now that the training script is saved on our local drive, we create a YAML file to describe it as a component to AzureML. This involves defining the inputs and outputs, specifing the AzureML environment which can run the script, and telling AzureML how to invoke the training script:<jupyter_code>from azure.ai.ml import load_component

yaml_contents = f"""
$schema: http://azureml/sdk-2-0/CommandComponent.json
name: rai_housing_training_component
display_name: Housing training component for RAI example
version: {rai_housing_example_version_string}
type: command
inputs:
  training_data:
    type: path
  target_column_name:
    type: string
outputs:
  model_output:
    type: path
code: ./component_src/
environment: azureml:AML-RAI-Environment:{version_string}
""" + r"""
command: >-
  python housing_training_script.py
  --training_data ${{{{inputs.training_data}}}}
  --target_column_name ${{{{inputs.target_column_name}}}}
  --model_output ${{{{outputs.model_output}}}}
"""

yaml_filename = "RAIHousingTrainingComponent.yaml"

with open(yaml_filename, 'w') as f:
    f.write(yaml_contents.format(yaml_contents))
    
train_component_definition = load_component(
    path=yaml_filename
)

ml_client.components.create_or_update(train_component_definition)<jupyter_output><empty_output><jupyter_text>We need a compute target on which to run our jobs. The following checks whether the compute specified above is present; if not, then the compute target is created.<jupyter_code>from azure.ai.ml.entities import AmlCompute

all_compute_names = [x.name for x in ml_client.compute.list()]

if compute_name in all_compute_names:
    print(f"Found existing compute: {compute_name}")
else:
    my_compute = AmlCompute(
        name=compute_name,
        size="Standard_DS2_v2",
        min_instances=0,
        max_instances=4,
        idle_time_before_scale_down=3600
    )
    ml_client.compute.begin_create_or_update(my_compute)
    print("Initiated compute creation")<jupyter_output><empty_output><jupyter_text>Running a training pipelineThe component to register the model is part of the suite of RAI components, so we do not have to define it here. As such, we are now ready to run the training pipeline itself.We start by defining the name under which we want to register the model:<jupyter_code>import time

model_name_suffix = int(time.time())
model_name = 'rai_housing_classifier'<jupyter_output><empty_output><jupyter_text>Next, we define the pipeline using objects from the AzureML SDKv2. As mentioned above, there are two component jobs: one to train the model, and one to register it:<jupyter_code>from azure.ai.ml import dsl, Input

register_component = ml_client.components.get(
    name="register_model", version=version_string
)
train_model_component = ml_client.components.get(
    name="rai_housing_training_component", version=rai_housing_example_version_string
)

housing_train_pq = Input(
    type="uri_file", path=f"{input_train_data}:{rai_housing_example_version_string}", mode="download"
)
housing_test_pq = Input(
    type="uri_file", path=f"{input_test_data}:{rai_housing_example_version_string}", mode="download"
)

@dsl.pipeline(
    compute=compute_name,
    description="Register Model for RAI Housing example",
    experiment_name=f"RAI_Housing_Example_Model_Training_{model_name_suffix}",
)
def my_training_pipeline(target_column_name, training_data):
    trained_model = train_component_definition(
        target_column_name=target_column_name,
        training_data=training_data
    )
    trained_model.set_limits(timeout=120)

    _ = register_component(
        model_input_path=trained_model.outputs.model_output,
        model_base_name=model_name,
        model_name_suffix=model_name_suffix,
    )

    return {}

model_registration_pipeline_job = my_training_pipeline(target_feature, housing_train_pq)<jupyter_output><empty_output><jupyter_text>With the pipeline definition created, we can submit it to AzureML. We define a helper function to do the submission, which waits for the submitted job to complete:<jupyter_code>from azure.ai.ml.entities import PipelineJob

def submit_and_wait(ml_client, pipeline_job) -> PipelineJob:
    created_job = ml_client.jobs.create_or_update(pipeline_job)
    assert created_job is not None

    while created_job.status not in ['Completed', 'Failed', 'Canceled', 'NotResponding']:
        time.sleep(30)
        created_job = ml_client.jobs.get(created_job.name)
        print("Latest status : {0}".format(created_job.status))
    assert created_job.status == 'Completed'
    return created_job

# This is the actual submission
training_job = submit_and_wait(ml_client, model_registration_pipeline_job)<jupyter_output><empty_output><jupyter_text>Creating the RAI InsightsWe have a registered model, and can now run a pipeline to create the RAI insights. First off, compute the name of the model we registered:<jupyter_code>expected_model_id = f'{model_name}_{model_name_suffix}:1'<jupyter_output><empty_output><jupyter_text>Now, we create the RAI pipeline itself. There are four 'component stages' in this pipeline:1. Fetch the model1. Construct an empty `RAIInsights` object1. Run the RAI tool components1. Gather the tool outputs into a single `RAIInsights` objectWe start by loading the RAI component definitions for use in our pipeline:<jupyter_code>fetch_model_component = ml_client.components.get(
    name='fetch_registered_model', version=version_string
)

rai_constructor_component = ml_client.components.get(
    name="rai_insights_constructor", version=version_string
)

rai_explanation_component = ml_client.components.get(
    name="rai_insights_explanation", version=version_string
)

rai_causal_component = ml_client.components.get(
    name="rai_insights_causal", version=version_string
)

rai_counterfactual_component = ml_client.components.get(
    name="rai_insights_counterfactual", version=version_string
)

rai_erroranalysis_component = ml_client.components.get(
    name="rai_insights_erroranalysis", version=version_string
)

rai_gather_component = ml_client.components.get(
    name="rai_insights_gather", version=version_string
)<jupyter_output><empty_output><jupyter_text>Now the pipeline itself. This fetches the registered model, creates an empty `RAIInsights` object, adds the analyses, and then gathers everything into the final `RAIInsights` output. More complex objects (such as lists of feature names) have to be converted to JSON strings before being passed to the components. Note that the counterfactual generation has a longer timeout set, since it is a relatively slow process:<jupyter_code>import json

classes_in_target = json.dumps(["Less than median", "More than median"])
treatment_features = json.dumps(["OverallCond", "OverallQual", "Fireplaces", "GarageCars", "ScreenPorch"])

@dsl.pipeline(
        compute=compute_name,
        description="Example RAI computation on housing data",
        experiment_name=f"RAI_Housing_Example_RAIInsights_Computation_{model_name_suffix}",
    )
def rai_classification_pipeline(
        target_column_name,
        train_data,
        test_data,
    ):
        # Fetch the model
        fetch_job = fetch_model_component(
            model_id=expected_model_id
        )
        
        # Initiate the RAIInsights
        create_rai_job = rai_constructor_component(
            title="RAI Dashboard Example",
            task_type="classification",
            model_info_path=fetch_job.outputs.model_info_output_path,
            train_dataset=train_data,
            test_dataset=test_data,
            target_column_name=target_column_name,
            categorical_column_names=json.dumps(categorical_features),
            classes=classes_in_target
        )
        create_rai_job.set_limits(timeout=120)
        
        # Add an explanation
        explain_job = rai_explanation_component(
            comment="Explanation for the housing dataset",
            rai_insights_dashboard=create_rai_job.outputs.rai_insights_dashboard,
        )
        explain_job.set_limits(timeout=120)
        
        # Add causal analysis
        causal_job = rai_causal_component(
            rai_insights_dashboard=create_rai_job.outputs.rai_insights_dashboard,
            treatment_features=treatment_features,
        )
        causal_job.set_limits(timeout=120)
        
        # Add counterfactual analysis
        counterfactual_job = rai_counterfactual_component(
            rai_insights_dashboard=create_rai_job.outputs.rai_insights_dashboard,
            total_cfs=10,
            desired_class='opposite',
        )
        counterfactual_job.set_limits(timeout=600)
        
        # Add error analysis
        erroranalysis_job = rai_erroranalysis_component(
            rai_insights_dashboard=create_rai_job.outputs.rai_insights_dashboard,
        )
        erroranalysis_job.set_limits(timeout=120)
        
        # Combine everything
        rai_gather_job = rai_gather_component(
            constructor=create_rai_job.outputs.rai_insights_dashboard,
            insight_1=explain_job.outputs.explanation,
            insight_2=causal_job.outputs.causal,
            insight_3=counterfactual_job.outputs.counterfactual,
            insight_4=erroranalysis_job.outputs.error_analysis,
        )
        rai_gather_job.set_limits(timeout=120)

        rai_gather_job.outputs.dashboard.mode = "upload"
        rai_gather_job.outputs.ux_json.mode = "upload"

        return {
            "dashboard": rai_gather_job.outputs.dashboard,
            "ux_json": rai_gather_job.outputs.ux_json,
        }<jupyter_output><empty_output><jupyter_text>With all of our jobs defined, we can assemble them into the pipeline itself.<jupyter_code>import uuid
from azure.ai.ml import Output

# Pipeline to construct the RAI Insights
insights_pipeline_job = rai_classification_pipeline(
    target_column_name=target_feature,
    train_data=housing_train_pq,
    test_data=housing_test_pq,
)

# Workaround to enable the download
rand_path = str(uuid.uuid4())
insights_pipeline_job.outputs.dashboard = Output(
    path=f"azureml://datastores/workspaceblobstore/paths/{rand_path}/dashboard/",
    mode="upload",
    type="uri_folder",
)
insights_pipeline_job.outputs.ux_json = Output(
    path=f"azureml://datastores/workspaceblobstore/paths/{rand_path}/ux_json/",
    mode="upload",
    type="uri_folder",
)<jupyter_output><empty_output><jupyter_text>Now, submit the pipeline job and wait for it to complete:<jupyter_code>insights_job = submit_and_wait(ml_client, insights_pipeline_job)<jupyter_output><empty_output><jupyter_text>Download RAI insight and start widget<jupyter_code>import tempfile
import pathlib
from responsibleai import RAIInsights
from raiwidgets import ResponsibleAIDashboard
with tempfile.TemporaryDirectory() as dashboard_path:
        ml_client.jobs.download(
            insights_job.name, download_path=dashboard_path, output_name="dashboard"
        )
        expected_path = pathlib.Path(dashboard_path) / 'named-outputs' / 'dashboard'
        # This load is very fragile with respect to Python version and conda environment
        rai_i = RAIInsights.load(expected_path)
        ResponsibleAIDashboard(rai_i)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07959687371452077, 'fraction_numerical': 0.003805018510900864, 'mean_word_length': 4.805671641791045, 'pattern_counts': {'":': 3, '<': 85, '<?xml version=': 0, '>': 87, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '60359', 'license': 'MIT', 'n_tokens_mistral': 5958, 'n_tokens_neox': 5903, 'n_words': 1485, 'path': 'examples/notebooks/responsibleaidashboard-housing-classification-model-debugging.ipynb', 'repo_name': 'Azure/RAI-vNext-Preview', 'size': '19448'}