<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.png)  Automated Machine Learning_**Classification of credit card fraudulent transactions with local run **_ Contents1. [Introduction](Introduction)1. [Setup](Setup)1. [Train](Train)1. [Results](Results)1. [Test](Tests)1. [Explanation](Explanation)1. [Acknowledgements](Acknowledgements)  IntroductionIn this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.This notebook is using the local machine compute to train the model.If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. In this notebook you will learn how to:1. Create an experiment using an existing workspace.2. Configure AutoML using `AutoMLConfig`.3. Train the model.4. Explore the results.5. Test the fitted model.6. Explore any model's explanation and explore feature importance in azure portal.7. Create an AKS cluster, deploy the webservice of AutoML scoring model and the explainer model to the AKS and consume the web service.  SetupAs part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments.<jupyter_code>import logging

from matplotlib import pyplot as plt
import pandas as pd

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.core.dataset import Dataset
from azureml.train.automl import AutoMLConfig
from azureml.interpret import ExplanationClient<jupyter_output><empty_output><jupyter_text>This sample notebook may use features that are not available in previous versions of the Azure ML SDK.<jupyter_code>print("This notebook was created using version 1.20.0 of the Azure ML SDK")
print("You are currently using version", azureml.core.VERSION, "of the Azure ML SDK")
ws = Workspace.from_config()

# choose a name for experiment
experiment_name = 'automl-classification-ccard-local'

experiment=Experiment(ws, experiment_name)

output = {}
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Experiment Name'] = experiment.name
pd.set_option('display.max_colwidth', -1)
outputDf = pd.DataFrame(data = output, index = [''])
outputDf.T<jupyter_output><empty_output><jupyter_text>Load DataLoad the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model.<jupyter_code>data = "https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv"
dataset = Dataset.Tabular.from_delimited_files(data)
training_data, validation_data = dataset.random_split(percentage=0.8, seed=223)
label_column_name = 'Class'<jupyter_output><empty_output><jupyter_text>TrainInstantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.|Property|Description||-|-||**task**|classification or regression||**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: accuracyAUC_weightedaverage_precision_score_weightednorm_macro_recallprecision_score_weighted||**enable_early_stopping**|Stop the run if the metric score is not showing improvement.||**n_cross_validations**|Number of cross validation splits.||**training_data**|Input dataset, containing both features and label column.||**label_column_name**|The name of the label column.|**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-trainprimary-metric)<jupyter_code>automl_settings = {
    "n_cross_validations": 3,
    "primary_metric": 'average_precision_score_weighted',
    "experiment_timeout_hours": 0.25, # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ability to find the best model possible
    "verbosity": logging.INFO,
    "enable_stack_ensemble": False
}

automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             training_data = training_data,
                             label_column_name = label_column_name,
                             **automl_settings
                            )<jupyter_output><empty_output><jupyter_text>Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while.In this example, we specify `show_output = True` to print currently running iterations to the console.<jupyter_code>local_run = experiment.submit(automl_config, show_output = True)
# If you need to retrieve a run that already started, use the following code
#from azureml.train.automl.run import AutoMLRun
#local_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')
local_run<jupyter_output><empty_output><jupyter_text>Results  Widget for Monitoring RunsThe widget will first report a "loading" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details<jupyter_code>from azureml.widgets import RunDetails
RunDetails(local_run).show()<jupyter_output><empty_output><jupyter_text>Analyze results Retrieve the Best ModelBelow we select the best pipeline from our iterations. The `get_output` method on `automl_classifier` returns the best run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*.<jupyter_code>best_run, fitted_model = local_run.get_output()
fitted_model<jupyter_output><empty_output><jupyter_text>Print the properties of the modelThe fitted_model is a python object and you can read the different properties of the object.  TestsNow that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values.<jupyter_code># convert the test data to dataframe
X_test_df = validation_data.drop_columns(columns=[label_column_name]).to_pandas_dataframe()
y_test_df = validation_data.keep_columns(columns=[label_column_name], validate=True).to_pandas_dataframe()
# call the predict functions on the model
y_pred = fitted_model.predict(X_test_df)
y_pred<jupyter_output><empty_output><jupyter_text>Calculate metrics for the predictionNow visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values from the trained model that was returned.<jupyter_code>from sklearn.metrics import confusion_matrix
import numpy as np
import itertools

cf =confusion_matrix(y_test_df.values,y_pred)
plt.imshow(cf,cmap=plt.cm.Blues,interpolation='nearest')
plt.colorbar()
plt.title('Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
class_labels = ['False','True']
tick_marks = np.arange(len(class_labels))
plt.xticks(tick_marks,class_labels)
plt.yticks([-0.5,0,1,1.5],['','False','True',''])
# plotting text value inside cells
thresh = cf.max() / 2.
for i,j in itertools.product(range(cf.shape[0]),range(cf.shape[1])):
    plt.text(j,i,format(cf[i,j],'d'),horizontalalignment='center',color='white' if cf[i,j] >thresh else 'black')
plt.show()<jupyter_output><empty_output><jupyter_text>ExplanationIn this section, we will show how to compute model explanations and visualize the explanations using azureml-interpret package. We will also show how to run the automl model and the explainer model through deploying an AKS web service.Besides retrieving an existing model explanation for an AutoML model, you can also explain your AutoML model with different test data. The following steps will allow you to compute and visualize engineered feature importance based on your test data. Run the explanation Download the engineered feature importance from artifact storeYou can use ExplanationClient to download the engineered feature explanations from the artifact store of the best_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features.<jupyter_code>client = ExplanationClient.from_run(best_run)
engineered_explanations = client.download_model_explanation(raw=False)
print(engineered_explanations.get_feature_importance_dict())
print("You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n" + best_run.get_portal_url())<jupyter_output><empty_output><jupyter_text>Download the raw feature importance from artifact storeYou can use ExplanationClient to download the raw feature explanations from the artifact store of the best_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the raw features.<jupyter_code>raw_explanations = client.download_model_explanation(raw=True)
print(raw_explanations.get_feature_importance_dict())
print("You can visualize the raw explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n" + best_run.get_portal_url())<jupyter_output><empty_output><jupyter_text>Retrieve any other AutoML model from training<jupyter_code>automl_run, fitted_model = local_run.get_output(metric='accuracy')<jupyter_output><empty_output><jupyter_text>Setup the model explanations for AutoML modelsThe fitted_model can generate the following which will be used for getting the engineered explanations using automl_setup_model_explanations:-1. Featurized data from train samples/test samples2. Gather engineered name lists3. Find the classes in your labeled column in classification scenariosThe automl_explainer_setup_obj contains all the structures from above list.<jupyter_code>X_train = training_data.drop_columns(columns=[label_column_name])
y_train = training_data.keep_columns(columns=[label_column_name], validate=True)
X_test = validation_data.drop_columns(columns=[label_column_name])
from azureml.train.automl.runtime.automl_explain_utilities import automl_setup_model_explanations

automl_explainer_setup_obj = automl_setup_model_explanations(fitted_model, X=X_train, 
                                                             X_test=X_test, y=y_train, 
                                                             task='classification')<jupyter_output><empty_output><jupyter_text>Initialize the Mimic Explainer for feature importanceFor explaining the AutoML models, use the MimicWrapper from azureml-interpret package. The MimicWrapper can be initialized with fields in automl_explainer_setup_obj, your workspace and a surrogate model to explain the AutoML model (fitted_model here). The MimicWrapper also takes the automl_run object where engineered explanations will be uploaded.<jupyter_code>from interpret.ext.glassbox import LGBMExplainableModel
from azureml.interpret.mimic_wrapper import MimicWrapper
explainer = MimicWrapper(ws, automl_explainer_setup_obj.automl_estimator,
                         explainable_model=automl_explainer_setup_obj.surrogate_model, 
                         init_dataset=automl_explainer_setup_obj.X_transform, run=automl_run,
                         features=automl_explainer_setup_obj.engineered_feature_names, 
                         feature_maps=[automl_explainer_setup_obj.feature_map],
                         classes=automl_explainer_setup_obj.classes,
                         explainer_kwargs=automl_explainer_setup_obj.surrogate_model_params)<jupyter_output><empty_output><jupyter_text>Use Mimic Explainer for computing and visualizing engineered feature importanceThe explain() method in MimicWrapper can be called with the transformed test samples to get the feature importance for the generated engineered features. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features.<jupyter_code># Compute the engineered explanations
engineered_explanations = explainer.explain(['local', 'global'], eval_dataset=automl_explainer_setup_obj.X_test_transform)
print(engineered_explanations.get_feature_importance_dict())
print("You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n" + automl_run.get_portal_url())<jupyter_output><empty_output><jupyter_text>Use Mimic Explainer for computing and visualizing raw feature importanceThe explain() method in MimicWrapper can be called with the transformed test samples to get the feature importance for the original features in your data. You can also use azure portal url to view the dash board visualization of the feature importance values of the original/raw features.<jupyter_code># Compute the raw explanations
raw_explanations = explainer.explain(['local', 'global'], get_raw=True,
                                     raw_feature_names=automl_explainer_setup_obj.raw_feature_names,
                                     eval_dataset=automl_explainer_setup_obj.X_test_transform,
                                     raw_eval_dataset=automl_explainer_setup_obj.X_test_raw)
print(raw_explanations.get_feature_importance_dict())
print("You can visualize the raw explanations under the 'Explanations (preview)' tab in the AutoML run at:-\n" + automl_run.get_portal_url())<jupyter_output><empty_output><jupyter_text>Initialize the scoring Explainer, save and upload it for later use in scoring explanation<jupyter_code>from azureml.interpret.scoring.scoring_explainer import TreeScoringExplainer
import joblib

# Initialize the ScoringExplainer
scoring_explainer = TreeScoringExplainer(explainer.explainer, feature_maps=[automl_explainer_setup_obj.feature_map])

# Pickle scoring explainer locally to './scoring_explainer.pkl'
scoring_explainer_file_name = 'scoring_explainer.pkl'
with open(scoring_explainer_file_name, 'wb') as stream:
    joblib.dump(scoring_explainer, stream)

# Upload the scoring explainer to the automl run
automl_run.upload_file('outputs/scoring_explainer.pkl', scoring_explainer_file_name)<jupyter_output><empty_output><jupyter_text>Deploying the scoring and explainer models to a web service to Azure Kubernetes Service (AKS)We use the TreeScoringExplainer from azureml.interpret package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service.<jupyter_code># Register trained automl model present in the 'outputs' folder in the artifacts
original_model = automl_run.register_model(model_name='automl_model', 
                                           model_path='outputs/model.pkl')
scoring_explainer_model = automl_run.register_model(model_name='scoring_explainer',
                                                    model_path='outputs/scoring_explainer.pkl')<jupyter_output><empty_output><jupyter_text>Create the conda dependencies for setting up the serviceWe need to download the conda dependencies using the automl_run object.<jupyter_code>from azureml.automl.core.shared import constants
from azureml.core.environment import Environment

automl_run.download_file(constants.CONDA_ENV_FILE_PATH, 'myenv.yml')
myenv = Environment.from_conda_specification(name="myenv", file_path="myenv.yml")
myenv<jupyter_output><empty_output><jupyter_text>Write the Entry ScriptWrite the script that will be used to predict on your model<jupyter_code>%%writefile score.py
import joblib
import pandas as pd
from azureml.core.model import Model
from azureml.train.automl.runtime.automl_explain_utilities import automl_setup_model_explanations


def init():
    global automl_model
    global scoring_explainer

    # Retrieve the path to the model file using the model name
    # Assume original model is named original_prediction_model
    automl_model_path = Model.get_model_path('automl_model')
    scoring_explainer_path = Model.get_model_path('scoring_explainer')

    automl_model = joblib.load(automl_model_path)
    scoring_explainer = joblib.load(scoring_explainer_path)


def run(raw_data):
    data = pd.read_json(raw_data, orient='records')    
    # Make prediction
    predictions = automl_model.predict(data)
    # Setup for inferencing explanations
    automl_explainer_setup_obj = automl_setup_model_explanations(automl_model,
                                                                 X_test=data, task='classification')
    # Retrieve model explanations for engineered explanations
    engineered_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform)
    # Retrieve model explanations for raw explanations
    raw_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform, get_raw=True)
    # You can return any data type as long as it is JSON-serializable
    return {'predictions': predictions.tolist(),
            'engineered_local_importance_values': engineered_local_importance_values,
            'raw_local_importance_values': raw_local_importance_values}<jupyter_output><empty_output><jupyter_text>Create the InferenceConfig Create the inference config that will be used when deploying the model<jupyter_code>from azureml.core.model import InferenceConfig

inf_config = InferenceConfig(entry_script='score.py', environment=myenv)<jupyter_output><empty_output><jupyter_text>Provision the AKS ClusterThis is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it.<jupyter_code>from azureml.core.compute import ComputeTarget, AksCompute
from azureml.core.compute_target import ComputeTargetException

# Choose a name for your cluster.
aks_name = 'scoring-explain'

# Verify that cluster does not exist already
try:
    aks_target = ComputeTarget(workspace=ws, name=aks_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    prov_config = AksCompute.provisioning_configuration(vm_size='STANDARD_D3_V2')
    aks_target = ComputeTarget.create(workspace=ws, 
                                      name=aks_name,
                                      provisioning_configuration=prov_config)
aks_target.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Deploy web service to AKS<jupyter_code># Set the web service configuration (using default here)
from azureml.core.webservice import AksWebservice
from azureml.core.model import Model

aks_config = AksWebservice.deploy_configuration()
aks_service_name ='model-scoring-local-aks'

aks_service = Model.deploy(workspace=ws,
                           name=aks_service_name,
                           models=[scoring_explainer_model, original_model],
                           inference_config=inf_config,
                           deployment_config=aks_config,
                           deployment_target=aks_target)

aks_service.wait_for_deployment(show_output = True)
print(aks_service.state)<jupyter_output><empty_output><jupyter_text>View the service logs<jupyter_code>aks_service.get_logs()<jupyter_output><empty_output><jupyter_text>Consume the web service using run method to do the scoring and explanation of scoring.We test the web sevice by passing data. Run() method retrieves API keys behind the scenes to make sure that call is authenticated.<jupyter_code># Serialize the first row of the test data into json
X_test_json = X_test_df[:1].to_json(orient='records')
print(X_test_json)

# Call the service to get the predictions and the engineered and raw explanations
output = aks_service.run(X_test_json)

# Print the predicted value
print('predictions:\n{}\n'.format(output['predictions']))
# Print the engineered feature importances for the predicted value
print('engineered_local_importance_values:\n{}\n'.format(output['engineered_local_importance_values']))
# Print the raw feature importances for the predicted value
print('raw_local_importance_values:\n{}\n'.format(output['raw_local_importance_values']))<jupyter_output><empty_output><jupyter_text>Clean upDelete the service.<jupyter_code>aks_service.delete()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06849892312014233, 'fraction_numerical': 0.002668789212473078, 'mean_word_length': 5.293164407778432, 'pattern_counts': {'":': 5, '<': 106, '<?xml version=': 0, '>': 107, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '8162', 'license': 'MIT', 'n_tokens_mistral': 5982, 'n_tokens_neox': 5771, 'n_words': 1856, 'path': 'how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb', 'repo_name': 'AaronZhangAus/MachineLearningNotebooks', 'size': '21358'}