<jupyter_start><jupyter_text>AutoML Classification experiment using Local Compute and Pandas DataFrames- Save trained model as Scikit-Learn model (.pkl) and as ONNX model (.onnx file)- Data: IBM Employee Attrition dataset loaded from Azure ML Dataset**Note:** Install `onnxruntime` via pip and then restart the kernel of this notebook.<jupyter_code>! pip install onnxruntime
# Restart Kernel afterwards!<jupyter_output><empty_output><jupyter_text>Get Azure ML Workspace to use<jupyter_code># Check core SDK version number
import azureml.core

print("SDK version:", azureml.core.VERSION)
from azureml.core import Workspace, Dataset

# Get Workspace defined in by default config.json file
ws = Workspace.from_config()
from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id="7<PASSWORD>")

ws = Workspace(subscription_id="3c8972d9-f541-46b2-b70b-d81baba3595d",
                  resource_group="amlworkshop",
                  workspace_name="amlworkshop",
                  auth=interactive_auth)
print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\n')<jupyter_output><empty_output><jupyter_text>Load data from Azure ML Datasets into Pandas DataFrame<jupyter_code># Load Data
aml_dataset = ws.datasets['IBM-Employee-Attrition']

# Use Pandas DataFrame just to sneak peak some data and schema
full_df = aml_dataset.to_pandas_dataframe()
# .to_pandas_dataframe().dropna()
full_df.head(5)
# Use Pandas DataFrame just to investigate the dataset's schema and info
full_df.describe()<jupyter_output><empty_output><jupyter_text>Clean up the initial dataset (Using related Pandas DataFrame)<jupyter_code># Dropping Employee count as all values are 1 and hence attrition is independent of this feature
full_df = full_df.drop(['EmployeeCount'], axis=1)

# Dropping Employee Number since it is merely an identifier
full_df = full_df.drop(['EmployeeNumber'], axis=1)

full_df = full_df.drop(['Over18'], axis=1)

# Since all values are 80
full_df = full_df.drop(['StandardHours'], axis=1)<jupyter_output><empty_output><jupyter_text>Split original dataset in test/train sets using Scikit-Learn train_test_split function<jupyter_code>from sklearn.model_selection import train_test_split

# Split using ScikitLearn train_test_split function using Dataframes
# Will use test dataframe at the end, without AutoML, just for testing predictions with the model

# Only split in test/train
train_df, test_df = train_test_split(full_df, test_size=0.2, random_state=1)
train_df.describe()

# Split in x/y and test/train
# y_df = full_df.pop("Attrition")
# x_df = full_df
# x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.1, random_state=1)

#Another possibility would be to split using the Azure ML Datasets (Better for Remote Compute): 
# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py#random-split-percentage--seed-none-
test_df.describe()<jupyter_output><empty_output><jupyter_text>List and select primary metric to drive the AutoML classification problem<jupyter_code>from azureml.train import automl

# List of possible primary metrics is here:
# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric
    
# Get a list of valid metrics for your given task
automl.utilities.get_primary_metrics('classification')

# We'll use 'accuracy' as primary metric (Closer to 1.00 is better)<jupyter_output><empty_output><jupyter_text>Define AutoML Experiment settings<jupyter_code>import logging

# Explanation of Settings: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#configure-your-experiment-settings

# AutoMLConfig info on: 
# https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig

# You can provide additional settings as a **kwargs parameter for the AutoMLConfig object
# automl_settings = {
#     "whitelist_models": 'XGBoostClassifier'
# }

from azureml.train.automl import AutoMLConfig

project_folder = './automl'
os.makedirs(project_folder, exist_ok=True)

automl_config = AutoMLConfig(task='classification',
                             primary_metric='accuracy',
                             experiment_timeout_minutes=5,
                             training_data=train_df,
                             label_column_name="Attrition",
                             n_cross_validations=5,
                             # blacklist_models='XGBoostClassifier', 
                             # iteration_timeout_minutes=5,                                                    
                             enable_early_stopping= True,
                             featurization='auto',
                             debug_log='automated_ml_errors.log',
                             verbosity=logging.INFO,
                             enable_onnx_compatible_models=True,
                             path=project_folder
                             # **automl_settings
                             )<jupyter_output><empty_output><jupyter_text>Run Experiment with multiple child runs under the covers<jupyter_code>from azureml.core import Experiment
from datetime import datetime

now = datetime.now()
time_string = now.strftime("%m-%d-%Y-%H")
print(time_string)
experiment_name = "classif-automl-local-{0}".format(time_string)
print(experiment_name)

experiment = Experiment(workspace=ws, name=experiment_name)

run = experiment.submit(automl_config, show_output=True)<jupyter_output><empty_output><jupyter_text>Explore results with Widget<jupyter_code># Explore the results of automatic training with a Jupyter widget: https://docs.microsoft.com/en-us/python/api/azureml-widgets/azureml.widgets?view=azure-ml-py
from azureml.widgets import RunDetails
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>Retrieve the 'Best' Scikit-Learn Model<jupyter_code>best_run, fitted_model = run.get_output()
print(best_run)
print('--------')
print(fitted_model)<jupyter_output><empty_output><jupyter_text>Retrieve the 'Best' ONNX ModelBelow we select the best pipeline from our iterations. The get_output method returns the best run and the fitted model. The Model includes the pipeline and any pre-processing. Overloads on get_output allow you to retrieve the best run and fitted model for any logged metric or for a particular iteration.Set the parameter return_onnx_model=True to retrieve the best ONNX model, instead of the Python model.<jupyter_code>best_run, onnx_mdl = run.get_output(return_onnx_model=True)<jupyter_output><empty_output><jupyter_text>Explicetely Save the best ONNX model on local drive path<jupyter_code>from azureml.automl.runtime.onnx_convert import OnnxConverter
onnx_fl_path = "./best_model.onnx"
OnnxConverter.save_onnx_model(onnx_mdl, onnx_fl_path)<jupyter_output><empty_output><jupyter_text>See files associated with the 'Best run'<jupyter_code>print(best_run.get_file_names())

# best_run.download_file('azureml-logs/70_driver_log.txt')<jupyter_output><empty_output><jupyter_text>Download experiment run related files Model files (.pkl and .onnx), Environment files to see Conda and Environment dependencies used by AutoML, etc.<jupyter_code># Download the run's files
best_run.download_file('outputs/model.pkl')
best_run.download_file('outputs/model.onnx')
best_run.download_file('outputs/model_onnx.json')
best_run.download_file('outputs/conda_env_v_1_0_0.yml')
best_run.download_file('outputs/env_dependencies.json')
best_run.download_file('outputs/scoring_file_v_1_0_0.py')
best_run.download_file('pipeline_graph.json')<jupyter_output><empty_output><jupyter_text>Register the Scikit-Learn model (.pkl file)Once you've trained the model, you can save and register it to your workspace. Model registration lets you store and version your models in your workspace to simplify model management and deployment.Running the following code will register the model to your workspace, and will make it available to reference by name in remote compute contexts or deployment scripts.<jupyter_code>from azureml.core.model import Model

model_reg = best_run.register_model(model_name='classif-empl-automl-local',      # Name of the registered model in your workspace.
                                    description='Binary classification model for employees attrition. From AutoML local training',
                                    model_path='outputs/model.pkl',              # Local file to upload and register as a model.
                                    model_framework=Model.Framework.SCIKITLEARN, # Framework used to create the model.
                                    model_framework_version='0.20.3',            # Version of scikit-learn used to create the model.
                                    tags={'ml-task': "binary-classification", 'business-area': "HR"},
                                    properties={'pandas-version': "0.23.4"},
                                    sample_input_dataset=aml_dataset
                                  )

print(model_reg)

# (Q1:) How can we know what frameworks/libraries versions were used by AutoML when training the model?
#         - If investigating the outputs/conda_env_v_1_0_0.yml file, some libraries like Pandas and Scikit-Learn are not showing any version...

# (Q2:) Why best_run.get_environment() fails?
# best_run_environment = best_run.get_environment() 
# print(best_run_environment)<jupyter_output><empty_output><jupyter_text>Make Predictions  Prep Test Data: Extract X values (feature columns) from test dataset and convert to NumPi array for predicting<jupyter_code>import pandas as pd

#Remove Label/y column
if 'Attrition' in test_df.columns:
    y_test_df = test_df.pop('Attrition')

x_test_df = test_df
x_test_df.describe()<jupyter_output><empty_output><jupyter_text>Make Predictions with Scikit-Learn Model  (Optional) Download Model from Registry and load in-memory<jupyter_code>print(Model.get_model_path('classif-empl-automl-local', _workspace=ws))

model_definition_from_registry = Model(ws,'classif-empl-automl-local')
model_definition_from_registry.download(target_dir='.', exist_ok=True)
print(model_definition_from_registry)
print('-------')

# Load the model into memory
import joblib
fitted_model = joblib.load('model.pkl')
print(fitted_model)
# Try the best model making predictions with the test dataset
y_predictions = fitted_model.predict(x_test_df)

print('10 predictions: ')
print(y_predictions[:10])
y_predictions.shape<jupyter_output><empty_output><jupyter_text>Make Predictions with the ONNX model, using onnxruntime packageNeeds pip install onnxruntime==1.0.0' in environment (Also try with 1.1.0 version)<jupyter_code>import sys
import json
from azureml.automl.core.onnx_convert import OnnxConvertConstants
from azureml.train.automl import constants

if sys.version_info < OnnxConvertConstants.OnnxIncompatiblePythonVersion:
    python_version_compatible = True
else:
    python_version_compatible = False
    
import onnxruntime
from azureml.automl.runtime.onnx_convert import OnnxInferenceHelper

def get_onnx_res(run):
    res_path = 'onnx_resource.json'
    run.download_file(name=constants.MODEL_RESOURCE_PATH_ONNX, output_file_path=res_path)
    with open(res_path) as f:
        onnx_res = json.load(f)
    return onnx_res

if python_version_compatible:
    # test_df = test_dataset.to_pandas_dataframe()
    mdl_bytes = onnx_mdl.SerializeToString()
    onnx_res = get_onnx_res(best_run)

    onnxrt_helper = OnnxInferenceHelper(mdl_bytes, onnx_res)
    pred_onnx, pred_prob_onnx = onnxrt_helper.predict(x_test_df)

    print('Predicting with ONNX model...')
    print(pred_onnx)
    print(pred_prob_onnx)
else:
    print('Please use Python version 3.6 or 3.7 to run the inference helper.')<jupyter_output><empty_output><jupyter_text>Calculate the Accuracy with Test Dataset (Data not used for training)<jupyter_code>from sklearn.metrics import accuracy_score

print('Accuracy with Scikit-Learn model:')
print(accuracy_score(y_test_df, y_predictions))

print('Accuracy with ONNX model:')
print(accuracy_score(y_test_df, pred_onnx))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08669999180529378, 'fraction_numerical': 0.0064738179136277964, 'mean_word_length': 4.359683794466403, 'pattern_counts': {'":': 1, '<': 79, '<?xml version=': 0, '>': 78, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '471309', 'license': 'MIT', 'n_tokens_mistral': 3818, 'n_tokens_neox': 3679, 'n_words': 894, 'path': '2-training-inference/2.3-automl-training/local-compute/binaryclassification-employee-attrition-autoaml-local-compute.ipynb', 'repo_name': 'krisbock/azureml-workshop-2020', 'size': '12228'}