<jupyter_start><jupyter_text>AutoML Classification experiment using Local Compute and Pandas DataFrames Data: IBM Employee Attrition dataset loaded from Azure ML Dataset   Get Azure ML Workspace to use<jupyter_code># azureml-core of version 1.0.72 or higher is required
from azureml.core import Workspace, Dataset

# Get Workspace defined in by default config.json file
ws = Workspace.from_config()<jupyter_output><empty_output><jupyter_text>Load data from Azure ML Datasets into Pandas DataFrame<jupyter_code># Load Data
aml_dataset = ws.datasets['IBM-Employee-Attrition']

# Use Pandas DataFrame just to sneak peak some data and schema
full_df = aml_dataset.to_pandas_dataframe()
# .to_pandas_dataframe().dropna()
full_df.head(5)
# Use Pandas DataFrame just to investigate the dataset's schema and info
full_df.describe()<jupyter_output><empty_output><jupyter_text>Clean up the initial dataset (Using related Pandas DataFrame)<jupyter_code># Dropping Employee count as all values are 1 and hence attrition is independent of this feature
full_df = full_df.drop(['EmployeeCount'], axis=1)
# Dropping Employee Number since it is merely an identifier
full_df = full_df.drop(['EmployeeNumber'], axis=1)

full_df = full_df.drop(['Over18'], axis=1)

# Since all values are 80
full_df = full_df.drop(['StandardHours'], axis=1)<jupyter_output><empty_output><jupyter_text>Split original dataset in test/train sets using Scikit-Learn train_test_split function<jupyter_code>from sklearn.model_selection import train_test_split

# Split using ScikitLearn train_test_split function using Dataframes
# Will use test dataframe at the end, without AutoML, just for testing predictions with the model

# Only split in test/train
train_df, test_df = train_test_split(full_df, test_size=0.2, random_state=1)
train_df.describe()

# Split in x/y and test/train
# y_df = full_df.pop("Attrition")
# x_df = full_df
# x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.1, random_state=1)

#Another possibility would be to split using the Azure ML Datasets (Better for Remote Compute): 
# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py#random-split-percentage--seed-none-
test_df.describe()<jupyter_output><empty_output><jupyter_text>List and select primary metric to drive the AutoML classification problem<jupyter_code>from azureml.train import automl

# List of possible primary metrics is here:
# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric
    
# Get a list of valid metrics for your given task
automl.utilities.get_primary_metrics('classification')

# I'll use 'accuracy' as primary metric (Closer to 1.00 is better)<jupyter_output><empty_output><jupyter_text>Define AutoML Experiment settings<jupyter_code>import logging

# Explanation of Settings: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#configure-your-experiment-settings

# AutoMLConfig info on: 
# https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig

# You can provide additional settings as a **kwargs parameter for the AutoMLConfig object
# automl_settings = {
#     "whitelist_models": 'XGBoostClassifier'
# }

from azureml.train.automl import AutoMLConfig

automl_config = AutoMLConfig(task='classification',
                             primary_metric='accuracy',
                             # experiment_timeout_minutes= 20,                            
                             training_data=train_df,
                             label_column_name="Attrition",
                             # X=x_train.values,             # X parameter is deprecated 
                             # y=y_train.values.flatten(),   # y parameter is deprecated 
                             n_cross_validations= 5,
                             # blacklist_models='XGBoostClassifier', 
                             # iteration_timeout_minutes= 5,                                                    
                             enable_early_stopping= True,
                             featurization= 'auto',
                             debug_log='automated_ml_errors.log',
                             verbosity= logging.INFO,
                             # **automl_settings
                             )

# WARNING: If using X and y parameters (deprecated) you get the following warning
# WARNING - The AutoMLConfig inputs you have specified will soon be deprecated. Please use the AutoMLConfig shown in our documentation: https://aka.ms/AutoMLConfig<jupyter_output><empty_output><jupyter_text>Run Experiment with multiple child runs under the covers<jupyter_code>from azureml.core import Experiment
from datetime import datetime

now = datetime.now()
time_string = now.strftime("%m-%d-%Y-%H")
#time_string = now.strftime("%m-%d-%Y-%H-%M")
print(time_string)
experiment_name = "classif-automl-local-{0}".format(time_string)
print(experiment_name)

experiment = Experiment(workspace=ws, 
                        name=experiment_name)

run = experiment.submit(automl_config, show_output=True)<jupyter_output>01-12-2020-21
classif-automl-local-01-12-2020-21
Running on local machine
Parent Run ID: AutoML_0ec54b76-21b0-4bb2-9de6-be311285ba89

Current status: DatasetFeaturization. Beginning to featurize the dataset.
Current status: DatasetEvaluation. Gathering dataset statistics.
Current status: FeaturesGeneration. Generating features for the dataset.
Current status: DatasetFeaturizationCompleted. Completed featurizing the dataset.
Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.

****************************************************************************************************
DATA GUARDRAILS SUMMARY:
For more details, use API: run.get_guardrails()

TYPE:         Class balancing detection
STATUS:       PASSED
DESCRIPTION:  Classes are balanced in the training data.

TYPE:         Missing values imputation
STATUS:       PASSED
DESCRIPTION:  There were no missing values found in the training data.

TYPE:         High cardinality feature detection
STAT[...]<jupyter_text>Explore results with Widget<jupyter_code># Explore the results of automatic training with a Jupyter widget: https://docs.microsoft.com/en-us/python/api/azureml-widgets/azureml.widgets?view=azure-ml-py
from azureml.widgets import RunDetails
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>Retrieve the 'Best Model'<jupyter_code>best_run, fitted_model = run.get_output()
print(best_run)
print('--------')
print(fitted_model)<jupyter_output>Run(Experiment: classif-automl-local-01-12-2020-21,
Id: AutoML_0ec54b76-21b0-4bb2-9de6-be311285ba89_30,
Type: None,
Status: Completed)
--------
Pipeline(memory=None,
     steps=[('datatransformer', DataTransformer(enable_dnn=None, enable_feature_sweeping=None,
        feature_sweeping_config=None, feature_sweeping_timeout=None,
        featurization_config=None, is_cross_validation=None,
        is_onnx_compatible=None, logger=None, observer=None, task=None)), ('pref...  flatten_transform=None,
               weights=[0.125, 0.125, 0.125, 0.125, 0.125, 0.25, 0.125]))])<jupyter_text>See files associated with the 'Best run'<jupyter_code>print(best_run.get_file_names())

# best_run.download_file('azureml-logs/70_driver_log.txt')<jupyter_output>['accuracy_table', 'confusion_matrix', 'outputs/conda_env_v_1_0_0.yml', 'outputs/env_dependencies.json', 'outputs/model.pkl', 'outputs/scoring_file_v_1_0_0.py', 'pipeline_graph.json']<jupyter_text>Download run model .pkl file and related files to see Conda and Environment dependencies used by AutoML<jupyter_code># Download the run's files
best_run.download_file('outputs/model.pkl')
best_run.download_file('outputs/conda_env_v_1_0_0.yml')
best_run.download_file('outputs/env_dependencies.json')
best_run.download_file('outputs/scoring_file_v_1_0_0.py')
best_run.download_file('pipeline_graph.json')<jupyter_output><empty_output><jupyter_text>Register the modelOnce you've trained the model, you can save and register it to your workspace. Model registration lets you store and version your models in your workspace to simplify model management and deployment.Running the following code will register the model to your workspace, and will make it available to reference by name in remote compute contexts or deployment scripts.<jupyter_code>from azureml.core.model import Model

model_reg = best_run.register_model(model_name='classif-empl-automl-local',      # Name of the registered model in your workspace.
                                    description='Binary classification model for employees attrition. From AutoML local training',
                                    model_path='outputs/model.pkl',              # Local file to upload and register as a model.
                                    model_framework=Model.Framework.SCIKITLEARN, # Framework used to create the model.
                                    model_framework_version='0.XX.X',            # Version of scikit-learn used to create the model.
                                    tags={'ml-task': "binary-classification", 'business-area': "HR"},
                                    properties={'pandas-version': "0.XX.X"},
                                    sample_input_dataset=aml_dataset
                              )

print(model_reg)

# (Q1:) How can we know what frameworks/libraries versions were used by AutoML when training the model?
#         - If investigating the outputs/conda_env_v_1_0_0.yml file, some libraries like Pandas and Scikit-Learn are not showing any version...

# (Q2:) Why best_run.get_environment() fails?
# best_run_environment = best_run.get_environment() 
# print(best_run_environment)<jupyter_output>Model(workspace=Workspace.create(name='cesardl-automl-northcentralus-ws', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo'), name=classif-empl-automl-local, id=classif-empl-automl-local:7, version=7, tags={'ml-task': 'binary-classification', 'business-area': 'HR'}, properties={'pandas-version': '0.XX.X'})<jupyter_text>Make Predictions  Extract X values (feature columns) from test dataset and convert to NumPi array for predicting<jupyter_code>import pandas as pd

#Remove Label/y column
if 'Attrition' in test_df.columns:
    y_test_df = test_df.pop('Attrition')

x_test_df = test_df
x_test_df.describe()<jupyter_output><empty_output><jupyter_text>Make the actual Predictions  (Optional) Download Model from Registry and load in-memory<jupyter_code>print(Model.get_model_path('classif-empl-automl-local', _workspace=ws))

model_definition_from_registry = Model(ws,'classif-empl-automl-local')
model_definition_from_registry.download(target_dir='.', exist_ok=True)
print(model_definition_from_registry)
print('-------')

# Load the model into memory
import joblib
fitted_model = joblib.load('model.pkl')
print(fitted_model)
# Try the best model making predictions with the test dataset
y_predictions = fitted_model.predict(x_test_df)

print('10 predictions: ')
print(y_predictions[:10])
y_predictions.shape<jupyter_output><empty_output><jupyter_text>Calculate the Accuracy with Test Dataset (Not used for training)<jupyter_code>from sklearn.metrics import accuracy_score

print('Accuracy:')
accuracy_score(y_test_df, y_predictions)<jupyter_output>Accuracy:
{'fraction_non_alphanumeric': 0.10105963744636133, 'fraction_numerical': 0.0155880549960592, 'mean_word_length': 4.0778123610493555, 'pattern_counts': {'":': 1, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '573798', 'license': 'MIT', 'n_tokens_mistral': 3569, 'n_tokens_neox': 3373, 'n_words': 826, 'path': '2-training-and-interpretability/2.3-automl-training/local-compute/binayclassification-employee-attrition-autoaml-local-compute.ipynb', 'repo_name': 'gvashishtha/azureml-workshop-2019', 'size': '11419'}