<jupyter_start><jupyter_text>Use CasePredict Total Salary of San Francisco Government employees based on what type of job they are working.![dataset](./dataset.png "Dataset") Prerequisities- microsoft account- active subscription on given microsoft accountThis is necessary to log in to Azure Portal and start creating your new resource.1. Firstly create **Machine Learning** resource.2. After successfully creating resource , open it and go to resource overview.  Copy **Studio Web Url**.![Studio web url](./aml_url.png "Studio web url") 3. Log in to **ml.azure.com**, click on **Notebooks** tab.4. Import all files from [Jupyter AML Notebook - SF Salary](https://github.com/kolendomichal/AI-on-Microsoft-Azure/blob/master/Automated%20Machine%20Learning/Jupyter%20AML%20Notebook)5. Select the notebook and run all cells.  Authenticate to Azure
In this step you will be prompted to authenticate to your Azure account.    
After inserting provided code, Jupyter Notebook will gain access to your Azure resources.<jupyter_code># azureml-core of version 1.0.72 or higher is required
from azureml.core import Workspace, Dataset

# Get Workspace defined in by default config.json file
ws = Workspace.from_config()<jupyter_output><empty_output><jupyter_text>In order to run the algorithm, download the [Employee Compensation in SF dataset](https://data.world/data-society/employee-compensation-in-sf).  
Then, import it into **Azure Machine Learning Studio** in **Datasets** tab.  
Azure Machine Learning studio will automatically set type of the column and recognize how the file is formatted.


Give the dataset a name: **"San-Francisco-Employee-Salaries"**. It will be used in further steps.

![import_dataset_to_workspace](./import_dataset_to_workspace.png "Importing Dataset")  Display loaded dataset
In step below, after setting dataset_name, Jupyter Notebook will load the data from your **Workspace** and display few rows for confirmation.<jupyter_code># Load Data
dataset_name = 'San-Francisco-Employee-Salaries'
aml_dataset = ws.datasets[dataset_name]

# Use Pandas DataFrame just to sneak peak some data and schema
full_df = aml_dataset.to_pandas_dataframe()
full_df.head()
# Use Pandas DataFrame just to investigate the dataset's schema and info
full_df.describe()<jupyter_output><empty_output><jupyter_text>Clean up the initial dataset
As the dataset is vast, not all information is needed for the model to be trained.  
If we omit redundant informaton, trained model will be much more efficient.

We will be dropping columns:
- *Employee identifier* since it is merely an identifier
- *Department Code* since its' equivalent is *Department* 
- *Organization Group Code* since *Organization Group*
- *Union Code* since its' equivalent is *Union*
- *Job Family Code* since its' equivalent is *Job Family*
- *Job Code* since its' equivalent is *Job*<jupyter_code>aml_dataset = aml_dataset.drop_columns([
    'Employee Identifier',
    'Department Code',
    'Organization Group Code',
    'Union Code',
    'Job Family Code',
    'Job Code'])

full_df = aml_dataset.to_pandas_dataframe()
full_df.describe()<jupyter_output><empty_output><jupyter_text>Split data into 'Test' and 'Train'
A model has to be trained on a subset of the data.  
After its' training, we will want to test the precision of our model - that is where test data will be used.  

Code below:
- Splits data into two sub-datasets *train* and *test*. 
  - Ratio for datasets will be respectively **0.9 : 0.1**<jupyter_code># Split using Azure Tabular Datasets
# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py#random-split-percentage--seed-none-

train_dataset, test_dataset = aml_dataset.random_split(0.9, seed=1)

# Use Pandas DF only to check the data
train_dataset_df = train_dataset.to_pandas_dataframe()
test_dataset_df = test_dataset.to_pandas_dataframe()

print(train_dataset_df.describe())<jupyter_output>Salaries       Overtime  Other Salaries   Total Salary  \
count  262643.000000  262643.000000   262643.000000  262643.000000   
mean    63193.957098    4409.742251     3774.593238   71378.292586   
std     44639.959738   11080.242853     7648.548279   52191.296475   
min    -68771.780000  -12308.660000   -19131.100000  -68771.780000   
25%     23277.000000       0.000000        0.000000   25280.280000   
50%     62482.600000       0.000000      700.000000   67805.720000   
75%     92890.230000    2777.740000     4619.535000  103180.000000   
max    507831.600000  227313.620000   342802.630000  507831.600000   

          Retirement  Health/Dental  Other Benefits  Total Benefits  \
count  262643.000000  262643.000000   262643.000000   262643.000000   
mean    12935.454862    8924.648188     4646.259312    26506.362362   
std      9780.529058    4899.564239     3788.766179    16797.345943   
min    -30621.430000   -1427.890000   -10469.090000   -21295.150000   
25%      3550.[...]<jupyter_text>Create Azure Machine Learning compute target
We want to utilize remote resources for doing the "heavy lifting".
For that purpose we will create a remote compute instance - **salarySF**.  
This name will be used in next cell instance.  


Compute instance can be created manually from **Compute** tab or by running the code below.
![compute_instance](./compute.png "Compute Instance")  List remote Azure Machine Learning compute targets
To confirm that our compute resource has been created, let's display all available compute targets.<jupyter_code>from azureml.core.compute import AmlCompute
from azureml.core.compute import ComputeTarget

ComputeTarget.list(ws)<jupyter_output><empty_output><jupyter_text>Connect to Remote Azure Machine Learning Computer
Now we have to select the compute target we will use for training our model.  
We will select **salarySF** compute instance created in previous step.<jupyter_code># Define remote compute target to use
# Further docs on Remote Compute Target: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-remote

# Choose a name for your cluster.
amlcompute_cluster_name = "salarySF"

found = False
# Check if this compute target already exists in the workspace.
cts = ws.compute_targets

if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'ComputeInstance':
     found = True
     print('Found existing training cluster.')
     # Get existing cluster
     # Method 1:
     aml_remote_compute = cts[amlcompute_cluster_name]
     # Method 2:
     # aml_remote_compute = ComputeTarget(ws, amlcompute_cluster_name)
    
if not found:
     print('Creating a new training cluster...')
     provisioning_config = AmlCompute.provisioning_configuration(vm_size = "STANDARD_D13_V2", # for GPU, use "STANDARD_NC12"
                                                                 #vm_priority = 'lowpriority', # optional
                                                                 max_nodes = 20)
     # Create the cluster.
     aml_remote_compute = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)
    
print('Checking cluster status...')
# Can poll for a minimum number of nodes and for a specific timeout.
# If no min_node_count is provided, it will use the scale settings for the cluster.
aml_remote_compute.wait_for_completion(show_output = True)
# For additional details of current AmlCompute status:
aml_remote_compute.get_status().serialize()<jupyter_output><empty_output><jupyter_text>List and select primary metric to drive the AutoML regression problem
The most important part of doing any kind of machine learning is knowing what problem we are solving.  
In our case, we are trying to predict salary of San Francisco employees - in that case we are facing **Regression** problem.<jupyter_code>from azureml.train import automl

# List of possible primary metrics is here:
# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric
    
# Get a list of valid metrics for your given task
automl.utilities.get_primary_metrics('Regression')

# We'll use 'r2_score' as primary metric (Closer to 1.00 is better, as correlation is higher)<jupyter_output><empty_output><jupyter_text>Define AutoML Experiment settings (With AML Remote Compute)
Lets define the run **configuration**.  
We set:
- regression as task we are doing
- r2_score as a primary metric to be calculated
   - r2 score is used for checking whether some variables are correlated. So if it is near 1 - variables are perfectly correlated.
- *Total Salary* as label column<jupyter_code>import logging
import os

from azureml.train.automl import AutoMLConfig

project_folder = './mchkolendo'
os.makedirs(project_folder, exist_ok=True)

automl_config = AutoMLConfig(compute_target=aml_remote_compute,
                             task='regression',
                             primary_metric='r2_score',
                             experiment_timeout_minutes=15,                            
                             training_data=train_dataset,
                             label_column_name="Total Salary",
                             n_cross_validations=5,
                             enable_early_stopping=True,
                             featurization='auto',
                             debug_log='automated_ml_errors.log',
                             verbosity=logging.INFO,
                             path=project_folder
                             )


# https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig?view=azure-ml-py
# Explanation of Settings: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#configure-your-experiment-settings
# AutoMLConfig info on: 
# https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig<jupyter_output><empty_output><jupyter_text>Run experiment with multiple child runs
Now it is time to run our experiment.  
Beware that it can take up to 30 mins if you decide to run it on the lowest CPU compute target.<jupyter_code>from azureml.core import Experiment
from datetime import datetime

now = datetime.now()
time_string = now.strftime("%m-%d-%Y-%H")
experiment_name = "regress-automl-remote-{0}".format(time_string)
print(experiment_name)

experiment = Experiment(workspace=ws, name=experiment_name)

import time
start_time = time.time()
            
run = experiment.submit(automl_config, show_output=True)

print('Manual run timing: --- %s seconds needed for running the whole Remote AutoML Experiment ---' % (time.time() - start_time))<jupyter_output>regress-automl-remote-12-26-2020-11
Running on remote.
No run_configuration provided, running on salarySF with default configuration
Running on remote compute: salarySF
Parent Run ID: AutoML_d4a44bd2-f951-428a-bd04-ed80fa932d00

Current status: FeaturesGeneration. Generating features for the dataset.
Current status: DatasetFeaturization. Beginning to fit featurizers and featurize the dataset.
Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.
Current status: ModelSelection. Beginning model selection.

****************************************************************************************************
DATA GUARDRAILS: 

TYPE:         Missing feature values imputation
STATUS:       PASSED
DESCRIPTION:  No feature missing values were detected in the training data.
              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization

******************************************************************************************[...]<jupyter_text>Example of a result of the 28 min run
![run_result](./run_result.png "Run result")  Measure total time needed for the whole AutoML process<jupyter_code>import time
import datetime as dt

run_details = run.get_details()

# Like: 2020-01-12T23:11:56.292703Z
end_time_utc_str = run_details['endTimeUtc'].split(".")[0]
start_time_utc_str = run_details['startTimeUtc'].split(".")[0]
timestamp_end = time.mktime(datetime.strptime(end_time_utc_str, "%Y-%m-%dT%H:%M:%S").timetuple())
timestamp_start = time.mktime(datetime.strptime(start_time_utc_str, "%Y-%m-%dT%H:%M:%S").timetuple())

parent_run_time = timestamp_end - timestamp_start
print('Run Timing: --- %s seconds needed for running the whole Remote AutoML Experiment ---' % (parent_run_time))<jupyter_output>Run Timing: --- 1687.0 seconds needed for running the whole Remote AutoML Experiment ---<jupyter_text>Find the Best Model 
Now , let us select the best model, which we will use to on our **test** data.<jupyter_code>best_run, fitted_model = run.get_output()
print(best_run)
print(fitted_model)<jupyter_output>Run(Experiment: regress-automl-remote-12-26-2020-11,
Id: AutoML_d4a44bd2-f951-428a-bd04-ed80fa932d00_4,
Type: azureml.scriptrun,
Status: Completed)
RegressionPipeline(pipeline=Pipeline(memory=None,
                                     steps=[('datatransformer',
                                             DataTransformer(enable_dnn=None,
                                                             enable_feature_sweeping=None,
                                                             feature_sweeping_config=None,
                                                             feature_sweeping_timeout=None,
                                                             featurization_config=None,
                                                             force_text_dnn=None,
                                                             is_cross_validation=None,
                                                             is_onnx_compatible=None,
                                           [...]<jupyter_text>Make predictions
 Extract X values (feature columns) from test dataset and convert to NumPi array for predicting
We will extract the data which will be used for testing, except **Total Salary** column, which our algorithm  
will try to predict.<jupyter_code>import pandas as pd

#Remove Label/y column
if 'Total Salary' in test_dataset_df.columns:
    y_test_df = test_dataset_df.pop('Total Salary')

x_test_df = test_dataset_df<jupyter_output><empty_output><jupyter_text>Make the actual predictions<jupyter_code># Try the best model
y_predictions = fitted_model.predict(x_test_df)

print('10 predictions: ')
print(y_predictions[:10])
y_predictions.shape<jupyter_output><empty_output><jupyter_text>Calculate the R2 Score with Test Dataset
![r2_score](./r2_score.png "r2_score")
As we can see on the screen up, our r2_score is near 100%,  
which means a predicted value is nearly identical.<jupyter_code>from sklearn.metrics import r2_score

print('R2 Score:')
r2_score(y_test_df, y_predictions)<jupyter_output>R2 Score:
{'fraction_non_alphanumeric': 0.09162303664921466, 'fraction_numerical': 0.045576587461404215, 'mean_word_length': 3.3260743321718933, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '647175', 'license': 'MIT', 'n_tokens_mistral': 4727, 'n_tokens_neox': 4217, 'n_words': 1291, 'path': 'Automated Machine Learning/Jupyter AML Notebook/Salary_Prediction_SF.ipynb', 'repo_name': 'kolendomichal/AI-on-Microsoft-Azure', 'size': '14898'}