<jupyter_start><jupyter_text>Automated ML<jupyter_code># Updating SDK as suggested by mentor
# I ran the line and then deleted the cell´s output to declutter the notebook
!pip install --upgrade --upgrade-strategy eager azureml-sdk[automl,widgets,notebooks]
from azureml.core import Workspace, Experiment, Dataset, Datastore, Model
from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException
from azureml.widgets import RunDetails
from azureml.data.dataset_factory import TabularDatasetFactory
from azureml.train.automl import AutoMLConfig
from azureml.core.environment import Environment
from azureml.core.model import InferenceConfig
from azureml.core.webservice import AciWebservice, Webservice
import pandas as pd
import numpy as np
import joblib
import pickle
import json
import requests
ws = Workspace.from_config()
experiment_name = 'california-housing'
experiment2=Experiment(ws, experiment_name)<jupyter_output><empty_output><jupyter_text>Create or use a compute target<jupyter_code>compute_name = "nuria-p3"
try:
    compute_target = ComputeTarget(workspace=ws, name=compute_name)
    print('Using existing compute target.')
except ComputeTargetException:
    print('Creating compute target.')
    cluster_type = 1
    provisioning_config = AmlCompute.provisioning_configuration(vm_size='Standard_D12_V2', min_nodes=1, max_nodes=5)
    compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)

compute_target.wait_for_completion(show_output=True)<jupyter_output>Using existing compute target.
Succeeded
AmlCompute wait for completion finished

Minimum number of nodes requested have been provisioned<jupyter_text>Dataset OverviewFor this project, I use the California Housing dataset available in Scikit-Learn, submitted by <NAME> (<EMAIL>) on 9/Nov/99. According to the documentation, this dataset was derived from the 1990 U.S. census, using one row per census block group (which is the smallest geographical unit for which the U.S. Census Bureau publishes sample data). The dataset contains 20,640 observations on housing prices with 8 predictive atributes and the median house value for California districts, wich is the target value to be predicted using regression models. More details on the data can be found in this [notebook](./sklearn_california_dataset.ipynb)<jupyter_code># load data and create datasets for training and testing
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

def create_datasets(x,y):
    columns_x = ['MedInc','HouseAge','AveRooms','AveBedrms','Population','AveOccup','Latitude','Longitude']
    x_df = pd.DataFrame(x,columns=columns_x)  
    columns_y = ['MedHouseVal']
    y_df = pd.DataFrame(y,columns=columns_y)  
    x_tr,x_test,y_tr,y_test = train_test_split(x_df,y_df, test_size = 20, random_state=0)
    return x_tr,x_test,y_tr,y_test
  
x,y = fetch_california_housing(return_X_y=True)
x_train,x_test,y_train,y_test = create_datasets(x,y)
data = pd.concat([x_train,y_train], axis=1)
# Check if train dataset is already registered, if not, register it.
found = False
datastore=ws.get_default_datastore() 
if 'data' in ws.datasets.keys(): 
        found = True
if not found:
        TabularDatasetFactory.register_pandas_dataframe(data, datastore,'data')
ds = Dataset.get_by_name(ws,'data')<jupyter_output><empty_output><jupyter_text>NOTE: the method used above is experimental, if it does break in the future, use this procedure (create data directory if needed)                     `` data.to_csv('data/data_csv.csv')``                  `` datastore.upload(src_dir='data', target_path='data')``                                                   `` ds = Dataset.Tabular.from_delimited_files(path = [(datastore, ('data/data_csv.csv'))])``   AutoML Configuration- SettingsIn the AutoML settings for the project, I established a maximum time of one hour (to be able to finish the entire project within the four hours of the lab), I set the maximum number of concurrent iterations to five (because it has to be set, at most, at the value of the maximum nodes of the compute cluster created), and I set the primary metric to be $r^2$, to match the primary metric logged by the hyperdrive experiment.- ConfigurationIn the configuration part I set parameters for AutoML training. I included a reference to the compute target created for training, I named the task to be performed (regression) as well as the dataset and the target (label) variable, I enable early stopping to save time and resources in case the training does not improve, I set it to do automatic featurization, I established a validation size of 20% instead of number of cross-validations to match the hyperdrive experiment, and I enabled explainability on the best model. I left out the metrics goal, as it defaults to maximize, I did not include deep learning models, and I did not black-listed any algorithms (except for deep learning).<jupyter_code># TODO: Put your automl settings here
automl_settings = { "experiment_timeout_minutes": 60,
                    "max_concurrent_iterations": 5,
                    "primary_metric" : 'r2_score'}      

# TODO: Put your automl config here
automl_config = AutoMLConfig(compute_target=compute_target,
                             task = 'regression',
                             training_data = ds,
                             label_column_name ='MedHouseVal',
                             featurization = 'auto',
                             enable_early_stopping = True,
                             model_explainability = True,
                             validation_size = 0.2,
                             path = './automl',
                             debug_log = 'automl_errors.log',
                             **automl_settings)
# TODO: Submit your experiment
remote_run = experiment2.submit(automl_config)<jupyter_output>Running on remote.<jupyter_text>Run Details<jupyter_code>RunDetails(remote_run).show()
remote_run.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Best Model  Inspect Some Results From Experiment<jupyter_code># Summary of models attempted by AUTOML
remote_run.summary()
# Get all metrics for all iteartions  (I will print only 5, to save space)

N = 39             # change for number of iterations in the experiment
list_metrics = []
for i in range(N):
    try:
        r, m = remote_run.get_output(iteration = i + 1)
        temp = r.get_metrics()
        temp.pop('residuals', 'No Key found') 
        temp.pop('predicted_true', 'No Key found') 
        temp['Run ID'] = r.id
        list_metrics.append(temp)
    except:    
        pass                              ## this skips failed iterations
out = pd.DataFrame(list_metrics)
print(out.head())<jupyter_output>median_absolute_error  spearman_correlation  \
0                   0.25                  0.89   
1                   0.37                  0.79   
2                   0.31                  0.87   
3                   0.19                  0.92   
4                   0.45                  0.76   

   normalized_root_mean_squared_log_error  normalized_median_absolute_error  \
0                                    0.10                              0.05   
1                                    0.13                              0.08   
2                                    0.11                              0.06   
3                                    0.08                              0.04   
4                                    0.14                              0.09   

   root_mean_squared_error  r2_score  explained_variance  \
0                     0.52      0.79                0.79   
1                     0.66      0.66                0.66   
2                     0.58      0.74        [...]<jupyter_text>Retieve Best Model<jupyter_code># Get best model metrics
best_metrics = remote_run.get_metrics()
print(best_metrics)
# Retrive best model
best_run, fitted_model = remote_run.get_output()
# RunID of the best model
print(best_run)
# Best Model
print(fitted_model)
fitted_model.steps
# Helper function to get model.steps information
# source: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-features#featurization-transparency
from pprint import pprint

def print_model(model, prefix=""):
    for step in model.steps:
        print(prefix + step[0])
        if hasattr(step[1], 'estimators') and hasattr(step[1], 'weights'):
            pprint({'estimators': list(
                e[0] for e in step[1].estimators), 'weights': step[1].weights})
            print()
            for estimator in step[1].estimators:
                print_model(estimator[1], estimator[0] + ' - ')
        else:
            pprint(step[1].get_params())
            print()

print_model(fitted_model)
fitted_model.named_steps['datatransformer'].get_featurization_summary()<jupyter_output><empty_output><jupyter_text>Save and Register Best Model<jupyter_code>#TODO: Save the best model
joblib.dump(fitted_model, filename='automl/best-automl.joblib')
# register model
model = remote_run.register_model(model_name='best-automl')
# look at the model
print(model.name,model.id,model.version,sep='\t')
# download the model
model.download(target_dir='automl/model.pkl', exist_ok=False, exists_ok=None)<jupyter_output><empty_output><jupyter_text>Try Best Model on the Test DatasetWhen loading and registering the dataset, I left part of the data to be used for testing. It is important to test the model on data not previously either for training or for validation. These are the data I use here.<jupyter_code>!pip install seaborn
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style="whitegrid")
from sklearn.metrics import r2_score
try:
    y_pred = fitted_model.predict(x_test)
    residuals = y_test['MedHouseVal'] - y_pred
    r2 = r2_score(y_test, y_pred)
    print('R2 score for test data: ',r2)
except Exception as e:
    print(str(e))
# Plot actual vs. fitted values for the test dataset
plt.figure(figsize=(6, 6))
sns.regplot(x=y_test['MedHouseVal'], y=y_pred, x_estimator=np.mean, ci=None, scatter=False, color='darkred')
sns.regplot(x=y_test['MedHouseVal'], y=y_pred, x_estimator=np.mean, ci=None, fit_reg=False, color='lightcoral')
plt.xlabel('Actual House Prices',fontsize = 15)
plt.ylabel('Predicted House Prices',fontsize = 15)
plt.show()
# Distribution of the residuals
plt.figure(figsize=(6, 6))
ax = sns.histplot(data=residuals, kde=True, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>Model DeploymentThe model is already registered, so I only need to load it.<jupyter_code># Retrive the registered model
model = Model(ws, 'best-automl')
# Environment
try:
    best_run.download_file('outputs/conda_env_v_1_0_0.yml', 'automl/auto_conda_env.yml')  
    env = Environment.from_conda_specification('myenv','automl/auto_conda_env.yml')
except Exception as e:
    print(str(e))
# Scoring file
try:
    best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'automl/score.py')  
except Exception as e:
    print(str(e)) 
# inference config
inference_config = InferenceConfig(entry_script='automl/score.py', environment=env)

# deployment config
deployment_config = AciWebservice.deploy_configuration(cpu_cores = 1, memory_gb = 1, auth_enabled=True, enable_app_insights=True)
# deploy
try:
    service = Model.deploy(ws, "california-housing", [model], inference_config, deployment_config, overwrite=True)
    service.wait_for_deployment(show_output = True)
    print(service.state)
except Exception as e:
    print(str(e))<jupyter_output>Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running...................................................................................................................................................................................................................
Succeeded
ACI service creation operation finished, operation "Succeeded"
Healthy<jupyter_text>Test the service and the endpoint  Get service information<jupyter_code>scoring_uri = service.scoring_uri
print('URI: ', scoring_uri)
keys = service.get_keys()
print('KEYS: ', keys)
swag = service.swagger_uri
print(swag)
serv = service.serialize()
#print(serv)<jupyter_output><empty_output><jupyter_text>try the service<jupyter_code>test = x_test[0:10]
input_payload = json.dumps({'data': test.to_numpy().tolist()})
output = service.run(input_payload)
print(output)<jupyter_output>{"result": [1.4618726543302587, 2.6647999671275993, 1.4843392111378706, 0.7611031035869795, 4.598881336878318, 1.6788095733029562, 2.512055535799478, 3.724994229352508, 3.1593949606066145, 2.098471795122479]}<jupyter_text>send request to the endpoint<jupyter_code># prepare data
x_test_json = test[0:10].to_json(orient='records') 
data = "{\"data\": " + x_test_json +"}"
# prepare headers
headers = {'Content-Type':'application/json',  'Authorization':('Bearer '+ keys[0])}
# send request
resp = requests.post(scoring_uri, data, headers=headers)
print(resp)
# print result
y_pred = json.loads(json.loads(resp.text))['result']
print('result:', y_pred)<jupyter_output>result: [1.4618726543302587, 2.6647999671275993, 1.4843392111378706, 0.7611031035869795, 4.598881336878318, 1.6788095733029562, 2.512055535799478, 3.724994229352508, 3.1593949606066145, 2.098471795122479]<jupyter_text>Print service logs<jupyter_code>print(service.get_logs())<jupyter_output>2021-01-26T20:15:11,344479271+00:00 - gunicorn/run 
2021-01-26T20:15:11,345360572+00:00 - rsyslog/run 
2021-01-26T20:15:11,345679372+00:00 - iot-server/run 
2021-01-26T20:15:11,353103378+00:00 - nginx/run 
rsyslogd: /azureml-envs/azureml_4f26b5b9300bb3a3b89415c34dafac63/lib/libuuid.so.1: no version information available (required by rsyslogd)
/usr/sbin/nginx: /azureml-envs/azureml_4f26b5b9300bb3a3b89415c34dafac63/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_4f26b5b9300bb3a3b89415c34dafac63/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_4f26b5b9300bb3a3b89415c34dafac63/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_4f26b5b9300bb3a3b89415c34dafac63/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml[...]<jupyter_text>CLEAN-UP RESOURCES<jupyter_code># Delete web service
service.delete()
# Delete cluster
compute_target.delete()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10763488293179505, 'fraction_numerical': 0.049202578893790296, 'mean_word_length': 3.4613987284287013, 'pattern_counts': {'":': 4, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '482792', 'license': 'MIT', 'n_tokens_mistral': 5043, 'n_tokens_neox': 4573, 'n_words': 1161, 'path': 'capstoneProject/automl.ipynb', 'repo_name': 'malet-ds/MLEngineerWithAzure', 'size': '14757'}