<jupyter_start><jupyter_text>Automated MLwe start by importing all the dependencies that we will need to complete the project.<jupyter_code>from azureml.core import Workspace, Experiment
from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException
from azureml.data.dataset_factory import TabularDatasetFactory
import pandas as pd
from azureml.train.automl import AutoMLConfig
from azureml.widgets import RunDetails
from azureml.core.model import Model
from azureml.core.model import InferenceConfig
from azureml.core import Workspace, Environment
from azureml.core import Model
from azureml.core.webservice import AciWebservice, Webservice
import json
import joblib<jupyter_output><empty_output><jupyter_text>Dataset OverviewWe first of all define the workspace and create the experiment 'Parkinson-classification-AutoML' We then access to the dataset from the external link (placed in the github)We move then to prepare the data(read the data using TabularDatasetFactory  then transforme it to a dataframe then store it in a csv file ) and send it to the default datastore so that we can use it in the Automl Config To finally create the compute cluster The dataset used in this notebook  is composed of a range of biomedical voice measurements from 31 people, 23 with Parkinson's disease (PD). Each column in the data is a particular voice measure, and each row corresponds one of 195 voice recording from these individuals ("name" column)<jupyter_code>#Connection to the workspace and definition of the experiment 
ws = Workspace.from_config()
experiment_name = 'Parkinson-classification-AutoML'

experiment=Experiment(workspace=ws, name=experiment_name)

print('Workspace name: ' + ws.name, 
      'Azure region: ' + ws.location, 
      'Subscription id: ' + ws.subscription_id, 
      'Resource group: ' + ws.resource_group, sep = '\n')

run = experiment.start_logging()
#Get the data from github
path_url = 'https://raw.githubusercontent.com/hananeouhammouch/Parkinsons-detection/master/parkinsons.data'
ds = TabularDatasetFactory.from_delimited_files(path = path_url)
ds.to_pandas_dataframe().head()
#Prepare the data and send it to the datastore
df=ds.to_pandas_dataframe()
df.drop("name", inplace=True, axis=1)  
df.to_csv("./training_data.csv")
datastore = ws.get_default_datastore()
datastore.upload(src_dir = "./", target_path = "data/")
training_data = TabularDatasetFactory.from_delimited_files(path = [(datastore, ("data/training_data.csv"))])

df.head()
#Creation of the compute-cluster
amlcompute_cluster_name = "cpu-cluster"

try:
    aml_compute = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS3_V2',
                                                           max_nodes=5)
    aml_compute = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)

aml_compute.wait_for_completion(show_output=True , min_node_count = 1, timeout_in_minutes = 2)<jupyter_output>Creating
Succeeded.....................
AmlCompute wait for completion finished

Wait timeout has been reached
Current provisioning state of AmlCompute is "Succeeded" and current node count is "0"<jupyter_text>AutoML ConfigurationIn this section we configure the automnl setting and submit the experiment And Bellow is the description/Reason for the configuration used in Automl|Setting |Reasons ?||-|-||**experiment_timeout_minutes**|Maximum amount of time in minutes that all iterations combined can take before the experiment terminates (15 minute because the dataset include only 195 lines)||**max_concurrent_iterations**|To help manage child runs  in parallele mode and when they can be performed, we create a dedicated cluster per experiment, and match the number of this setting (4) to the number of nodes in the cluster(5-1))||**n_cross_validations**|Number of cross validation (5) splits to ensure that they will be no overfiting ||**primary_metric**|This is the metric that we want to optimize (accuracy) ||**task**|classification ||**compute_target**|To define the compute cluster we will be using ||**training_data**|To specify the training dataset stored in the datastore  ||**label_column_name**|To specify the dependent variable that we are trying to classify |<jupyter_code># Automl settings 
automl_settings = {
    "experiment_timeout_minutes" :15,
    "max_concurrent_iterations": 4,
    "n_cross_validations": 5,
    "primary_metric": 'accuracy',
}

# Automl config 
automl_config = AutoMLConfig(
    task="classification",
    compute_target=aml_compute,
    training_data=training_data,
    label_column_name="status",
    **automl_settings
)
# Submit experiment
auto_ml_run = experiment.submit(config = automl_config, show_output = True)<jupyter_output>Running on remote.
No run_configuration provided, running on cpu-cluster with default configuration
Running on remote compute: cpu-cluster
Parent Run ID: AutoML_15b2b046-252e-4a41-80fa-7fda6fffe1a0

Current status: FeaturesGeneration. Generating features for the dataset.
Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.
Current status: ModelSelection. Beginning model selection.

****************************************************************************************************
DATA GUARDRAILS: 

TYPE:         Class balancing detection
STATUS:       PASSED
DESCRIPTION:  Your inputs were analyzed, and all classes are balanced in your training data.
              Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData

****************************************************************************************************

TYPE:         Missing feature values imputation
STATUS:       PASSED
DESCRIPTION:  No feature missing values we[...]<jupyter_text>Run Details`RunDetails` widgets used here to show the different experiments.<jupyter_code>RunDetails(auto_ml_run).show()
auto_ml_run.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Best ModelThe best model from the automl experiments is retrieved and  all the properties of the model are displayed<jupyter_code># Retrieve the best automl model.
best_run, fitted_model = auto_ml_run.get_output()
print(best_run)
print(fitted_model)

#Register the model
description ='Parkinson detection classification '
model_name='Parkinson-detection-automl'
model_path='./'
tags = None
model = auto_ml_run.register_model(model_name = model_name, description = description , tags = tags)
print(auto_ml_run.model_id)<jupyter_output>Run(Experiment: Parkinson-classification-AutoML,
Id: AutoML_15b2b046-252e-4a41-80fa-7fda6fffe1a0_0,
Type: azureml.scriptrun,
Status: Completed)
Pipeline(memory=None,
         steps=[('datatransformer',
                 DataTransformer(enable_dnn=None, enable_feature_sweeping=None,
                                 feature_sweeping_config=None,
                                 feature_sweeping_timeout=None,
                                 featurization_config=None, force_text_dnn=None,
                                 is_cross_validation=None,
                                 is_onnx_compatible=None, logger=None,
                                 observer=None, task=None, working_dir=None)),
                ('MaxAbsScaler', MaxAbsScaler(copy...
                 LightGBMClassifier(boosting_type='gbdt', class_weight=None,
                                    colsample_bytree=1.0,
                                    importance_type='split', learning_rate=0.1,
                                [...]<jupyter_text>Model DeploymentIn the cell below we first  register the model, then define the environement to finally  create the inference config and deploy the model as a web service.<jupyter_code>#Save the best model for the deployement
import os
os.makedirs('./amlmodel', exist_ok=True)

best_run.download_file('/outputs/model.pkl',os.path.join('./amlmodel','automl_best_model_park.pkl'))

for f in best_run.get_file_names():
    if f.startswith('outputs'):
        output_file_path = os.path.join('./amlmodel', f.split('/')[-1])
        print(f'Downloading from {f} to {output_file_path} ...')
        best_run.download_file(name=f, output_file_path=output_file_path)
#Register the best model for the deployement

model=best_run.register_model(
            model_name = 'automl-bestmodel-parkinson', 
            model_path = './outputs/model.pkl',
            model_framework=Model.Framework.SCIKITLEARN,
            description='Parkinson detection'
)
# Download the conda environment file and define the environement
best_run.download_file('outputs/conda_env_v_1_0_0.yml', 'conda_env.yml')
myenv = Environment.from_conda_specification(name = 'myenv',
                                             file_path = 'conda_env.yml')
# download the scoring file produced by AutoML
best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'score_auto_park.py')

# set inference config
inference_config = InferenceConfig(entry_script= 'score_auto_park.py',
                                    environment=myenv)
# set Aci Webservice config
aci_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1, auth_enabled=True)
# deploye the model as a web service 
service = Model.deploy(workspace=ws, 
                       name='automl-bestmodel-parkinso', 
                       models=[model], 
                       inference_config=inference_config,
                       deployment_config=aci_config,
                       overwrite=True)
service
# wait for deployment to finish and display the scoring uri and swagger uri
service.wait_for_deployment(show_output=True)

print('Service state:')
print(service.state)

print('Scoring URI:')
print(service.scoring_uri)

print('Swagger URI:')
print(service.swagger_uri)
# select 3  samples from the dataframe
x_df=df.sample(3)
y_df = x_df.pop('status')

x_df['Column1'] = 0.0

# convert the records to a json data file
recored=x_df.to_dict(orient='records')

scoring_json = json.dumps({'data': recored})
print(scoring_json)<jupyter_output>{"data": [{"MDVP:Fo(Hz)": 148.09, "MDVP:Fhi(Hz)": 162.824, "MDVP:Flo(Hz)": 67.343, "MDVP:Jitter(%)": 0.00762, "MDVP:Jitter(Abs)": 5e-05, "MDVP:RAP": 0.00467, "MDVP:PPQ": 0.00354, "Jitter:DDP": 0.014, "MDVP:Shimmer": 0.05428, "MDVP:Shimmer(dB)": 0.497, "Shimmer:APQ3": 0.03357, "Shimmer:APQ5": 0.02494, "MDVP:APQ": 0.03635, "Shimmer:DDA": 0.1007, "NHR": 0.02431, "HNR": 21.718, "RPDE": 0.487407, "DFA": 0.727313, "spread1": -6.261141, "spread2": 0.120956, "D2": 2.137075, "PPE": 0.141958, "Column1": 0.0}, {"MDVP:Fo(Hz)": 128.451, "MDVP:Fhi(Hz)": 150.449, "MDVP:Flo(Hz)": 75.632, "MDVP:Jitter(%)": 0.01551, "MDVP:Jitter(Abs)": 0.00012, "MDVP:RAP": 0.00905, "MDVP:PPQ": 0.00909, "Jitter:DDP": 0.02716, "MDVP:Shimmer": 0.0617, "MDVP:Shimmer(dB)": 0.584, "Shimmer:APQ3": 0.03223, "Shimmer:APQ5": 0.03672, "MDVP:APQ": 0.05174, "Shimmer:DDA": 0.09669, "NHR": 0.11843, "HNR": 15.06, "RPDE": 0.639808, "DFA": 0.643327, "spread1": -4.20273, "spread2": 0.310163, "D2": 2.638279, "PPE": 0.356881, "Column1": 0.0[...]<jupyter_text>Send a request to the web deployed service  to test it.<jupyter_code>output = service.run(scoring_json)<jupyter_output><empty_output><jupyter_text>The result of the model vs the expectation<jupyter_code>print (output)
print(y_df)<jupyter_output>69     1
101    1
166    0
Name: status, dtype: int64<jupyter_text>In the cell below, the logs of the web service are printed once over the service is deleted<jupyter_code>print(service.get_logs())<jupyter_output>2020-12-31T22:18:28,809853837+00:00 - iot-server/run 
2020-12-31T22:18:28,809983939+00:00 - rsyslog/run 
2020-12-31T22:18:28,814131779+00:00 - gunicorn/run 
2020-12-31T22:18:28,818435521+00:00 - nginx/run 
/usr/sbin/nginx: /azureml-envs/azureml_9c539d20199ae6be65c41c0382029684/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_9c539d20199ae6be65c41c0382029684/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_9c539d20199ae6be65c41c0382029684/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_9c539d20199ae6be65c41c0382029684/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)
/usr/sbin/nginx: /azureml-envs/azureml_9c539d20199ae6be65c41c0382029684/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)
rsyslogd[...]<jupyter_text>to delete the service and the compute target<jupyter_code>service.delete()
aml_compute.delete()<jupyter_output>No service with name automl-bestmodel-parkinso found to delete.
{'fraction_non_alphanumeric': 0.11843540595293751, 'fraction_numerical': 0.047685834502103785, 'mean_word_length': 4.546672428694901, 'pattern_counts': {'":': 50, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '517436', 'license': 'MIT', 'n_tokens_mistral': 4494, 'n_tokens_neox': 4042, 'n_words': 1020, 'path': 'automl.ipynb', 'repo_name': 'rutujak24/MS-Azure-for-Parkinson-disease', 'size': '12834'}