<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.png)  Automated Machine Learning_**Remote Execution with DataStore**_ Contents1. [Introduction](Introduction)1. [Setup](Setup)1. [Data](Data)1. [Train](Train)1. [Results](Results)1. [Test](Test)  IntroductionThis sample accesses a data file on a remote DSVM through DataStore. Advantages of using data store are:1. DataStore secures the access details.2. DataStore supports read, write to blob and file store3. AutoML natively supports copying data from DataStore to DSVMMake sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.In this notebook you would see1. Storing data in DataStore.2. get_data returning data from DataStore.  SetupAs part of the setup you have already created a Workspace. For AutoML you would need to create an Experiment. An Experiment is a named object in a Workspace, which is used to run experiments.<jupyter_code>import logging
import os
import time

import numpy as np
import pandas as pd

import azureml.core
from azureml.core.compute import DsvmCompute
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.train.automl import AutoMLConfig
ws = Workspace.from_config()

# choose a name for experiment
experiment_name = 'automl-remote-datastore-file'
# project folder
project_folder = './sample_projects/automl-remote-datastore-file'

experiment=Experiment(ws, experiment_name)

output = {}
output['SDK version'] = azureml.core.VERSION
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Project Directory'] = project_folder
output['Experiment Name'] = experiment.name
pd.set_option('display.max_colwidth', -1)
outputDf = pd.DataFrame(data = output, index = [''])
outputDf.T<jupyter_output><empty_output><jupyter_text>Create a Remote Linux DSVMNote: If creation fails with a message about Marketplace purchase eligibilty, go to portal.azure.com, start creating DSVM there, and select "Want to create programmatically" to enable programmatic creation. Once you've enabled it, you can exit without actually creating VM.**Note**: By default SSH runs on port 22 and you don't need to specify it. But if for security reasons you can switch to a different port (such as 5022), you can append the port number to the address. [Read more](https://docs.microsoft.com/en-us/azure/virtual-machines/troubleshooting/detailed-troubleshoot-ssh-connection) on this.<jupyter_code>compute_target_name = 'mydsvmc'

try:
    while ws.compute_targets[compute_target_name].provisioning_state == 'Creating':
        time.sleep(1)
        
    dsvm_compute = DsvmCompute(workspace=ws, name=compute_target_name)
    print('found existing:', dsvm_compute.name)
except:
    dsvm_config = DsvmCompute.provisioning_configuration(vm_size="Standard_D2_v2")
    dsvm_compute = DsvmCompute.create(ws, name=compute_target_name, provisioning_configuration=dsvm_config)
    dsvm_compute.wait_for_completion(show_output=True)
    print("Waiting one minute for ssh to be accessible")
    time.sleep(90) # Wait for ssh to be accessible<jupyter_output><empty_output><jupyter_text>Data Copy data file to localDownload the data file.<jupyter_code>if not os.path.isdir('data'):
    os.mkdir('data')    
from sklearn.datasets import fetch_20newsgroups
import csv

remove = ('headers', 'footers', 'quotes')
categories = [
        'alt.atheism',
        'talk.religion.misc',
        'comp.graphics',
        'sci.space',
    ]
data_train = fetch_20newsgroups(subset = 'train', categories = categories,
                                    shuffle = True, random_state = 42,
                                    remove = remove)
    
pd.DataFrame(data_train.data).to_csv("data/X_train.tsv", index=False, header=False, quoting=csv.QUOTE_ALL, sep="\t")
pd.DataFrame(data_train.target).to_csv("data/y_train.tsv", index=False, header=False, sep="\t")<jupyter_output><empty_output><jupyter_text>Upload data to the cloud Now make the data accessible remotely by uploading that data from your local machine into Azure so it can be accessed for remote training. The datastore is a convenient construct associated with your workspace for you to upload/download data, and interact with it from your remote compute targets. It is backed by Azure blob storage account.The data.tsv files are uploaded into a directory named data at the root of the datastore.<jupyter_code>#blob_datastore = Datastore(ws, blob_datastore_name)
ds = ws.get_default_datastore()
print(ds.datastore_type, ds.account_name, ds.container_name)
# ds.upload_files("data.tsv")
ds.upload(src_dir='./data', target_path='data', overwrite=True, show_progress=True)<jupyter_output><empty_output><jupyter_text>Configure & RunFirst let's create a DataReferenceConfigruation object to inform the system what data folder to download to the compute target.The path_on_compute should be an absolute path to ensure that the data files are downloaded only once.   The get_data method should use this same path to access the data files.<jupyter_code>from azureml.core.runconfig import DataReferenceConfiguration
dr = DataReferenceConfiguration(datastore_name=ds.name, 
                   path_on_datastore='data', 
                   path_on_compute='/tmp/azureml_runs',
                   mode='download', # download files from datastore to compute target
                   overwrite=False)
from azureml.core.runconfig import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies
import pkg_resources

# create a new RunConfig object
conda_run_config = RunConfiguration(framework="python")

# Set compute target to the Linux DSVM
conda_run_config.target = dsvm_compute
# set the data reference of the run coonfiguration
conda_run_config.data_references = {ds.name: dr}

pandas_dependency = 'pandas==' + pkg_resources.get_distribution("pandas").version

cd = CondaDependencies.create(pip_packages=['azureml-sdk[automl]'], conda_packages=['numpy','py-xgboost<=0.80',pandas_dependency])
conda_run_config.environment.python.conda_dependencies = cd<jupyter_output><empty_output><jupyter_text>Create Get Data FileFor remote executions you should author a get_data.py file containing a get_data() function. This file should be in the root directory of the project. You can encapsulate code to read data either from a blob storage or local disk in this file.The *get_data()* function returns a [dictionary](README.mdgetdata).The read_csv uses the path_on_compute value specified in the DataReferenceConfiguration call plus the path_on_datastore folder and then the actual file name.<jupyter_code>if not os.path.exists(project_folder):
    os.makedirs(project_folder)
%%writefile $project_folder/get_data.py

import pandas as pd

def get_data():
    X_train = pd.read_csv("/tmp/azureml_runs/data/X_train.tsv", delimiter="\t", header=None, quotechar='"')
    y_train = pd.read_csv("/tmp/azureml_runs/data/y_train.tsv", delimiter="\t", header=None, quotechar='"')

    return { "X" : X_train.values, "y" : y_train[0].values }<jupyter_output><empty_output><jupyter_text>TrainYou can specify automl_settings as **kwargs** as well. Also note that you can use the get_data() symantic for local excutions too. Note: For Remote DSVM and Batch AI you cannot pass Numpy arrays directly to AutoMLConfig.|Property|Description||-|-||**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: accuracyAUC_weightedaverage_precision_score_weightednorm_macro_recallprecision_score_weighted||**iteration_timeout_minutes**|Time limit in minutes for each iteration||**iterations**|Number of iterations. In each iteration Auto ML trains a specific pipeline with the data||**n_cross_validations**|Number of cross validation splits||**max_concurrent_iterations**|Max number of iterations that would be executed in parallel.  This should be less than the number of cores on the DSVM|**preprocess**| *True/False* Setting this to *True* enables Auto ML to perform preprocessing on the input to handle *missing data*, and perform some common *feature extraction*||**enable_cache**|Setting this to *True* enables preprocess done once and reuse the same preprocessed data for all the iterations. Default value is True.||**max_cores_per_iteration**| Indicates how many cores on the compute target would be used to train a single pipeline. Default is *1*, you can set it to *-1* to use all cores|<jupyter_code>automl_settings = {
    "iteration_timeout_minutes": 60,
    "iterations": 4,
    "n_cross_validations": 5,
    "primary_metric": 'AUC_weighted',
    "preprocess": True,
    "max_cores_per_iteration": 1,
    "verbosity": logging.INFO
}
automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             path=project_folder,
                             run_configuration=conda_run_config,
                             #compute_target = dsvm_compute,
                             data_script = project_folder + "/get_data.py",
                             **automl_settings
                            )<jupyter_output><empty_output><jupyter_text>For remote runs the execution is asynchronous, so you will see the iterations get populated as they complete. You can interact with the widgets/models even when the experiment is running to retreive the best model up to that point. Once you are satisfied with the model you can cancel a particular iteration or the whole run.<jupyter_code>remote_run = experiment.submit(automl_config, show_output=False)
remote_run<jupyter_output><empty_output><jupyter_text>Results Widget for monitoring runsThe widget will sit on "loading" until the first iteration completed, then you will see an auto-updating graph and table show up. It refreshed once per minute, so you should see the graph update as child runs complete.You can click on a pipeline to see run properties and output logs. Logs are also available on the DSVM under /tmp/azureml_run/{iterationid}/azureml-logsNOTE: The widget displays a link at the bottom. This links to a web-ui to explore the individual run details.<jupyter_code>from azureml.widgets import RunDetails
RunDetails(remote_run).show() 
# Wait until the run finishes.
remote_run.wait_for_completion(show_output = True)<jupyter_output><empty_output><jupyter_text>Retrieve All Child RunsYou can also use sdk methods to fetch all the child runs and see individual metrics that we log.<jupyter_code>children = list(remote_run.get_children())
metricslist = {}
for run in children:
    properties = run.get_properties()
    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    
    metricslist[int(properties['iteration'])] = metrics

rundata = pd.DataFrame(metricslist).sort_index(1)
rundata<jupyter_output><empty_output><jupyter_text>Canceling RunsYou can cancel ongoing remote runs using the *cancel()* and *cancel_iteration()* functions<jupyter_code># Cancel the ongoing experiment and stop scheduling new iterations
# remote_run.cancel()

# Cancel iteration 1 and move onto iteration 2
# remote_run.cancel_iteration(1)<jupyter_output><empty_output><jupyter_text>Pre-process cache cleanupThe preprocess data gets cache at user default file store. When the run is completed the cache can be cleaned by running below cell<jupyter_code>remote_run.clean_preprocessor_cache()<jupyter_output><empty_output><jupyter_text>Retrieve the Best ModelBelow we select the best pipeline from our iterations. The *get_output* method returns the best run and the fitted model. There are overloads on *get_output* that allow you to retrieve the best run and fitted model for *any* logged metric or a particular *iteration*.<jupyter_code>best_run, fitted_model = remote_run.get_output()<jupyter_output><empty_output><jupyter_text>Best Model based on any other metric<jupyter_code># lookup_metric = "accuracy"
# best_run, fitted_model = remote_run.get_output(metric=lookup_metric)<jupyter_output><empty_output><jupyter_text>Model from a specific iteration<jupyter_code># iteration = 1
# best_run, fitted_model = remote_run.get_output(iteration=iteration)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Load test data.
from pandas_ml import ConfusionMatrix

data_test = fetch_20newsgroups(subset = 'test', categories = categories,
                               shuffle = True, random_state = 42,
                               remove = remove)

X_test = np.array(data_test.data).reshape((len(data_test.data),1))
y_test = data_test.target

# Test our best pipeline.

y_pred = fitted_model.predict(X_test)
y_pred_strings = [data_test.target_names[i] for i in y_pred]
y_test_strings = [data_test.target_names[i] for i in y_test]

cm = ConfusionMatrix(y_test_strings, y_pred_strings)
print(cm)
cm.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08502116082224909, 'fraction_numerical': 0.0048367593712212815, 'mean_word_length': 5.050754458161865, 'pattern_counts': {'":': 7, '<': 66, '<?xml version=': 0, '>': 65, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '3196', 'license': 'MIT', 'n_tokens_mistral': 3931, 'n_tokens_neox': 3783, 'n_words': 1217, 'path': 'how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.ipynb', 'repo_name': 'waith/MachineLearningNotebooks', 'size': '13232'}