<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  Image Classification Using Scikit-learn  Prerequisites*    [ A Kubernetes cluster deployed on Azure Stack Hub, connected to Azure through ARC](https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/AML-ARC-Compute.md).     *     [Datastore setup in Azure Machine Learning workspace backed up by Azure Stack Hub storage account](https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/Train-AzureArc.md) *      Last but not least, you need to be able to run a Notebook.    If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the configuration Notebook located at [here](https://github.com/Azure/MachineLearningNotebooks) first. This sets you up with a working config file that has information on your workspace, subscription id, etc.  Initialize AzureML workspaceInitialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architectureworkspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`. If you haven't done already please go to `config.json` file and fill in your workspace information.<jupyter_code>from azureml.core.workspace import Workspace,  ComputeTarget
from azureml.exceptions import ComputeTargetException

ws = Workspace.from_config()
print('Workspace name: ' + ws.name, 
      'Azure region: ' + ws.location, 
      'Subscription id: ' + ws.subscription_id, 
      'Resource group: ' + ws.resource_group, sep='\n')<jupyter_output><empty_output><jupyter_text>Download mnist dataPerform pip install azureml-opendatasets to get the open dataset package, use this function to download mnist data later. This allows you to avoid download the data again when you run this notebook multiple times. The actual download time may take 2 minutes.<jupyter_code>from azureml.core import Dataset
from azureml.opendatasets import MNIST
import os

def download_mnist_data():
    data_folder = os.path.join(os.getcwd(), 'mnist_data')
    os.makedirs(data_folder, exist_ok=True)

    mnist_file_dataset = MNIST.get_file_dataset()
    path = mnist_file_dataset.download(data_folder, overwrite=True)
    downloaded_folder = os.path.dirname(path[0])
    print("downloaded to", downloaded_folder)
    
    return downloaded_folder<jupyter_output><empty_output><jupyter_text>Prepare the datasetThe above download_mnist_data() function will download four files  t10k-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz, train-images-idx3-ubyte.gz and train-labels-idx1-ubyte.gz to downloaded_folder.  Your next step is to upload these files to datastore of the workspace, and then registered as dataset in the workspace. "datastore_name" is the name of the datastore you setup in [this step](https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/Train-AzureArc.md).Upload and dataset registration take less than 1 min.<jupyter_code>from azureml.core import Workspace, Dataset, Datastore

dataset_name = "mnist_ash_o"
datastore_name = "<NAME_OF_ASH_HOSTED_AML_DATASTORE>"

if dataset_name not  in ws.datasets:
    downloaded_folder = download_mnist_data()
    datastore =  Datastore.get(ws, datastore_name)
    
    src_dir, target_path =downloaded_folder, 'mnistdataash'
    datastore.upload(src_dir, target_path)

    # register data uploaded as AML dataset
    datastore_paths = [(datastore, target_path)]
    mnist_ds = Dataset.File.from_files(path=datastore_paths)
    mnist_ds.register(ws, dataset_name, "mnist data from http://yann.lecun.com/exdb/mnist/")<jupyter_output><empty_output><jupyter_text>Setup compute targetFind the attach name for the Arc enabled  Azure Stack Hub kubernetes cluster in your AzureML workspace to create a ComputeTarget:attach_name is the attached name for your ASH cluster you setup in [this step](https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/AML-ARC-Compute.md)<jupyter_code>from azureml.core.compute import KubernetesCompute

attach_name = "<NAME_OF_AML_ATTACHED_COMPUTE_OF_YOUR_ASH_CLUSTER>"
arcK_target = KubernetesCompute(ws, attach_name)<jupyter_output><empty_output><jupyter_text>Configure the training job and submit  Create an experiement<jupyter_code>from azureml.core import Experiment

experiment_name = 'mnist-demo'

exp = Experiment(workspace=ws, name=experiment_name)<jupyter_output><empty_output><jupyter_text>Create an environment<jupyter_code>from azureml.core.environment import Environment
from azureml.core.conda_dependencies import CondaDependencies

# to install required packages
env = Environment('tutorial-env')
cd = CondaDependencies.create(pip_packages=['azureml-dataset-runtime[pandas,fuse]', 'azureml-defaults'], conda_packages = ['scikit-learn==0.22.1'])

env.python.conda_dependencies = cd<jupyter_output><empty_output><jupyter_text>Configure the training jobThe training takes about 15 mins with vm size comparable  to Standard_DS3_v2<jupyter_code>from azureml.core import ScriptRunConfig

args = ['--data-folder', ws.datasets[dataset_name].as_mount(), '--regularization', 0.5]
script_folder =  "mnist_script"
src = ScriptRunConfig(source_directory=script_folder,
                      script='train.py', 
                      arguments=args,
                      compute_target=arcK_target,
                      environment=env)<jupyter_output><empty_output><jupyter_text>Submit the jobRun your experiment by submitting your ScriptRunConfig object. Note that this call is asynchronous.<jupyter_code>run = exp.submit(config=src)
run.wait_for_completion(show_output=True)  # specify True for a verbose log<jupyter_output><empty_output><jupyter_text>Register the modelRegister the trained model.<jupyter_code># register model
model_name='sklearn_mnist'
model = run.register_model(model_name=model_name,
                           model_path='outputs/sklearn_mnist_model.pkl')<jupyter_output><empty_output><jupyter_text>The machine learning model named "sklearn_mnist" should be registered in your AzureML workspace.  Test the registered modelTo test the trained model, you can create (or use existing) a AKS cluster for serving the model using AML deployment.Note: You will test the model in azure cloud because AzureML inferencing on Azure Stack Hub in currently unavailable.<jupyter_code>from azureml.core import Environment, Workspace, Model, ComputeTarget
from azureml.core.compute import AksCompute
from azureml.core.model import InferenceConfig
from azureml.core.webservice import Webservice, AksWebservice
from azureml.core.compute_target import ComputeTargetException
import numpy as np
import json<jupyter_output><empty_output><jupyter_text>Provision the AKS ClusterThis is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it. It may take 5 mins to create a new AKS cluster.<jupyter_code>ws = Workspace.from_config()

# Choose a name for your AKS cluster
aks_name = 'aks-service-2'

# Verify that cluster does not exist already
try:
    aks_target = ComputeTarget(workspace=ws, name=aks_name)
    is_new_compute  = False
    print('Found existing cluster, use it.')
except ComputeTargetException:
    # Use the default configuration (can also provide parameters to customize)
    prov_config = AksCompute.provisioning_configuration()

    # Create the cluster
    aks_target = ComputeTarget.create(workspace = ws, 
                                    name = aks_name, 
                                    provisioning_configuration = prov_config)
    is_new_compute  = True
    
print("using compute target: ", aks_target.name)
from azureml.core.webservice import Webservice
from azureml.core.model import InferenceConfig
from azureml.core.environment import Environment
from azureml.core import Workspace
from azureml.core.model import Model

ws = Workspace.from_config()
model = Model(ws, model_name)

env = Environment('tutorial-env')
cd = CondaDependencies.create(pip_packages=['azureml-dataset-runtime[pandas,fuse]', 'azureml-defaults'], conda_packages = ['scikit-learn==0.22.1'])

env.python.conda_dependencies = cd

inference_config = InferenceConfig(entry_script="score.py", environment=env)
deploy_config = AksWebservice.deploy_configuration()

service = Model.deploy(workspace=ws, 
                       name='sklearn-mnist-svc', 
                       models=[model], 
                       inference_config=inference_config, 
                       deployment_target=aks_target,
                       deployment_config=deploy_config,
                       overwrite=True)

service.wait_for_deployment(show_output=True)<jupyter_output><empty_output><jupyter_text>Test with inputsHere you may use the first image (saved as digit_7.jpg) from test set, as show here:![fishy](digit_7.jpg)<jupyter_code>from mnist_script.utils import load_data
import os
import glob

data_folder = os.path.join(os.getcwd(), 'mnist_data')

X_test = load_data(glob.glob(os.path.join(data_folder,"**/t10k-images-idx3-ubyte.gz"), recursive=True)[0], False) / 255.0
y_test = load_data(glob.glob(os.path.join(data_folder,"**/t10k-labels-idx1-ubyte.gz"), recursive=True)[0], True).reshape(-1)

import json
test = json.dumps({"data": X_test.tolist()[:1]})
test = bytes(test, encoding='utf8')

# get predicted digits:

predictions = service.run(input_data=test)
print(predictions)<jupyter_output><empty_output><jupyter_text>You should get 7 as predicted digit for the first image in test set .  Delete the newly created clusterNote: This is important if you wish to avoid the cost of this cluster<jupyter_code>if is_new_compute:
    aks_target.delete()
    service.delete()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08506979567064536, 'fraction_numerical': 0.004551891563827635, 'mean_word_length': 4.923906530856801, 'pattern_counts': {'":': 1, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '469285', 'license': 'MIT', 'n_tokens_mistral': 3039, 'n_tokens_neox': 2921, 'n_words': 807, 'path': 'docs/ASH/notebooks/mnist/MNIST_Training_with_ASH_Cluster_and_Storage.ipynb', 'repo_name': 'frcai/AML-Kubernetes', 'size': '9886'}