<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  Train in a remote VM (MLC managed DSVM)* Create Workspace* Create Experiment* Upload data to a blob in workspace* Configure ACI run config* Submit the experiment in ACI* Register the retrained model  PrerequisitesMake sure you go through the [00. Installation and Configuration](00.configuration.ipynb) Notebook first if you haven't. Install Azure ML SDK* !pip install azureml-core* !pip install azureml-contrib-iot* !pip install azure-mgmt-containerregistry Check the conda environmentMake sure you have started the notebook from the correct conda environment<jupyter_code>import os
print(os.__file__)
# Check core SDK version number
import azureml.core as azcore

print("SDK version:", azcore.VERSION)<jupyter_output>SDK version: 1.0.2<jupyter_text>Initialize WorkspaceInitialize a workspace object from persisted configuration.<jupyter_code>from azureml.core import Workspace

ws = Workspace.from_config('./aml_config/config.json')
print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\n')<jupyter_output>Found the config file in: /home/arun/Documents/tensorflow-for-poets-2/aml_config/config.json
peabody
peabody
eastus
54646fde-e2bd-4f13-bb8a-2eb1174d1240<jupyter_text>Create Experiment**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments.<jupyter_code>experiment_name = 'soda_cans'

from azureml.core import Experiment
exp = Experiment(workspace = ws, name = experiment_name)<jupyter_output><empty_output><jupyter_text>Upload data files into datastoreEvery workspace comes with a default datastore (and you can register more) which is backed by the Azure blob storage account associated with the workspace. We can use it to transfer data from local to the cloud, and access it from the compute target.<jupyter_code># get the default datastore
ds = ws.get_default_datastore()
print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)<jupyter_output>workspaceblobstore AzureBlob peabody2715213321 azureml-blobstore-f8535f97-d577-4cb0-bdc8-71e1e2d2ac94<jupyter_text>Now let's upload the data into the default datastore:<jupyter_code>data_path = experiment_name + '_training_data'
#ds.upload(src_dir='./soda_cans', target_path=data_path, overwrite=True)<jupyter_output><empty_output><jupyter_text>If you have an existing azure storage, register it as a datastore to your workspace.<jupyter_code>from azureml.core.datastore import Datastore
ds = Datastore.register_azure_blob_container(workspace=ws, 
                                         datastore_name='henry', 
                                         container_name='images',
                                         account_name='hjtrainblob', 
                                         account_key='<KEY>',
                                         create_if_not_exists=False)
data_path = None # This is the path to the folder in the blob container. Set this to None to get all the contents.
print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)<jupyter_output><empty_output><jupyter_text>Configure for using ACILinux-based ACI is available in `West US`, `East US`, `West Europe`, `North Europe`, `West US 2`, `Southeast Asia`, `Australia East`, `East US 2`, and `Central US` regions. See details [here](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-quotasregion-availability). Create a `DataReferenceConfiguration` object to inform the system what data folder to download to the copmute target.<jupyter_code>from azureml.core.runconfig import DataReferenceConfiguration
dr = DataReferenceConfiguration(datastore_name=ds.name, 
                   path_on_datastore=data_path, 
                   mode='download', # download files from datastore to compute target
                   overwrite=True)<jupyter_output><empty_output><jupyter_text>Set the system to build a conda environment based on the run configuration. Once the environment is built, and if you don't change your dependencies, it will be reused in subsequent runs.<jupyter_code>from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# choose a name for your cluster
cluster_name = "cpucluster3"

try:
    compute_target = ComputeTarget(workspace=ws, name=cluster_name)
    print('Found existing compute target.')
except ComputeTargetException:
    print('Creating a new compute target...')
    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_D3', max_nodes=2)

    # create the cluster
    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)

    compute_target.wait_for_completion(show_output=True)

# Use the 'status' property to get a detailed status for the current AmlCompute. 
print(compute_target.status.serialize())

from azureml.core.runconfig import RunConfiguration, DEFAULT_CPU_IMAGE
from azureml.core.conda_dependencies import CondaDependencies

# create a new runconfig object
run_config = RunConfiguration(framework = "python")

# Set compute target
run_config.target = compute_target.name

# set the data reference of the run configuration
run_config.data_references = {ds.name: dr}

# enable Docker 
run_config.environment.docker.enabled = True

# set Docker base image to the default CPU-based image
run_config.environment.docker.base_image = DEFAULT_CPU_IMAGE

# use conda_dependencies.yml to create a conda environment in the Docker image for execution
run_config.environment.python.user_managed_dependencies = False

# auto-prepare the Docker image when used for execution (if it is not already prepared)
run_config.auto_prepare_environment = True

# specify CondaDependencies obj
run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['tensorflow==1.8.0'])<jupyter_output><empty_output><jupyter_text>Submit the ExperimentSubmit script to run in the Docker image in the remote VM. If you run this for the first time, the system will download the base image, layer in packages specified in the `conda_dependencies.yml` file on top of the base image, create a container and then execute the script in the container.<jupyter_code>from azureml.core import Run
from azureml.core import ScriptRunConfig

src = ScriptRunConfig(source_directory = './scripts', script = 'retrain.py', run_config = run_config, 
                      # pass the datastore reference as a parameter to the training script
                      arguments=['--image_dir', str(ds.as_download()),
                                 '--architecture', 'mobilenet_1.0_224',
                                 '--output_graph', 'outputs/retrained_graph.pb',
                                 '--output_labels', 'outputs/output_labels.txt',
                                 '--model_download_url', 'https://raw.githubusercontent.com/rakelkar/models/master/model_output/',
                                 '--model_file_name', 'imagenet_2_frozen.pb'
                                ])
run = exp.submit(config=src)<jupyter_output><empty_output><jupyter_text>View run history details<jupyter_code>run
run.wait_for_completion(show_output=True)<jupyter_output>RunId: soda_cans_1547167621516

Streaming azureml-logs/80_driver_log.txt
========================================

https://raw.githubusercontent.com/rakelkar/models/master/model_output/mobilenet_v1_1.0_224_frozen.tgz
/tmp/imagenet
/tmp/imagenet/mobilenet_v1_1.0_224_frozen.tgz

>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.0%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.1%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.1%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.2%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.2%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.3%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.3%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.4%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.4%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.5%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.5%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.6%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz 0.6%
>> Downloading mobilenet_v1_1.0_224_frozen.tgz[...]<jupyter_text>Register the Model<jupyter_code>from azureml.core.model import Model

model = run.register_model(model_name = experiment_name, model_path = 'outputs/')
#model = Model.register(model_path = "model",
#                      model_name = "distracted_driver",
#                      tags = {"data": "Imagenet", "model": "object_detection", "type": "imagenet"},
#                      description = "Retrained with model downloaded from Github",
#                      workspace = ws)
print(model.name, model.url, model.version, model.id, model.created_time)<jupyter_output>Deprecated, use RunHistoryFacade.assets instead.<jupyter_text>Convert Model<jupyter_code>from azureml.contrib.iot.model_converters import SnpeConverter

# submit a compile request
compile_request = SnpeConverter.convert_tf_model(
    ws,
    source_model=model,
    input_node="input",
    input_dims="1,224,224,3",
    outputs_nodes = ["final_result"],
    allow_unconsumed_nodes = True)
print(compile_request._operation_id)
# wait for the request to complete
compile_request.wait_for_completion(show_output=True)
# get the compiled model
compiled_model = compile_request.result
print(compiled_model.name, compiled_model.url, compiled_model.version, compiled_model.id, compiled_model.created_time)
compiled_model.download(target_dir="./converted/", exist_ok=True)<jupyter_output><empty_output><jupyter_text>Create Docker Image  Show the sample application file<jupyter_code>with open('./main.py', 'r') as f:
    print(f.read())

from azureml.core.image import Image
from azureml.contrib.iot import IotContainerImage

image_config = IotContainerImage.image_configuration(
                                 architecture="arm32v7",
                                 execution_script="main.py", 
                                 dependencies=["cameraapi.py","iot.py","ipcprovider.py","utility.py"],
                                 docker_file="Dockerfile",
                                 tags = ["mobilenet"],
                                 description = "MobileNet based demo module")
image = Image.create(name = "peabodymobilenet",
                     # this is the model object 
                     models = [compiled_model],
                     image_config = image_config, 
                     workspace = ws)
image.wait_for_creation(show_output = True)<jupyter_output><empty_output><jupyter_text>Enter your container registry credentials  List the image to get URI<jupyter_code>container_reg = ws.get_details()["containerRegistry"]
reg_name=container_reg.split("/")[-1]
resource_group_name = ws.resource_group
container_url = "\"" + image.image_location + "\","
subscription_id = ws.subscription_id
print('{}'.format(image.image_location))
print('{}'.format(reg_name))
print('{}'.format(subscription_id))
from azure.mgmt.containerregistry import ContainerRegistryManagementClient
from azure.mgmt import containerregistry
client = ContainerRegistryManagementClient(ws._auth,subscription_id)
result= client.registries.list_credentials(resource_group_name, reg_name, custom_headers=None, raw=False)
username = result.username
password = result.passwords[0].value<jupyter_output><empty_output><jupyter_text>Build your Deployment.json file<jupyter_code>%%writefile ./deploymentpb.json

{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
                
#Automatically adding your acr details
acr_details = "\"" + reg_name +"\": {\n\t\t\t\"username\": \""+ username + "\",\n\t\t\t" + "\"password\":\"" + password + "\",\n\t\t\t" + "\"address\":\"" + reg_name + ".azurecr.io\"" + ",\n\t\t}"
print('{}'.format(acr_details))
%store acr_details >> deploymentpb.json
%%writefile -a ./deploymentpb.json
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": "{}",
              "env": {
                "UpstreamProtocol": {
                  "value": "MQTT"
                }
              }
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": "{\"User\":\"root\",\"HostConfig\":{\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}], \"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}",
              "env": {
                "UpstreamProtocol": {
                  "value": "MQTT "
                }
              }
            }
          }
        },
        "modules": {
          "VisionSampleModule": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": 
#adding your container URL
%store container_url >> deploymentpb.json
%%writefile -a ./deploymentpb.json
              "createOptions": "{\"HostConfig\":{\"Binds\":[\"/data/misc/camera:/app/vam_model_folder\"],\"NetworkMode\":\"host\"},\"NetworkingConfig\":{\"EndpointsConfig\":{\"host\":{}}}}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "route": "FROM /messages/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}<jupyter_output><empty_output><jupyter_text>Deploy image as an IoT module  Set subscription to the same as your workspace<jupyter_code>%%writefile ./setsub
az account set --subscription 
iot_sub=ws.subscription_id
%store iot_sub >> setsub
!sh setsub 
print ('{}'.format(iot_sub))<jupyter_output><empty_output><jupyter_text>Provision Azure IoT Hub<jupyter_code>#RG and location to create hub
iot_rg="vaidk_"+resource_group_name
iot_location=ws.get_details()["location"]
#temp to delete
iot_location="eastus2"
iot_hub_name="iothub-"+ ws.get_details()["name"]
iot_device_id="vadik_"+ ws.get_details()["name"]
iot_deployment_id="dpl"+ "cstmvaidk"
print('{}'.format(iot_hub_name))
%%writefile ./create
#Command to create hub and device

# Adding Intialization steps
regcommand="\n echo Installing Extension ... \naz extension add --name azure-cli-iot-ext \n"+ "\n echo CREATING RG "+iot_rg+"... \naz group create --name "+ iot_rg +" --location "+ iot_location+ "\n" +"\n echo CREATING HUB "+iot_hub_name+"... \naz iot hub create --name "+ iot_hub_name + " --resource-group "+ iot_rg +" --sku S1"
#print('{}'.format(regcommand))
%store regcommand >> create<jupyter_output><empty_output><jupyter_text>Create Identity for your device<jupyter_code>#Adding Device ID 
create_device="\n echo CREATING DEVICE ID "+iot_device_id+"... \n az iot hub device-identity create --device-id "+ iot_device_id + " --hub-name " +  iot_hub_name +" --edge-enabled"
#print('{}'.format(create_device))
%store create_device >> create
#Create command and vonfigure device 
!sh create<jupyter_output><empty_output><jupyter_text>Create Deployment<jupyter_code>%%writefile ./deploy
#Command to create hub and device

#Add deployment command
deploy_device="\necho DELETING "+iot_deployment_id+" ... \naz iot edge deployment delete --deployment-id \"" + iot_deployment_id +"\" --hub-name \"" +  iot_hub_name +"\"\necho DEPLOYING "+iot_deployment_id+" ... \naz iot edge deployment create --deployment-id \"" + iot_deployment_id + "\" --content \"deploymentpb.json\" --hub-name \"" +  iot_hub_name +"\" --target-condition \"deviceId='"+iot_device_id+"'\" --priority 1"
print('{}'.format(deploy_device))
%store deploy_device >> deploy
#run deployment to stage all work for when the model is ready 
!sh deploy<jupyter_output><empty_output><jupyter_text>Use this conenction string on your camera to Initialize it<jupyter_code>%%writefile ./showdetails
#Command to create hub and device
#Add deployment command
get_string="\n echo THIS IS YOUR CONNECTION STRING ... \naz iot hub device-identity show-connection-string --device-id  \"" + iot_device_id + "\" --hub-name \"" +  iot_hub_name+"\""
#print('{}'.format(get_string))
%store get_string >> showdetails
!sh showdetails<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11849213416444049, 'fraction_numerical': 0.015553576728999704, 'mean_word_length': 3.6394932525475077, 'pattern_counts': {'":': 67, '<': 77, '<?xml version=': 0, '>': 119, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '280550', 'license': 'Apache-2.0', 'n_tokens_mistral': 5386, 'n_tokens_neox': 5099, 'n_words': 1174, 'path': 'Transfer learning soda cans.ipynb', 'repo_name': 'arunsu/tensorflow-for-poets-2', 'size': '16928'}