<jupyter_start><jupyter_text>![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.png)  YOLO Real-time Object Detection using ONNX on AzureMLThis example shows how to use the YOLO v3 model as a web service using Azure Machine Learning services and the ONNX Runtime. What is ONNXONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai). YOLO DetailsYou Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. For more information about YOLO, please visit the [YOLO website](https://pjreddie.com/darknet/yolo/).  PrerequisitesTo make the best use of your time, make sure you have done the following:* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning* Follow the instructions in the readme file before going through the steps in this notebook<jupyter_code># Check core SDK version number
import azureml.core

print("SDK version:", azureml.core.VERSION)<jupyter_output><empty_output><jupyter_text>Download YOLO v3 ONNX model First we download the model. This may take a few minutes. The model will be downloaded to the same folder as this notebook.<jupyter_code>import urllib.request

onnx_model_url = "https://onnxzoo.blob.core.windows.net/models/opset_10/yolov3/yolov3.onnx"
urllib.request.urlretrieve(onnx_model_url, filename="yolov3.onnx")<jupyter_output><empty_output><jupyter_text>Load Azure ML workspaceWe begin by instantiating a workspace object from the existing workspace created in the configuration notebook.<jupyter_code>from azureml.core import Workspace

ws = Workspace.from_config()
print(ws.name, ws.location, ws.resource_group, sep = '\n')<jupyter_output><empty_output><jupyter_text>Registering your model with Azure MLNow we upload the model and register it in the workspace.<jupyter_code>from azureml.core.model import Model

model = Model.register(model_path = "yolov3.onnx",
                       model_name = "yolov3",
                       tags = {"onnx": "yolov3"},
                       description = "YOLOv3 from ONNX Model Zoo",
                       workspace = ws)<jupyter_output><empty_output><jupyter_text>Displaying your registered modelsYou can optionally list out all the models that you have registered in this workspace.<jupyter_code>models = ws.models
for name, m in models.items():
    print("Name:", name,"\tVersion:", m.version, "\tDescription:", m.description, m.tags)<jupyter_output><empty_output><jupyter_text>Write scoring fileWe are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object. The `run()` function is called when the webservice is invoked for inferencing. After running the code below you should see a score.py file in the same folder as this notebook.<jupyter_code>%%writefile score.py
import json
import time
import sys
import os
from azureml.core.model import Model
import numpy as np    # we're going to use numpy to process input and output data
import onnxruntime    # to inference ONNX models, we use the ONNX Runtime
import base64
from PIL import Image
import io

def init():
    global session
    model = Model.get_model_path(model_name = 'yolov3')
    session = onnxruntime.InferenceSession(model)

def letterbox_image(image, size):
    '''resize image with unchanged aspect ratio using padding'''
    iw, ih = image.size
    w, h = size
    scale = min(w/iw, h/ih)
    nw = int(iw*scale)
    nh = int(ih*scale)

    image = image.resize((nw,nh), Image.BICUBIC)
    new_image = Image.new('RGB', size, (128,128,128))
    new_image.paste(image, ((w-nw)//2, (h-nh)//2))
    return new_image


    
def preprocess(input_data_json):
    # convert the JSON data into the tensor input    
    imgb64 = json.loads(input_data_json)['data']    
    
    # Base64 decoding
    image_64_decode = base64.b64decode(imgb64)
    
    # Open the image 
    img = Image.open(io.BytesIO(image_64_decode))
    
    
    model_image_size = (416, 416)
    
    # Get the resized image
    boxed_image = letterbox_image(img, tuple(reversed(model_image_size)))
    
    # Convert image to numpy array
    image_data = np.array(boxed_image, dtype='float32')
    
    # Normalize image
    image_data /= 255.
    
     # Array has shape height x width x channel. We need to transpose it to channel x width x height            
    image_data = np.transpose(image_data, [2, 0, 1])
    
    # Add another dimension to make it an array of images    
    image_data = np.expand_dims(image_data, 0)
    
    image_size = np.array([img.size[1], img.size[0]], dtype=np.float32).reshape(1, 2)          
    
    return image_data, image_size

def postprocess(result):
    #r = np.array(result)
    boxes = result[0]
    scores = result[1]
    indices = result[2]
   
    
    out_boxes, out_scores, out_classes = [], [], []
    for idx_ in indices:
        out_classes.append(idx_[1].tolist())
        out_scores.append(scores[tuple(idx_)].tolist())
        idx_1 = (idx_[0], idx_[2])
        out_boxes.append(boxes[idx_1].tolist())    
                   
    er = {'boxes':out_boxes, 'scores':out_scores, 'classes':out_classes}

    
    return json.dumps(er)

def run(input_data_json):
    try:
        start = time.time()   # start timer
        image_data, image_size = preprocess(input_data_json)
        
        input_feeds = {}
        input_feeds[session.get_inputs()[0].name] = image_data
        input_feeds[session.get_inputs()[1].name] = image_size
        
        #input_name = session.get_inputs()[0].name  # get the id of the first input of the model   
        result = session.run([], input_feeds)
        end = time.time()     # stop timer
        return {"result": postprocess(result),
                "time": end - start}
    except Exception as e:
        result = str(e)
        return {"error": result}<jupyter_output><empty_output><jupyter_text>Create dependencies fileCreate a YAML file that specifies which dependencies we would like to see in our container. After running the code below you should see myenv.yml in the same folder as this notebook.<jupyter_code>from azureml.core.conda_dependencies import CondaDependencies 

myenv = CondaDependencies.create(pip_packages=["numpy","pillow", "onnxruntime","azureml-defaults", "azureml-core"])

with open("myenv.yml","w") as f:
    f.write(myenv.serialize_to_string())<jupyter_output><empty_output><jupyter_text>Create container image in Azure MLUse Azure ML to create the container image. This step will likely take a few minutes.<jupyter_code>from azureml.core.model import InferenceConfig, Model

# Create inference configuration. This creates a docker image that contains the model.
inference_config = InferenceConfig(runtime="python",
                                   entry_script="score.py",
                                   conda_file="myenv.yml")

# Builds an image in ACR.
# TODO: Move to 1.12.0 SDK version, and specify image name, and tag.
package = Model.package(ws, [model], inference_config)
package.wait_for_creation(show_output=True)

print("ACR:", package.get_container_registry)
print("Image:", package.location)<jupyter_output><empty_output><jupyter_text>Setup Azure IoT Edge deviceFollow [documentation](https://docs.microsoft.com/en-us/azure/iot-edge/quickstart-linux) to setup a Linux VM as an Azure IoT Edge device  Deploy container to Azure IoT Edge device<jupyter_code>from azureml.core.image import ContainerImage

acr_name = package.location.split("/")[0]
reg_name = acr_name.split(".")[0]
subscription_id = ws.subscription_id

print('{}'.format(acr_name))
print('{}'.format(subscription_id))

# TODO: Derive image_location through code.
image_location = "<Fill image URL from ACR>"

print('{}'.format(image.image_location))

# Fetch username, password of ACR.
from azure.mgmt.containerregistry import ContainerRegistryManagementClient
from azure.mgmt import containerregistry

client = ContainerRegistryManagementClient(ws._auth,subscription_id)
result= client.registries.list_credentials(ws.resource_group, reg_name, custom_headers=None, raw=False)

username = result.username
password = result.passwords[0].value

print(username)
print(password)<jupyter_output><empty_output><jupyter_text>Create a deployment.json file using the template json. Then push the deployment json file to the IoT Hub, which will then send it to the IoT Edge device. The IoT Edge agent will then pull the Docker images and run them.<jupyter_code>module_name = "yolov3"

file = open('iotedge-yolov3-template.json')
contents = file.read()
contents = contents.replace('__MODULE_NAME', module_name)
contents = contents.replace('__REGISTRY_NAME', reg_name)
contents = contents.replace('__REGISTRY_USER_NAME', username)
contents = contents.replace('__REGISTRY_PASSWORD', password)
contents = contents.replace('__REGISTRY_IMAGE_LOCATION', image_location)
with open('./deployment.json', 'wt', encoding='utf-8') as output_file:
    output_file.write(contents)<jupyter_output><empty_output><jupyter_text>Enter your the IoT device id and the IoT Hub name in the command below<jupyter_code># Push the deployment JSON to the IOT Hub
!az iot edge set-modules --device-id <IoTdeviceid> --hub-name <IoTHubName> --content deployment.json<jupyter_output><empty_output><jupyter_text>TestingBefore testing, open up inbound port 5001 on your Edge device. You can use [Azure Portal](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/nsg-quickstart-portal) for this purpose. Update the scoring URI with the edge device public IP address<jupyter_code>import json
import requests

scoring_uri = 'http://<EdgeDeviceIPAddress>:5001/score'

# You cannot send a byte array in JSON and hence need to decode it to UTF-8
input_data = json.dumps({'data': image_64_encode.decode("utf-8")})

try:
    
    # Set the content type
    headers = {'Content-Type': 'application/json'}

    # Make the request and display the response
    resp = requests.post(scoring_uri, input_data, headers=headers)    
    
    plotImageWithBBoxesAndLabels(resp.text, downloaded_imagefile)
    
except KeyError as e:
    print(str(e))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09460206337509211, 'fraction_numerical': 0.009856300663227709, 'mean_word_length': 4.197223551938727, 'pattern_counts': {'":': 4, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '548028', 'license': 'MIT', 'n_tokens_mistral': 3360, 'n_tokens_neox': 3229, 'n_words': 1004, 'path': 'Research/aml-iot/onnx-deploy-yolov3.ipynb', 'repo_name': 'piyushka17/azure-intelligent-edge-patterns', 'size': '10856'}