<jupyter_start><jupyter_text>Converting a Keras model to ONNXIn the steps that follow, you will convert Keras model you just trained to the ONNX format. This will enable you to use this model for classification in a very broad range of environments, outside of Azure Databricks including:- Web services- iOS and Android mobile apps- Windows apps- IoT devicesFurthermore, ONNX runtimes and libraries are also designed to maximize performance on some of the best hardware in the industry. In this lab, we will compare the Inference performance of the ONNX vs Keras models.First we will load the trained Keras model from file, and then convert the model to ONNX.  Load the Keras ModelLoad the saved Keras model. We will convert the Keras model to ONNX format.<jupyter_code>import os
import numpy as np
import pandas as pd

np.random.seed(125)

from keras.models import load_model
import joblib

output_folder = './output'
model_filename = 'final_model.hdf5'

keras_model = load_model(os.path.join(output_folder, model_filename))
print(keras_model.summary())<jupyter_output><empty_output><jupyter_text>Convert to ONNXConvert the loaded Keras model to ONNX format, and save the ONNX model to the deployment folder.<jupyter_code>import onnxmltools

deployment_folder = 'deploy'
onnx_export_folder = 'onnx'

# Convert the Keras model to ONNX
onnx_model_name = 'claim_classifier.onnx'
converted_model = onnxmltools.convert_keras(keras_model, onnx_model_name, target_opset=7)

# Save the model locally...
onnx_model_path = os.path.join(deployment_folder, onnx_export_folder)
os.makedirs(onnx_model_path, exist_ok=True)
onnxmltools.utils.save_model(converted_model, os.path.join(onnx_model_path,onnx_model_name))<jupyter_output><empty_output><jupyter_text>Make Inference using the ONNX Model- Create an ONNX runtime InferenceSession- Review the expected input shape to make inferences- Prepare test data- Make inferences using both the ONNX and the Keras Model on the test data  ONNX Runtime InferenceSession<jupyter_code>import onnxruntime
# Load the ONNX model and observe the expected input shape
onnx_session = onnxruntime.InferenceSession(
    os.path.join(os.path.join(deployment_folder, onnx_export_folder), onnx_model_name))
input_name = onnx_session.get_inputs()[0].name
output_name = onnx_session.get_outputs()[0].name
print('Expected input shape: ', onnx_session.get_inputs()[0].shape)<jupyter_output><empty_output><jupyter_text>Prepare test data **Load the GloVe word vectors**<jupyter_code>word_vectors_dir = './word_vectors'

dictonary = np.load(os.path.join(word_vectors_dir, 'wordsList.npy'))
dictonary = dictonary.tolist()
dictonary = [word.decode('UTF-8') for word in dictonary]
print('Loaded the dictonary! Dictonary size: ', len(dictonary))

word_vectors = np.load(os.path.join(word_vectors_dir, 'wordVectors.npy'))
print ('Loaded the word vectors! Shape of the word vectors: ', word_vectors.shape)<jupyter_output><empty_output><jupyter_text>**Create the word contractions map**<jupyter_code>contractions_url = ('https://quickstartsws9073123377.blob.core.windows.net/'
                    'azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/glove50d/contractions.xlsx')
contractions_df = pd.read_excel(contractions_url)
contractions = dict(zip(contractions_df.original, contractions_df.expanded))<jupyter_output><empty_output><jupyter_text>**Setup the helper functions to process the test data**<jupyter_code>import re
import string

def remove_special_characters(token):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    filtered_token = pattern.sub('', token)
    return filtered_token

def convert_to_indices(corpus, dictonary, c_map, unk_word_index = 399999):
    sequences = []
    for i in range(len(corpus)):
        tokens = corpus[i].split()
        sequence = []
        for word in tokens:
            word = word.lower()
            if word in c_map:
                resolved_words = c_map[word].split()
                for resolved_word in resolved_words:
                    try:
                        word_index = dictonary.index(resolved_word)
                        sequence.append(word_index)
                    except ValueError:
                        sequence.append(unk_word_index) #Vector for unkown words
            else:
                try:
                    clean_word = remove_special_characters(word)
                    if len(clean_word) > 0:
                        word_index = dictonary.index(clean_word)
                        sequence.append(word_index)
                except ValueError:
                    sequence.append(unk_word_index) #Vector for unkown words
        sequences.append(sequence)
    return sequences<jupyter_output><empty_output><jupyter_text>**Preprocess the test data**<jupyter_code>from keras.preprocessing.sequence import pad_sequences
maxSeqLength = 125

test_claim = ['I crashed my car into a pole.']

test_claim_indices = convert_to_indices(test_claim, dictonary, contractions)
test_data = pad_sequences(test_claim_indices, maxlen=maxSeqLength, padding='pre', truncating='post')

# convert the data type to float
test_data_float = np.reshape(test_data.astype(np.float32), (1,maxSeqLength))<jupyter_output><empty_output><jupyter_text>Make InferencesMake inferences using both the ONNX and the Keras Model on the test data<jupyter_code># Run an ONNX session to classify the sample.
print('ONNX prediction: ', onnx_session.run([output_name], {input_name : test_data_float}))

# Use Keras to make predictions on the same sample
print('Keras prediction: ', keras_model.predict(test_data_float))<jupyter_output><empty_output><jupyter_text>Compare Inference Performance: ONNX vs KerasEvaluate the performance of ONNX and Keras by running the same sample 1,000 times. Run the next three cells and compare the performance in your environment.<jupyter_code># Next we will compare the performance of ONNX vs Keras
import timeit
n = 1000
start_time = timeit.default_timer()
for i in range(n):
    keras_model.predict(test_data_float)
keras_elapsed = timeit.default_timer() - start_time
print('Keras performance: ', keras_elapsed)
start_time = timeit.default_timer()
for i in range(n):
    onnx_session.run([output_name], {input_name : test_data_float})
onnx_elapsed = timeit.default_timer() - start_time
print('ONNX performance: ', onnx_elapsed)
print('ONNX is about {} times faster than Keras'.format(round(keras_elapsed/onnx_elapsed)))<jupyter_output><empty_output><jupyter_text>Deploy ONNX model to Azure Container Instance (ACI)  Create and connect to an Azure Machine Learning WorkspaceReview the workspace config file saved in the previous notebook.<jupyter_code>!cat .azureml/config.json<jupyter_output><empty_output><jupyter_text>**Create the `Workspace` from the saved config file**<jupyter_code>import azureml.core

print(azureml.core.VERSION)

from azureml.core.workspace import Workspace

ws = Workspace.from_config()
print(ws)<jupyter_output><empty_output><jupyter_text>Register the model with Azure Machine LearningIn the following, you register the model with Azure Machine Learning (which saves a copy in the cloud).<jupyter_code>#Register the model and vectorizer
from azureml.core.model import Model

registered_model_name = 'claim_classifier_onnx'
onnx_model_path = os.path.join(os.path.join(deployment_folder, onnx_export_folder), onnx_model_name)

registered_model = Model.register(model_path = onnx_model_path, # this points to a local file
                       model_name = registered_model_name, # this is the name the model is registered with         
                       description = "Claims classification model.",
                       workspace = ws)

print(registered_model.name, registered_model.description, registered_model.version)<jupyter_output><empty_output><jupyter_text>Create the scoring web serviceWhen deploying models for scoring with Azure Machine Learning services, you need to define the code for a simple web service that will load your model and use it for scoring. By convention this service has two methods init which loads the model and run which scores data using the loaded model.This scoring service code will later be deployed inside of a specially prepared Docker container. **Save the scoring web service Python file**Note that the scoring web service needs the registered model: the ONNX model to make inferences.<jupyter_code>%%writefile scoring_service.py
import string
import re
import os
import numpy as np
import pandas as pd
import urllib.request
import json
import keras
from keras.preprocessing.sequence import pad_sequences
import tensorflow as tf
from azureml.core.model import Model
import onnxruntime

def init():

    global onnx_session
    global dictonary
    global contractions
    
    try:
        words_list_url = ('https://quickstartsws9073123377.blob.core.windows.net/'
                          'azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/glove50d/wordsList.npy')
        word_vectors_dir = './word_vectors'
        os.makedirs(word_vectors_dir, exist_ok=True)
        urllib.request.urlretrieve(words_list_url, os.path.join(word_vectors_dir, 'wordsList.npy'))
        dictonary = np.load(os.path.join(word_vectors_dir, 'wordsList.npy'))
        dictonary = dictonary.tolist()
        dictonary = [word.decode('UTF-8') for word in dictonary]
        print('Loaded the dictonary! Dictonary size: ', len(dictonary))
        
        contractions_url = ('https://quickstartsws9073123377.blob.core.windows.net/'
                            'azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/glove50d/contractions.xlsx')
        contractions_df = pd.read_excel(contractions_url)
        contractions = dict(zip(contractions_df.original, contractions_df.expanded))
        print('Loaded contractions!')
        
        # Retrieve the path to the model file using the model name
        onnx_model_name = 'claim_classifier_onnx'
        onnx_model_path = Model.get_model_path(onnx_model_name)
        print('onnx_model_path: ', onnx_model_path)
        
        onnx_session = onnxruntime.InferenceSession(onnx_model_path)
        print('Onnx Inference Session Created!')
        
    except Exception as e:
        print(e)
        
def remove_special_characters(token):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    filtered_token = pattern.sub('', token)
    return filtered_token

def convert_to_indices(corpus, dictonary, c_map, unk_word_index = 399999):
    sequences = []
    for i in range(len(corpus)):
        tokens = corpus[i].split()
        sequence = []
        for word in tokens:
            word = word.lower()
            if word in c_map:
                resolved_words = c_map[word].split()
                for resolved_word in resolved_words:
                    try:
                        word_index = dictonary.index(resolved_word)
                        sequence.append(word_index)
                    except ValueError:
                        sequence.append(unk_word_index) #Vector for unkown words
            else:
                try:
                    clean_word = remove_special_characters(word)
                    if len(clean_word) > 0:
                        word_index = dictonary.index(clean_word)
                        sequence.append(word_index)
                except ValueError:
                    sequence.append(unk_word_index) #Vector for unkown words
        sequences.append(sequence)
    return sequences

def run(raw_data):
    try:
        print("Received input: ", raw_data)
        
        maxSeqLength = 125
        
        print('Processing input...')
        input_data_raw = np.array(json.loads(raw_data))
        input_data_indices = convert_to_indices(input_data_raw, dictonary, contractions)
        input_data_padded = pad_sequences(input_data_indices, maxlen=maxSeqLength, padding='pre', truncating='post')
        # convert the data type to float
        input_data = np.reshape(input_data_padded.astype(np.float32), (1,maxSeqLength))
        print('Done processing input.')
        
        # Run an ONNX session to classify the input.
        result = onnx_session.run(None, {onnx_session.get_inputs()[0].name: input_data})[0].argmax(axis=1).item()
        # return just the classification index (0 or 1)
        return result
    except Exception as e:
        print(e)
        error = str(e)
        return error<jupyter_output><empty_output><jupyter_text>Package Model and deploy to ACIYour scoring service can have dependencies install by using a Conda environment file. Items listed in this file will be conda or pip installed within the Docker container that is created and thus be available to your scoring web service logic.The recommended deployment pattern is to create a deployment configuration object with the `deploy_configuration` method and then use it with the deploy method of the [Model](https://docs.microsoft.com/python/api/azureml-core/azureml.core.model.model?view=azure-ml-py) class as performed below. In this case, we use the `AciWebservice`'s `deploy_configuration` and specify the CPU cores and memory size.You will see output similar to the following when your web service is ready: `Succeeded - ACI service creation operation finished, operation "Succeeded"`Run the following cell. This may take between 5-10 minutes to complete.<jupyter_code># create a Conda dependencies environment file
print("Creating conda dependencies file locally...")
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core import Environment
from azureml.core.model import InferenceConfig
from azureml.core.webservice import AciWebservice, Webservice

conda_packages = ['numpy==1.16.4', 'xlrd==1.2.0', 'pandas==0.25.1', 'scikit-learn==0.21.3']
pip_packages = ['azureml-defaults', 'azureml-sdk', 'tensorflow==1.13.1', 'keras==2.3.1', 'onnxruntime==1.0.0']

environment = Environment('my-environment')
environment.python.conda_dependencies = CondaDependencies.create(conda_packages=conda_packages, pip_packages=pip_packages)

execution_script = 'scoring_service.py'
service_name = "claimclassservice"

inference_config = InferenceConfig(entry_script=execution_script, environment=environment)
aci_config = AciWebservice.deploy_configuration(
    cpu_cores=1,
    memory_gb=1, 
    tags = {'name': 'Claim Classification'}, 
    description = "Classifies a claim as home or auto.")

service = Model.deploy(workspace=ws,
                      name=service_name,
                      models=[registered_model],
                      inference_config=inference_config,
                      deployment_config=aci_config)

# wait for the deployment to finish
service.wait_for_deployment(show_output=True)<jupyter_output><empty_output><jupyter_text>Test Deployment  Make direct calls on the service object<jupyter_code>import json

test_claims = ['I crashed my car into a pole.', 
               'The flood ruined my house.', 
               'I lost control of my car and fell in the river.']

for i in range(len(test_claims)):
    result = service.run(json.dumps([test_claims[i]]))
    print('Predicted label for test claim #{} is {}'.format(i+1, result))<jupyter_output><empty_output><jupyter_text>Make HTTP calls to test the deployed Web ServiceIn order to call the service from a REST client, you need to acquire the scoring URI. Take a note of printed scoring URI, you will need it in the last notebook.The default settings used in deploying this service result in a service that does not require authentication, so the scoring URI is the only value you need to call this service.<jupyter_code>import requests

url = service.scoring_uri
print('ACI Service: Claim Classification scoring URI is: {}'.format(url))
headers = {'Content-Type':'application/json'}

for i in range(len(test_claims)):
    response = requests.post(url, json.dumps([test_claims[i]]), headers=headers)
    print('Predicted label for test claim #{} is {}'.format(i+1, response.text))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08372877122877123, 'fraction_numerical': 0.011613386613386614, 'mean_word_length': 3.9222495390288876, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 67, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '64122', 'license': 'MIT', 'n_tokens_mistral': 4868, 'n_tokens_neox': 4650, 'n_words': 1326, 'path': 'Hands-on lab/notebooks/04 Deploy Classifier Web Service.ipynb', 'repo_name': 'marktab/MCW-Cognitive-services-and-deep-learning', 'size': '16016'}