<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  ONNX Pipeline This repository shows how to deploy and use ONNX pipeline with dockers including convert model, generate input and performance test.  Prerequisites Pull dockers from Azure. It should take several minutes.<jupyter_code># !sh build.sh # For Linux
!build.sh # For Windows<jupyter_output><empty_output><jupyter_text>Install the onnxpipeline SDK<jupyter_code>import sys
sys.path.append('../utils')
import onnxpipeline

# Initiate ONNX pipeline with local directory "model"
pipeline = onnxpipeline.Pipeline('model')

# onnx
#pipeline = onnxpipeline.Pipeline('onnx')

# tensorflow
#pipeline = onnxpipeline.Pipeline('mnist/model')

# pytorch
#pipeline = onnxpipeline.Pipeline('pytorch')

# cntk 
#pipeline = onnxpipeline.Pipeline('cntk')

# keras
#pipeline = onnxpipeline.Pipeline('KerasToONNX')

# sklearn
#pipeline = onnxpipeline.Pipeline('sklearn')

# caffe
#pipeline = onnxpipeline.Pipeline('caffe')

# current directory
#pipeline = onnxpipeline.Pipeline()

# test mxnet fail
#pipeline = onnxpipeline.Pipeline('mxnet')<jupyter_output><empty_output><jupyter_text>Run parameters(1) local_directory: string    Required. The path of local directory where would be mounted to the docker. All operations will be executed from this path.(2) mount_path: string    Optional. The path where the local_directory will be mounted in the docker. Default is "/mnt/model".(3) print_logs: boolean    Optional. Whether print the logs from the docker. Default is True.    (4) convert_directory: string    Optional. The directory path for converting model. Default is test/.    (5) convert_name: string    Optional. The model name for converting model. Default is model.onnx.      Config information for ONNX pipeline<jupyter_code>pipeline.config()<jupyter_output>-----------config----------------
           Container information: <docker.client.DockerClient object at 0x000001E5C77FD6D8>
 Local directory path for volume: E:\onnx-pipeline\notebook/model
Volume directory path in dockers: /mnt/model
                     Result path: result
        Converted directory path: test
        Converted model filename: model.onnx
            Converted model path: test/model.onnx
        Print logs in the docker: True<jupyter_text>Convert model to ONNX This image is used to convert model from major model frameworks to onnx. Supported frameworks are - caffe, cntk, coreml, keras, libsvm, mxnet, scikit-learn, tensorflow and pytorch.You can run the docker image with customized parameters.<jupyter_code>model = pipeline.convert_model(model='model.onnx', model_type='onnx')

# test tensorflow
#model = pipeline.convert_model(model_type='tensorflow')

# test pytorch
#model = pipeline.convert_model(model_type='pytorch', model='saved_model.pb', model_input_shapes='(1,3,224,224)')

# test cntk
#model = pipeline.convert_model(model_type='cntk', model='ResNet50_ImageNet_Caffe.model')

# test keras
#model = pipeline.convert_model(model_type='keras', model='keras_Average_ImageNet.keras')

# test sklearn
#model = pipeline.convert_model(model_type='scikit-learn', model='sklearn_svc.joblib', initial_types=("float_input", "FloatTensorType([1,4])"))

# test caffe
#model = pipeline.convert_model(model_type='caffe', model='bvlc_alexnet.caffemodel')

# test mxnet
#model = pipeline.convert_model(model_type='mxnet', model='resnet.json', model_params='resnet.params', model_input_shapes='(1,3,224,224)')<jupyter_output>-------------

Model Conversion



Input model is already ONNX model. Skipping conversion.



-------------

MODEL INPUT GENERATION(if needed)



Input.pb already exists. Skipping dummy input generation. 



-------------

MODEL CORRECTNESS VERIFICATION





Check the ONNX model for validity 

The ONNX model is valid.



The original model is already onnx. Skipping correctness test. 



-------------

MODEL CONVERSION SUMMARY (.json file generated at /mnt/model/test/output.json )



{'conversion_status': 'SUCCESS',

 'correctness_verified': 'SKIPPED',

 'error_message': '',

 'input_folder': '/mnt/model/test/test_data_set_0',

 'output_onnx_path': '/mnt/model/test/model.onnx'}<jupyter_text>Run parameters(1) model: string    Required. The path of the model that needs to be converted.        **IMPORTANT Only support the model path which is under the mounting directory (while initialization by Pipeline()).(2) output_onnx_path: string    Required. The path to store the converted onnx model. Should end with ".onnx". e.g. output.onnx(3) model_type: string    Required. The name of original model framework. Available types are caffe, cntk, coreml, keras, libsvm, mxnet, scikit-learn, tensorflow and pytorch.(4) model_inputs_names: string    (tensorflow) Optional. The model's input names. Required for tensorflow frozen models and checkpoints.(5) model_outputs_names: string    (tensorflow) Optional. The model's output names. Required for tensorflow frozen models checkpoints.(6) model_params: string     (mxnet) Optional. The params of the model if needed.(7) model_input_shapes: list of tuple     (pytorch, mxnet) Optional. List of tuples. The input shape(s) of the model. Each dimension separated by ','.(8) target_opset: String    Optional. Specifies the opset for ONNX, for example, 7 for ONNX 1.2, and 8 for ONNX 1.3. Defaults to '7'.    (9) caffe_model_prototxt: string    (caffe) Optional. The filename of deploy prototxt for the caffe madel. (10) initial_types: tuple (string, string)    (scikit-learn) Optional. A tuple consist two strings. The first is data type and the second is the size of tensor type e.g., ('float_input', 'FloatTensorType([1,4])')(11) input_json: string    Optional. Use JSON file as input parameters.        **IMPORTANT Only support the path which is under the mounting directory (while initialization by Pipeline()).    (12) model_inputs_names: string    (tensorflow) Optional.    (13) model_outputs_names: string    (tensorflow) Optional.      Performance test tool You can run perf_tuning using command python perf_tuning.py [Your model path] [Output path on the docker]. You can use the same arguments as for onnxruntime_pert_test tool, e.g. -m for mode, -e to specify execution provider etc. By default it will try all providers available.<jupyter_code>result = pipeline.perf_tuning(model=model)
#result = pipeline.perf_tuning()   # is ok, too<jupyter_output>2019-08-30 18:21:00.936705200 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'lstm2/cell_init'. It is not used by any node and should be removed from the model.

2019-08-30 18:21:00.936788600 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'lstm1/hidden_init'. It is not used by any node and should be removed from the model.

2019-08-30 18:21:00.936795600 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'lstm1/cell_init'. It is not used by any node and should be removed from the model.

2019-08-30 18:21:00.936800300 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'lstm2/hidden_init'. It is not used by any node and should be removed from the model.

Setting thread pool size to 1

Total time cost:0.0617255

Total iterations:20

Average time cost:3.08627 ms

2019-08-30 18:21:01.180742700 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers[...]<jupyter_text>Run parameters(1) model: string    Required. The path of the model that wants to be performed.    (2) result: string    Optional. The path of the result.    (3) config: string (choices=["Debug", "MinSizeRel", "Release", "RelWithDebInfo"])    Optional. Configuration to run. Default is "RelWithDebInfo".    (4) test_mode: string (choices=["duration", "times"])    Optional. Specifies the test mode. Value could be 'duration' or 'times'. Default is "times".(5) execution_provider: string (choices=["cpu", "cuda", "mkldnn"])    Optional. help="Specifies the provider 'cpu','cuda','mkldnn'. Default is ''.    (6) repeated_times: integer    Optional. Specifies the repeated times if running in 'times' test mode. Default:20.    (7) duration_times: integer    Optional. Specifies the seconds to run for 'duration' mode. Default:10.    (8) parallel: boolean    Optional. Use parallel executor, default (without -x): sequential executor.    (9) intra_op_num_threads: integer    Optional. Sets the number of threads used to parallelize the execution within nodes, A value of 0 means ORT will pick a default. Must >=0.    (10) inter_op_num_threads: integer    Optional. Sets the number of threads used to parallelize the execution of the graph (across nodes), A value of 0 means ORT will pick a default. Must >=0.    (11) top_n: integer    Optional. Show percentiles for top n runs. Default:5.    (12) runtime: boolean    Optional. Use this boolean flag to enable GPU if you have one.    (13) input_json: string    Optional. Use JSON file as input parameters.<jupyter_code>pipeline.print_performance(result)
#pipeline.print_result() # is ok, too<jupyter_output>mkldnn_openmp_OMP_WAIT_POLICY_active 0.7167509999999999 ms

mkldnn_openmp 0.97501 ms

mkldnn_openmp_1_threads_OMP_WAIT_POLICY_passive 1.2467949999999999 ms

mkldnn 0.903808 ms

mkldnn_1_threads 0.75656 ms

mkldnn_parallel_1_threads 5.793315000000001 ms

mklml_1_threads 1.280607 ms

mklml 1.27904 ms

mklml_parallel_1_threads 3.0862749999999997 ms

cpu_openmp_1_threads_OMP_WAIT_POLICY_active 1.2905449999999998 ms

cpu_openmp_1_threads 1.2489949999999999 ms

cpu_openmp_1_threads_OMP_WAIT_POLICY_passive 1.3137450000000002 ms

cpu_1_threads 1.451031 ms

cpu 1.304 ms

cpu_parallel_1_threads 3.044285 ms

ngraph_parallel_1_threads error

ngraph_1_threads error

ngraph error<jupyter_text>Result Visualization After performance test, there would be a directory for results. This libray use Pandas.read_json to visualize JSON file. (orient is changeable.)"latency.json" contains the raw data of results ordered by the average time. Use .latency to obtain the original latency JSON; Use .profiling to obtain original top 5 profiling JSON.<jupyter_code>r = pipeline.get_result(result)
#r.latency
#r.profiling
print(result)<jupyter_output>E:\onnx-pipeline\notebook/model/result<jupyter_text>Print latency.json Provide parameters for top 5 performace. Use the parameter "top" to visualize more results.<jupyter_code>r.prints()<jupyter_output><empty_output><jupyter_text>Print profiling.json Only provide profiling JSON for top 5 performace by giving certain index of the result. The file name is profile_[name].json(1) index: integer    Required. The index for top 5 profiling files.(2) top: integer    The number for top Ops.<jupyter_code>r.print_profiling(index=4, top=5)<jupyter_output><empty_output><jupyter_text>Get code snippets<jupyter_code>print(r.get_code(ep='cpu'))<jupyter_output>import onnxruntime as ort
so = rt.SessionOptions()
so.set_graph_optimization_level(99)
so.enable_sequential_execution = False
so.session_thread_pool_size(0)
session = rt.Session("/mnt/model/test/model.onnx", so)<jupyter_text>netron<jupyter_code># only workable for notebook in the local server 
import netron
netron.start("model/test/model.onnx", browse=False) # 'model.onnx'
from IPython.display import IFrame
IFrame('http://localhost:8080', width="100%", height=1000)
netron.stop()<jupyter_output>Stopping http://localhost:8080
{'fraction_non_alphanumeric': 0.09643294282217171, 'fraction_numerical': 0.03951739814652911, 'mean_word_length': 4.920807453416149, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 7, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '517018', 'license': 'MIT', 'n_tokens_mistral': 3824, 'n_tokens_neox': 3528, 'n_words': 1149, 'path': 'notebook/onnx-pipeline.ipynb', 'repo_name': 'KeDengMS/OLive', 'size': '11438'}