<jupyter_start><jupyter_text>Exercise: Multi Device Plugin and the DevCloudNow that we've walked through the process of using the **Multi device plugin** to load a model on all three hardware devices, CPU, GPU, and VPU, you will have the opportunity to do this yourself with the addition of running inference on an image using 3 device combinations.In this exercise, you will do the following:1. Write a Python script to load a model and run inference 1000 times on a device on Intel's DevCloud.    * Calculate the time it takes to load the model.    * Calculate the time it takes to run inference 1000 times.2. Write a shell script to submit a job to Intel's DevCloud.3. Submit a job using `qsub` on an **IEI Tank-870** edge node using `MULTI`, run `liveQStat` to view the status of your submitted jobs, and then retrieve and view the results from your job.    * One job using `CPU/VPU` as the device.    * One job using `GPU/VPU` as the device.    * One job using `CPU/GPU/VPU` as the device.4. Plot and compare the results using bar graphs with `matplotlib` for the following metrics:    * Model Loading Time    * Inference Time    * Frames Per Second (FPS)Click the **Exercise Overview** button below for a demonstration. Exercise Overview  IMPORTANT: Set up paths so we can run Dev Cloud utilitiesYou *must* run this every time you enter a Workspace session.<jupyter_code>%env PATH=/opt/conda/bin:/opt/spark-2.4.3-bin-hadoop2.7/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel_devcloud_support
import os
import sys
sys.path.insert(0, os.path.abspath('/opt/intel_devcloud_support'))
sys.path.insert(0, os.path.abspath('/opt/intel'))<jupyter_output><empty_output><jupyter_text>The ModelWe will be using the `vehicle-license-plate-detection-barrier-0106` model for this exercise.Remember to use the appropriate model precisions for each device:* GPU - `FP16`* VPU - `FP16`* CPU - It is prefered to use `FP32`, but we have to use `FP16` since **GPU** and **VPU** use `FP16`The model has already been downloaded for you in the `/data/models/intel` directory on Intel's DevCloud.We will be running inference on an image of a car. The path to the image is `/data/resources/car.png`.  Step 1: Creating a Python ScriptThe first step is to create a Python script that you can use to load the model and perform inference. We'll use the `%%writefile` magic to create a Python file called `inference_on_device.py`. In the next cell, you will need to complete the `TODO` items for this Python script.`TODO` items:1. Load the model2. Get the name of the input node3. Prepare the model for inference (create an input dictionary)4. Run inference 1000 times in a loopIf you get stuck, you can click on the **Show Solution** button below for a walkthrough with the solution code.<jupyter_code>%%writefile inference_on_device.py

import time
import numpy as np
import cv2
from openvino.inference_engine import IENetwork
from openvino.inference_engine import IEPlugin
import argparse

def main(args):
    model=args.model_path
    model_weights=model+'.bin'
    model_structure=model+'.xml'
    
    start=time.time()
    
    # TODO: Load the model
    
    load_time=time.time()-start
    print(f"Time taken to load model = {load_time} seconds")
    
    # Get the name of the input node
    
    # Reading and Preprocessing Image
    input_img=cv2.imread('car.png')
    input_img=cv2.resize(input_img, (300,300), interpolation = cv2.INTER_AREA)
    input_img=np.moveaxis(input_img, -1, 0)

    # TODO: Prepare the model for inference (create input dict etc.)
    
    start=time.time()
    for _ in range(1000):
        # TODO: Run Inference in a Loop
    
    inference_time=time.time()-start
    fps=100/inference_time
    
    print(f"Time Taken to run 1000 inference is = {inference_time} seconds")
    
    with open(f"/output/{args.path}.txt", "w") as f:
        f.write(str(load_time)+'\n')
        f.write(str(inference_time)+'\n')
        f.write(str(fps)+'\n')

if __name__=='__main__':
    parser=argparse.ArgumentParser()
    parser.add_argument('--model_path', required=True)
    parser.add_argument('--device', default=None)
    parser.add_argument('--path', default=None)
    
    args=parser.parse_args() 
    main(args)<jupyter_output><empty_output><jupyter_text>Show Solution  Step 2: Creating a Job Submission ScriptTo submit a job to the DevCloud, you'll need to create a shell script. Similar to the Python script above, we'll use the `%%writefile` magic command to create a shell script called `inference_model_job.sh`. In the next cell, you will need to complete the `TODO` items for this shell script.`TODO` items:1. Create three variables:    * `DEVICE` - Assign the value as the first argument passed into the shell script.    * `MODELPATH` - Assign the value as the second argument passed into the shell script.    * `SAVEPATH` - Assign the value as the third argument passed into the shell script.2. Call the Python script using the three variable values as the command line argumentIf you get stuck, you can click on the **Show Solution** button below for a walkthrough with the solution code.<jupyter_code>%%writefile inference_model_job.sh
#!/bin/bash

exec 1>/output/stdout.log 2>/output/stderr.log

mkdir -p /output

# TODO: Create DEVICE variable
# TODO: Create MODELPATH variable
# TODO: Create SAVEPATH variable

# TODO: Call the Python script

cd /output

tar zcvf output.tgz * # compresses all files in the current directory (output)<jupyter_output><empty_output><jupyter_text>Show Solution  Step 3: Submitting a Job to Intel's DevCloudIn the next three sub-steps, you will write your `!qsub` commands to submit your jobs to Intel's DevCloud to load your model and run inference on the **IEI Tank-870** edge node with an **Intel Core i5** CPU and an **Intel Neural Compute Stick 2** VPU. You will use the **MULTI** device plugin to run inference on three device combinations.Your `!qsub` command should take the following flags and arguments:1. The first argument should be the shell script filename2. `-d` flag - This argument should be `.`3. `-l` flag - This argument should request an edge node with an **IEI Tank-870**. The default quantity is 1, so the **1** after `nodes` is optional.     * **Intel Core i5 6500TE** for your `CPU`.    * **Intel HD Graphics 530** for your `IGPU`.    * **Intel Neural Compute Stick 2** for your `VPU`.To get the queue labels for these devices, you can go to [this link](https://devcloud.intel.com/edge/get_started/devcloud/)4. `-F` flag - This argument should contain the three values to assign to the variables of the shell script:    * **DEVICE** - Device type for the job: You will have to use `MULTI` with three different combinations of `CPU`,`GPU` or `MYRIAD`.        - `CPU,MYRIAD`        - `GPU,MYRIAD`        - `CPU,GPU,MYRIAD`    * **MODELPATH** - Full path to the model for the job. As a reminder, the model is located in `/data/models/intel`.    * **SAVEPATH** - Name of the file you want to save the performance metrics as. These should be named as the following:        - `cpu_vpu_stats` for the `CPU/VPU` job        - `gpu_vpu_stats` for the `GPU/VPU` job        - `cpu_gpu_vpu_stats` for the `CPU/GPU/VPU` job**Note**: There is an optional flag, `-N`, you may see in a few exercises. This is an argument that only works on Intel's DevCloud that allows you to name your job submission. This argument doesn't work in Udacity's workspace integration with Intel's DevCloud.  Step 3a: Running on the CPU and VPU (NCS2)In the cell below, write the qsub command that will submit your job to both the CPU and VPU (NCS2).If you get stuck, you can click on the **Show Solution** button below for a walkthrough with the solution code.<jupyter_code>cpu_vpu_job_id_core = # TODO: Write qsub command
print(cpu_vpu_job_id_core[0])<jupyter_output><empty_output><jupyter_text>Show Solution  Check Job StatusTo check on the job that was submitted, use `liveQStat` to check the status of the job. The cell is locked until this finishes polling 10 times or you can interrupt the kernel to stop it by pressing the stop button at the top: ![stop button](assets/interrupt_kernel.png)Column `S` shows the state of your running jobs.For example:- If `JOB ID`is in Q state, it is in the queue waiting for available resources.- If `JOB ID` is in R state, it is running.<jupyter_code>import liveQStat
liveQStat.liveQStat()<jupyter_output><empty_output><jupyter_text>Get ResultsRun the next cell to retrieve your job's results.<jupyter_code>import get_results

get_results.getResults(cpu_vpu_job_id_core[0], filename="output.tgz", blocking=True)<jupyter_output><empty_output><jupyter_text>Unpack your output files and view stdout.log<jupyter_code>!tar zxf output.tgz
!cat stdout.log<jupyter_output><empty_output><jupyter_text>View stderr.logThis can be used for debugging<jupyter_code>!cat stderr.log<jupyter_output><empty_output><jupyter_text>Step 3b: Running on the GPU and VPU (NCS2)In the cell below, write the qsub command that will submit your job to both the GPU and VPU (NCS2).If you get stuck, you can click on the **Show Solution** button below for a walkthrough with the solution code.<jupyter_code>gpu_vpu_job_id_core = # TODO: Write qsub command
print(gpu_vpu_job_id_core[0])<jupyter_output><empty_output><jupyter_text>Show Solution  Check Job StatusTo check on the job that was submitted, use `liveQStat` to check the status of the job. The cell is locked until this finishes polling 10 times or you can interrupt the kernel to stop it by pressing the stop button at the top: ![stop button](assets/interrupt_kernel.png)Column `S` shows the state of your running jobs.For example:- If `JOB ID`is in Q state, it is in the queue waiting for available resources.- If `JOB ID` is in R state, it is running.<jupyter_code>import liveQStat
liveQStat.liveQStat()<jupyter_output><empty_output><jupyter_text>Get ResultsRun the next cell to retrieve your job's results.<jupyter_code>import get_results

get_results.getResults(gpu_vpu_job_id_core[0], filename="output.tgz", blocking=True)<jupyter_output><empty_output><jupyter_text>Unpack your output files and view stdout.log<jupyter_code>!tar zxf output.tgz
!cat stdout.log<jupyter_output><empty_output><jupyter_text>View stderr.logThis can be used for debugging<jupyter_code>!cat stderr.log<jupyter_output><empty_output><jupyter_text>Step 3c: Running on the CPU, GPU and VPU (NCS2)In the cell below, write the qsub command that will submit your job to all three devices, CPU, GPU, and VPU (NCS2).If you get stuck, you can click on the **Show Solution** button below for a walkthrough with the solution code.<jupyter_code>cpu_gpu_vpu_job_id_core = # TODO: Write qsub command
print(cpu_gpu_vpu_job_id_core[0])<jupyter_output><empty_output><jupyter_text>Show Solution  Check Job StatusTo check on the job that was submitted, use `liveQStat` to check the status of the job. The cell is locked until this finishes polling 10 times or you can interrupt the kernel to stop it by pressing the stop button at the top: ![stop button](assets/interrupt_kernel.png)Column `S` shows the state of your running jobs.For example:- If `JOB ID`is in Q state, it is in the queue waiting for available resources.- If `JOB ID` is in R state, it is running.<jupyter_code>import liveQStat
liveQStat.liveQStat()<jupyter_output><empty_output><jupyter_text>Get ResultsRun the next cell to retrieve your job's results.<jupyter_code>import get_results

get_results.getResults(cpu_gpu_vpu_job_id_core[0], filename="output.tgz", blocking=True)<jupyter_output><empty_output><jupyter_text>Unpack your output files and view stdout.log<jupyter_code>!tar zxf output.tgz
!cat stdout.log<jupyter_output><empty_output><jupyter_text>View stderr.logThis can be used for debugging<jupyter_code>!cat stderr.log<jupyter_output><empty_output><jupyter_text>Step 4: Plot and Compare ResultsRun the cells below to plot and compare the results.<jupyter_code>import matplotlib.pyplot as plt
def plot(labels, data, title, label):
    fig = plt.figure()
    ax = fig.add_axes([0,0,1,1])
    ax.set_ylabel(label)
    ax.set_title(title)
    ax.bar(labels, data)
    
def read_files(paths, labels):
    load_time=[]
    inference_time=[]
    fps=[]
    
    for path in paths:
        if os.path.isfile(path):
            f=open(path, 'r')
            load_time.append(float(f.readline()))
            inference_time.append(float(f.readline()))
            fps.append(float(f.readline()))

    plot(labels, load_time, 'Model Load Time', 'seconds')
    plot(labels, inference_time, 'Inference Time', 'seconds')
    plot(labels, fps, 'Frames per Second', 'Frames')

paths=['cpu_vpu_stats.txt', 'gpu_vpu_stats.txt', 'cpu_gpu_vpu_stats.txt']
read_files(paths, ['CPU/VPU', 'GPU/VPU', 'CPU/GPU/VPU'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09129213483146068, 'fraction_numerical': 0.009675405742821473, 'mean_word_length': 4.5823170731707314, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 79, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '258441', 'license': 'MIT', 'n_tokens_mistral': 4063, 'n_tokens_neox': 3891, 'n_words': 1536, 'path': 'Jupyter Tutorial/Exercise_Multi_Device_Plugin_and_the_DevCloud.ipynb', 'repo_name': 'HuyDucVo/openvino-smart-queue-monitoring-system', 'size': '12816'}