<jupyter_start><jupyter_text>Learning how to run benchmarks on DevCloudOpenVINO includes a lot of sample and demo programs in the package. The most of sample programs and demo programs are intended to demonstrate the capability and functionality of the OpenVINO, or a simple sample program to demonstrate how to use the OpenVINO APIs to the users. **They are not designed to be used for performance benchmarking.**  Among those sample applications, `benchmark_app` sample application is developed and specialized for benchmaking.   You can specify many option parameters to the `benchmark_app` but it will choose the appropriate configuration for the specified hardware when you ommit those option parameters.  - Automatically choose and apply appropriate option parameters (note: may not be the best)- Use asynchronous and simultaneous inferencing for higher throughput- Almost all DL models can be used - benchmark_app won't care the input and output of the model. It purely run the model and measure the performance of the processing time - Output data won't be cared nor parsed. benchmark_app won't care the number of outputs and format of the outputs - benchmark_app doesn't require any input data. It won't care the shape, format and number of inputs- Both C++ version and Python version are available Here, we'll use the Python version of Benchmark tool and learn how to run benchmarking on the DevCloud In this section, you'll learn following things:- How to run a benchmark_app on the **host server system**- How to submit a job and run benchmark_app on **an edge computing node**- How to submit a job and run benchmark_app on **a specific edge computing node** ---- 1. Learn how to run benchmark_app on the development serverHere, we'll use the Python version of benchmark_app. It is convenient because it doesn't require compilation to use. (C++ version is available too)  It requires a deep learning model to run. We'll use the `Squeezenet1.1` model for this exercise.  1.1. Copy benchmark_app`benchmark_app.py` is in the OpenVINOの install directory. For convenience, copy the files to the current directory.<jupyter_code>!pwd
!cp -r $INTEL_OPENVINO_DIR/deployment_tools/tools/benchmark_tool .
!ls -l benchmark_tool<jupyter_output><empty_output><jupyter_text>1.2. Install required Python modules for the benchmark_appInstalls Python modules described in `reqirements.txt` file using `pip3` command. This is requied only the first time you do this.<jupyter_code>!pip3 install -r benchmark_tool/requirements.txt<jupyter_output><empty_output><jupyter_text>1.3. Download a neural network modelDownloads `squeezenet 1.1` DL model using an OpenVINO utility tool `Model downloader`. The `Model downloader` downloads the specified DL models. You can list the available models with `--print_all` option to the `Model downloader`.  The downloaded model is in the Caffe format. **You need to convert the model into IR model** which is the DL model format OpenVINO can read.<jupyter_code># Download a Squeezenet1.1 model (Caffe format)
!python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/downloader.py --name squeezenet1.1
# (optional) List all available models with Model downloader (OMZ=Open Model Zoo, OpenVINO model zoo)
!python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/downloader.py --print_all<jupyter_output><empty_output><jupyter_text>1.4. Convert the DL model from Caffe formato to IR formatIn general, `Model Optimizer` is the tool to convert the trained DL models into OpenVINO IR model format.  However, the DL model downloaded with `Model downloader` can be converted into IR model with `Model converter`. The `Model converter` is a front-end tool for the `Model optimizer` and it will call `Model optimizer` with appropriate option parameters for the model.<jupyter_code>!python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/converter.py --name squeezenet1.1 --precisions FP16<jupyter_output><empty_output><jupyter_text>1.5. Check the converted IR model filesIR model consists of 2 files, `.xml` and `.bin`. `.xml` represents the topology of the model (graph) and `.bin` contains weight and bias data of the model.<jupyter_code>!ls -l public/squeezenet1.1/FP16<jupyter_output><empty_output><jupyter_text>1.6. Running benchmark_app on the development serverFirst, let's try to run the benchmark_app on the **development server**. The development server is the machine which you are currently operating. It's a Xeon server and doesn't have neither integrated GPU nor DL accelerators such as `VPU` or `FPGA`. The `CPU` is the only available processor on the machine.    You can directly run the job on the development server without submitting a job. The job won't be kept waiting in the queue. It's simple as you run an OpenVINO application on the local machine.  Run the benchmark_app and check the performance result on the bottom of the log. benchmark_app options:  - `-m` specify the IR model file (.xml)- `-niter` specify the number of inference in the benchmarking<jupyter_code>!python3 benchmark_tool/benchmark_app.py \
        -m public/squeezenet1.1/FP16/squeezenet1.1.xml \
        -niter 100<jupyter_output><empty_output><jupyter_text>---- 2. Running benchmark_app on an edge inference nodeSend a benchmark job and run it on an edge compute node  2.1 Create a job script file to submit  Use `%%writefile` magic command to create a job script file which contains the tasks you want to run. `%%writefile` magic command will write the rest of contents in the cell to a file.<jupyter_code>%%writefile job.sh
cd ~/devcloud-workshop-en
pip3 install -r benchmark_tool/requirements.txt
python3 benchmark_tool/benchmark_app.py \
        -m public/squeezenet1.1/FP16/squeezenet1.1.xml \
        -niter 100 \
        $*
echo completed<jupyter_output><empty_output><jupyter_text>2.2 Send a job script to an edge computing nodeHere, we re-use the same Python code which we used in previous section to detect the completion of the job.  The job submitted with `qsub` command will be run on one of available edge computing node adhocery.<jupyter_code># submit a job
job_id=!qsub job.sh

# generate log file name from job_id
job_num = job_id[0].split('.')[0]
log_file='job.sh.o'+job_num
err_file='job.sh.e'+job_num
print('job_id={}, log_file={}'.format(job_id, log_file))

import time
def waitForJobCompletion(jobNumber):
    print('Waiting for job completion...', end='')
    running=True
    while running:
        time.sleep(1)
        running=False
        status_list=!qstat         # Check job status
        for status in status_list:
            if jobNumber in status:    # if job_num is found in the status list, the job is still running
                running = True
        print(status.split()[4], end='')
    print('...Job {} completed'.format(job_num))    

# wait for the job to complete
waitForJobCompletion(job_num)<jupyter_output><empty_output><jupyter_text>2.3 Check the log fileThe name of the node which run your job is displayed in `Resources:` line in the log file. Check which node is used to run the job.<jupyter_code># The log_file is set in the previous cell
import os
os.environ['log_file']=log_file

!cat $log_file<jupyter_output><empty_output><jupyter_text>---- 3. Running a job on a specific edge compute nodeIn previous section, you run a job without specifying the edge compute node. In this case, the job will be run on one of available edge computing node. You need to control the edge computing node to run your job when you run a benchmark.  In the DevCloud, wide variety of edge computing node with different hardware configuration are available. From Atom to Xeon, and various accelerators such as VPU and FPGA are available.  To display the list of available edge computing node, you can use `pbsnodes` command. The node name you need to specify the node can be found in the line of `properties =` in the output of the `pbsnodes` command.  The node name is displayed as a comma-separated list. For instance, your job will be one of edge computing node which contains `intel-hd-530` in the property field when you specify `intel-hd-530` as the node name to run your job. You can specify more specific hardware with more specific and dedicated node name. For example, if you specify `idc001skl` as the node name, your job will be run exactly on that node.  3.1 Display available edge computing nodesFirst, let's list the available edge computing nodes. The `pbsnodes` command will list the nodes.<jupyter_code>!pbsnodes | grep "properties =" | sort | uniq -c<jupyter_output><empty_output><jupyter_text>3.2 Submit a benchmarking job to a specific edge computing nodeUsing `qsub` command and specifying the compute node name with `-l` to submit a benchmarking job. In addition, using `-F` option to pass command option parameters to the job script file.~~~job_id=!qsub -l nodes=1:gold6138 job.sh -F "-d CPU"~~~- `qsub` command - You can use `-l` (limit) option to specify the edge computing node to run your job in `qsub` command. For example, if you want to run the job on a `skylake` node, you can specify it like this way `qsub -l nodes=1:skylake job.sh` - You can pass parameters to the job script file with `-F` option. Please be aware that you have to make sure that the job script file can receive the parameters  - We'll re-use the `job.sh` job script file which we have created in the previous section.- Log file name - Here, we'll auto generate the job file name using a Python snippet from the `job ID` (a list of string like `['27214.v-qsvr-1.devcloud-edge']`) returned from the `qsub` command- Job completion detection - You run `qstat` command multiple times manually to know the completion of a job in previous section. Here, we'll use a Python snippet to detect it automatically  Note: benchmark_app option parametersbenchmark_app has lot of option parameters. You can check available options with `--help` option. Following are the typical options for benchmark_app.  Here, we specifies `CPU` as the processor to run the benchmark with `"-d CPU"` but you can specify other accelerators such as integrated GPU (`"GPU"`), VPU (`"MYRIAD"`) and so on. You have to make sure that the specified edge computing node supports the processor you specified. |Option|Description||:--|:--||`-m` PATH_TO_MODEL|Specifies IR model to run (`.xml`)||`-d` TARGET_DEVICE|Specified processor to run inference. `CPU`, `GPU`, `MYRIAD`, `HDDL`, `HETERO:FPGA,CPU` and some others are available||`-niter` NUMBER_ITERATIONS|Number of inference to run. If you ommit this, benchmark will run for 1 min||`-nireq` NUMBER_INFER_REQUESTS|Number of inference simultaneously. If you specifies 4, 4 inference request will run at a time. To achieve higher throughput, you need to specify appropriate number of simultaneous inferencing. If you ommit this, benchmark_app will use appropriate number for  the inference processor.||`-b` BATCH_SIZE|number of batch in an inferencing||`-i` PATH_TO_INPUT|Specify the input image. You can ommit this.||`-pc`|Enable performance counter. You can get the detailed layer by layer performance report|<jupyter_code># submit a job
job_id=!qsub -l nodes=1:gold6138 job.sh -F "-d CPU"

# generate log file name from job_id
print('job_id=', job_id)
job_num = job_id[0].split('.')[0]
log_file='job.sh.o'+job_num
err_file='job.sh.e'+job_num
print('log_file=', log_file)

# wait for the job to complete
waitForJobCompletion(job_num)<jupyter_output><empty_output><jupyter_text>3.3 Check the job resultWe just displayed entire output log in the previous section. We'll use grep and display the minimum required information to make it concise.<jupyter_code>os.environ['log_file']=log_file

!grep '# Resources:'                                  $log_file
!grep '\[ INFO \] Device info'                   -A 3 $log_file
!grep '\[Step 11/11\] Dumping statistics report' -A 4 $log_file<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07361087795870405, 'fraction_numerical': 0.009316770186335404, 'mean_word_length': 4.604421448730009, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '299245', 'license': 'Apache-2.0', 'n_tokens_mistral': 3442, 'n_tokens_neox': 3269, 'n_words': 1580, 'path': 'benchmarking.ipynb', 'repo_name': 'yas-sim/devcloud-workshop-en', 'size': '11914'}