<jupyter_start><jupyter_text>[APPENDIX] Automate a complex and troublesome testing You may want to try as many conditions as possible to test your program on the DevCloud.  If you do it manually, it is a time consuming and boring task. Also, there's a risk of skipping a condition unintentionally. It is inefficient and low reproducibility.  Here, we introduce a simple example to make it automated using a piece of Python code. We'll use an OpenVINO sample application `benchmark_app` as an example and use `googlenet-v1` and `squeezenet1.1` as a DL model for this test.  1. Preparing required filesCopy `benchmark_app` from the OpenVINO install directory.  Download and convert DL models (`squeezenet1.1` and `googlenet-v1`) using `Model downloader` and `Model converter`. You'll get DL models in IR format.<jupyter_code>!cp -r $INTEL_OPENVINO_DIR/deployment_tools/tools/benchmark_tool .
!python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/downloader.py --name squeezenet1.1,googlenet-v1
!python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/converter.py  --name squeezenet1.1,googlenet-v1 --precisions FP16<jupyter_output><empty_output><jupyter_text>2. Defines a class and function for test automationA class and some functions are defined here. The `autoParam` is a class to update parameter state and generate option parameter string, and functions are to simplify job control. You can easily automate your task with those class and functions. You can register the test automation class by executing the cell below (The next cell just register the class. Nothing will be displayed)    - `autoParam class` : Update parameter state (`update()`) and generate option parameter string (`get()`) from a given parameter list- `waitForJobCompletion(jobNum)`: Wait for the completion of specified job. Display the status of the job while waitin for the job completion.- `runjob(nodeName, jobFile, params)` : Run a job on the specified node with. Pass option parameter to the job with run it.- `printLineOfInterest(logFile, matchString)` : Display the lines which include the `matchString` from a file<jupyter_code># Automated test library
# Intel Corporation
# <NAME>

#           0      1       2           3      4    5
# val:   prefix, type, initial,      start, end, step
# enum:  prefix, type, initial_idx, item1, item2, item3,...
#
# e.g. parameter_list = [ [ '-nireq', 'val', 0, 0, 5, 1 ], [ '-flag', 'enum', 0, 'true', 'false', 'none'], [ '-precision', 'val', 0.5, 0.3, 0.8, 0.1 ] ]

# usage example:
# a=autoParam(parameter_list)
# print(a.get())
# a.update()

class autoParam:
    def __init__(self, param_list):
        self.param_list = param_list

    def update(self):
        exit_flag=False
        for param in self.param_list:
            if param[1]=='val':
                param[2]+=param[5]
                if param[2]>param[4]:
                    param[2]=param[3]
                else:
                    exit_flag=True
            elif param[1]=='enum':
                param[2]+=1
                if param[2]>=len(param)-3:
                    param[2]=0
                else:
                    exit_flag=True
            if exit_flag:
                return False
        return True                 # reached to end of the update loop (==Performed all parameter combination)

    def get(self):
        ret=''
        for param in self.param_list:
            if param[1]=='val':
                tmp=' {} {}'.format(param[0], param[2])
            elif param[1]=='enum':
                tmp=' {} {}'.format(param[0], param[param[2]+3])
            ret+=tmp
        return ret

# -----------------------------

import time
def waitForJobCompletion(jobNumber):
    print('Waiting for job completion...', end='')
    running=True
    while running:
        time.sleep(1)
        running=False
        status_list=!qstat           # Check job status
        for status in status_list:
            if jobNumber in status:  # if job_num is found in the status list, the job is still running
                running = True
        print(status.split()[4], end='')
    print('...Job {} completed'.format(jobNumber))    

import subprocess
def runJob(node_name, jobFile, params):
    # submit a job
    cmd='qsub -l {} {} -F "{}"'.format(node_name, jobFile, params)
    print(cmd)
    job_id=subprocess.check_output(cmd, shell=True).decode().strip()
    job_num = job_id.split('.')[0]
    print('job_id=', job_id)

    # wait for the job to complete
    waitForJobCompletion(job_num)
    return job_num

def printLineOfInterest(logFile, matchString):
    for l in [ line.strip() for line in open(logFile).readlines() ]:
        if matchString in l:
            print(l)<jupyter_output><empty_output><jupyter_text>3. Generate a job script fileCreate a job script text file with `%%writefile` magic command.  The point here is, using `$*` in the script to make the script file accept the option parameters.<jupyter_code>%%writefile job.sh
root=~/devcloud-workshop-en
cd $root
python3 benchmark_tool/benchmark_app.py $*<jupyter_output><empty_output><jupyter_text>4. Learning how to use autoParamYou can set a parameter list with autoParam constructor and autoParam can update and generate the test option parameter string. - `autoParam.update()` : Update the state of the parameters. The update starts from the top parameter and if the parameter value reached to the end value, the parameter value will roll-back to the start value, and then the next parameter will be updated (something like carry in counting up). This returns `True` in case the all parameter reached to the end value and `False` in other cases.- `autoParam.get()` : Generate a parameter string based on the current parameter values  Parameter list formatBelow is the format of the parameter list. You can specify either `val` or `enum` as the parameter type.  `val` will increase the value from `start` to `end` with `step` increment. You can use real (floating point) values but need to use care of the precision and accuracy of the real value.  `enum` will use `item`s listed in order. You can specify discrete or random values with `enum` instead of using `val`  Both type start with the `initial_value` (not start with the `start` value). This will allow you to resume a suspended testing from the middle.  ~~~val:   [ option_flag_string , 'val',  initial_val, start, end, step ]enum:  [ option_flag_string , 'enum', initial_idx, item1, item2, item3,... ]~~~ Run the cell below and check how it works.<jupyter_code>param_list = [ [ '-niter', 'enum', 0, '10', '100' ],
               [ '-nireq', 'val',  1, 1, 4, 1 ],
               [ '-m',     'enum', 0, 'public/squeezenet1.1/FP16/squeezenet1.1.xml', 'public/googlenet-v1/FP16/googlenet-v1.xml' ]
             ]

param=autoParam(param_list)
status=False
while status==False:
    param_str = param.get()
    print(param_str)
    status=param.update()<jupyter_output><empty_output><jupyter_text>5. Running an automated testingLet's run a test using the automation library.  As you can see below, a simple Python script can automate the testing on DevCloud. In this script, `printLineOfinterest()` function is used to display only required information to make the report concise. Then, we are ready to run the automated test. Let's run the cell below.  Check the performance variation by given parameter.<jupyter_code>param_list = [
    [ '-niter', 'enum', 0, '10', '100' ],
    [ '-nireq', 'val',  1, 1, 4, 1 ],
    [ '-m',     'enum', 0, 'public/squeezenet1.1/FP16/squeezenet1.1.xml', 
                           'public/googlenet-v1/FP16/googlenet-v1.xml' ]
]

param=autoParam(param_list)
status=False
while status==False:
    param_str = param.get()
    job_num = runJob('nodes=1:skylake', 'job.sh', param_str)

    # Display the result from the log file
    log_file = 'job.sh.o'+job_num
    printLineOfInterest(log_file, 'Resources:')
    printLineOfInterest(log_file, 'Count:')
    printLineOfInterest(log_file, 'Duration:')
    printLineOfInterest(log_file, 'Latency:')
    printLineOfInterest(log_file, 'Throughput:')

    status=param.update()     # update option parameter list for next iteration
    print('-'*40)

print('All test completed')<jupyter_output><empty_output><jupyter_text>6. Cleaning upMany output log files will be generated. Let's delete them to clean up the directory.<jupyter_code>!rm job.sh.o*
!rm job.sh.e*<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10773971790920943, 'fraction_numerical': 0.01386748844375963, 'mean_word_length': 3.3161125319693094, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '335401', 'license': 'Apache-2.0', 'n_tokens_mistral': 2591, 'n_tokens_neox': 2462, 'n_words': 905, 'path': 'automated-testing.ipynb', 'repo_name': 'yas-sim/devcloud-workshop-en', 'size': '8447'}