<jupyter_start><jupyter_text>Compare Model Zoo Benchmark performance between Intel optimized and stock PyTorchThis jupyter notebook will help you evaluate performance benefits from Intel-optimized Tensorflow via several pre-trained models from Intel Model Zoo. The notebook will show users a bar chart like below for performance comparison among Stock and Intel Tensorflow.   Get Platform Information<jupyter_code># ignore all warning messages
import warnings
warnings.filterwarnings('ignore')
from profiling.profile_utils import PlatformUtils
plat_utils = PlatformUtils()
plat_utils.dump_platform_info()<jupyter_output>/tmp/models/docs/notebooks/perf_analysis/pytorch/profiling/../../../../../benchmarks/common
==================== CPU Info ====================
Physical cores per socket: 28
Total physical cores: 56
Max Frequency: 4000.0
Min Frequency: 1000.0
Socket Number: 2
==================== Memory Information ====================
Total:  251 GB<jupyter_text>Section 1: Run the benchmark on the selected Jupyter Kernels  Step 1: Check TensorFlow version and MKL enablement<jupyter_code>import torch
print ("We are using PyTorch version", torch.__version__)
try:
    import intel_pytorch_extension as ipex
    print("IPEX version: {}".format(ipex.__version__))
    has_ipex = True
except ImportError:
    print("WARNING: IPEX not available")
    has_ipex = False
search_str="USE_MKL="
offset = len(search_str)
config_str = torch.__config__.show()
idx = config_str.find(search_str)
if config_str[idx+offset:idx+offset+2] == "ON":
    on_mkl = True
else:
    on_mkl = False
print("MKL enabled:", on_mkl)<jupyter_output>We are using PyTorch version 1.7.0a0+e85d494
IPEX version: 1.2.0
MKL enabled: False<jupyter_text>Step 2: Configure parameters for launch_benchmark.py according to the selected Topology  Step 2.1: List out the supported topologies<jupyter_code>import sys
from profiling.profile_utils import ConfigFile

accuracy_only=False

config = ConfigFile()
sections = config.read_supported_section(accuracy_only=accuracy_only)
print("Supported topologies: ")
index = 0 
for section in sections:
    print(" %d: %s " %(index, section))
    index+=1<jupyter_output>Supported topologies: 
 0: resnet50 infer fp32 
 1: vgg16 infer fp32<jupyter_text>Step 2.2: Pick a topology.  ACTION : Please select one supported topology and change topo_index accordingly<jupyter_code># User picks a topology, Batch Size, and number of required threads
## USER INPUT
topo_index=0<jupyter_output><empty_output><jupyter_text>List out the selected topology name<jupyter_code>if topo_index >= len(sections):
    print("ERROR! please input a topo_index within range")
else:
    topology_name=sections[topo_index]
    print(topology_name)
import os
# Users should change ModelZooRoot path according to their environment
## USER INPUT
current_path = os.getcwd()
os.environ['ModelZooRoot'] = current_path + "/../../../../"
print(os.environ['ModelZooRoot'])<jupyter_output>/tmp/models/docs/notebooks/perf_analysis/pytorch/../../../../<jupyter_text>Check those mandatory python scripts after users assign ModelZooRoot and ProfileUtilsRoot<jupyter_code>sys.path.append(os.environ['ModelZooRoot']+os.sep+'benchmarks/common/')
from platform_util import CPUInfo 
cpu_info = CPUInfo()
import psutil
import subprocess
import os
cpu_count = cpu_info.cores_per_socket
cpu_socket_count =  cpu_info.sockets
print("CPU count per socket:" ,  cpu_count ," \nSocket count:",cpu_socket_count)<jupyter_output>CPU count per socket: 28  
Socket count: 2<jupyter_text>ACTION: Users can change the value of thread_number and batch_size to see different performance1. thread_umber: the value will apply to num_cores parameters in launch_benchmark.py  2. utilized_socket_number:  the value will apply to the socket-id parameter in launch_benchmark.py 3. num_inter_threads: the value will  apply to the num-inter-threads parameter in launch_benchmark.py 4. num_intra_threads: the value will  apply to the num-intra-threads parameter in launch_benchmark.py 5. batch_size: the value will apply to the batch_size parameter in launch_benchmark.py 6. log_folder: the folder where the logs are stored.<jupyter_code>## USER INPUT
thread_number=2 
utilized_socket_number=2 #cpu_socket_count
num_inter_threads = utilized_socket_number
num_intra_threads = thread_number
batch_size=32
log_folder=os.getcwd() + os.sep + "logs"
import os
current_path = os.getcwd()
log_dir = os.path.join(current_path, "logs")
log_filename = topology_name.replace(' ', '_') + '.log'
log_path = os.path.join(log_dir, log_filename)
from torch.distributed import launch<jupyter_output><empty_output><jupyter_text>Step 2.3: get required data and files if needed.No action if there is no output from this below cell.<jupyter_code># Get the parameters from config
config = ConfigFile()
config.read_config(topology_name)
if config.data_download != '' and config.data_location == '':
    print("\nPlease follow below command to get the data : ")
    val = config.read_value_from_section(topology_name, 'data-download')
    print(val)<jupyter_output><empty_output><jupyter_text>ACTION: Users set the configuration in topo.ini for those required data and files1. data_download_path: the value will be set as data-location parameters in topo.ini for the related topology.2. model_source_dir: the value will be set as the model-source-dir parameter in topo.ini for the related topology such as Wide and Deep. Those models use tensorflow-models.<jupyter_code>## USER INPUT
data_download_path = ''
model_source_dir = ''

# Overwrite configurations in topo.ini
config = ConfigFile()
config.read_config(topology_name)

if config.data_download != '' and data_download_path != '':
    config.write_value_from_section(topology_name, 'data-location', data_download_path)
    config.data_location = data_download_path
    
if accuracy_only == True and config.data_location == '':
    print("ERROR! STOP! need data for accuacy evaluatoin!")<jupyter_output><empty_output><jupyter_text>Step 2.4: Prepare pre-trained model and model parameters for running the benchmark1. Get related parameters according to selected topology2. Get pretrained model if needed<jupyter_code>config = ConfigFile()
configvals = []
# Get common parameters according to users' inputs
params = ["torch.distributed.launch"]
params.extend(config.get_launch_parameters(configvals, thread_number=thread_number,
                                socket_number=utilized_socket_number, num_inter_threads=num_inter_threads,
                                num_intra_threads=num_intra_threads,
                                accuracy_only=False))
main_script = os.environ['ModelZooRoot']+os.sep+'models/image_recognition/pytorch/common/main.py'
params += [main_script]
params.extend(config.get_main_parameters(topology_name = topology_name.split(" ")[0], mode = "infer",
                                         batch_size=batch_size, has_ipex = has_ipex,
                                         precision="fp32", warmup_iter=30, log_path=log_path))

    
sys.argv=params
print(sys.argv)<jupyter_output>['torch.distributed.launch', '--nnodes=2', '--node_rank=2', '--nproc_per_node=2', '--use_env', '/tmp/models/docs/notebooks/perf_analysis/pytorch/../../../..//models/image_recognition/pytorch/common/main.py', '--batch-size', '32', '--arch', 'vgg16', '--dummy', '--evaluate', '--pretrained', '--ipex', '--precision', 'fp32', '--jit', '--warmup-iterations', '30', '--log-path=/tmp/models/docs/notebooks/perf_analysis/pytorch/logs/vgg16_infer_fp32.log']<jupyter_text>Step 2.5: Create a CSV file to log the performance numbers<jupyter_code>from profiling.profile_utils import PerfPresenter
job_type = topology_name.split(' ')[1]#'inference'
csv_fname=job_type+'_'+topology_name.replace(' ', '_')+'.csv'
perfp=PerfPresenter()
perfp.create_csv_logfile(job_type, csv_fname)<jupyter_output>file exists<jupyter_text>Step 3:  Run the benchmark<jupyter_code>launch.main()<jupyter_output>*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************<jupyter_text>Step 4: Parse output for performance number  Found the file path of the related runtime log.<jupyter_code># identify the path of the latest log file
configvals=config.read_config(topology_name)
used_logpath = log_path
print(used_logpath)<jupyter_output>/tmp/models/docs/notebooks/perf_analysis/pytorch/logs/vgg16_infer_fp32.log<jupyter_text>Parse the logfile for performance number.<jupyter_code>if accuracy_only == False:
    print("get throughput")
    val = config.throughput_keyword
    index = int(config.throughput_index)
    line = perfp.read_throughput(used_logpath, keyword=val, index=index)
    if line!=None:
        throughput=line
        print(throughput)
        # log the perf number
        perfp.log_infer_perfcsv(0, throughput, 0, csv_fname, ipex_enabled = has_ipex)
    else:
        print("ERROR! can't find correct performance number from log. please check log for runtime issues")
else:
    # get accuracy number and caculate throughput
    print("get accuracy and throughput")
    #val = config.throughput_keyword
    #index = int(config.throughput_index)
    accuracy = perfp.read_accuracy(used_logpath)
    iternation = perfp.read_iteration_time(used_logpath)
    if accuracy != [] and iternation != []:
        final_accuracy=accuracy[-1]
        iternation_time = iternation[-1]
        throughput = float(batch_size)/iternation_time
        print(final_accuracy,throughput)
        # log the perf number
        perfp.log_infer_perfcsv(0, throughput, final_accuracy, csv_fname)
    else:
        print("ERROR! can't find correct performance number from log. please check log for runtime issues")<jupyter_output>get throughput
inference performance 5.456 fps

5.456
5.456<jupyter_text>Optional : print out the log file for runtime issues<jupyter_code>logfile = open(used_logpath)
logout = logfile.read()
print(logout)<jupyter_output>=> using pre-trained model 'vgg16'
using CPU, this will be slow
using ipex model to do inference
running jit fusion path
using dummy input data to run
Test: [ 0/50]	Time  0.000 ( 0.000)	Loss 9.0608e+00 (9.0608e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
Test: [10/50]	Time  0.000 ( 0.000)	Loss 9.0608e+00 (9.0608e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
Test: [20/50]	Time  0.000 ( 0.000)	Loss 9.0608e+00 (9.0608e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
Test: [30/50]	Time  5.866 ( 5.866)	Loss 9.0608e+00 (9.0608e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
Test: [40/50]	Time  5.867 ( 5.867)	Loss 9.0608e+00 (9.0608e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
inference latency 183.273 ms
inference performance 5.456 fps
 * Acc@1 0.000 Acc@5 0.000
DoneClosing<jupyter_text>Step 5: Draw the performance comparison diagram>NOTE: Please go over Section 1 on different Jupyter kernel before comparison<jupyter_code>import matplotlib
%matplotlib inline
from profiling.profile_utils import PerfPresenter

perfp=PerfPresenter(True)
# inference  throughput
perfp.draw_perf_diag_from_csv(csv_fname,'throughput','throughput (image/sec)', topology_name)
perfp.draw_perf_ratio_diag_from_csv(csv_fname,'throughput','speedup', topology_name)
if accuracy_only == True:
    perfp.draw_perf_diag_from_csv(csv_fname,'accuracy','accuracy', topology_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11341728180633956, 'fraction_numerical': 0.029700390794615717, 'mean_word_length': 5.04514435695538, 'pattern_counts': {'":': 1, '<': 54, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '536732', 'license': 'Apache-2.0', 'n_tokens_mistral': 3921, 'n_tokens_neox': 3662, 'n_words': 975, 'path': 'docs/notebooks/perf_analysis/pytorch/.ipynb_checkpoints/benchmark_perf_comparison-checkpoint.ipynb', 'repo_name': 'Pandinosaurus/models-1', 'size': '11515'}