<jupyter_start><jupyter_text>Intel® Neural Compressor Sample for Tensorflow  IntroductionThis is a demo to show an End-To-End pipeline to speed up AI model by Intel® Neural Compressor.1. Train a CNN AlexNet model by Keras and Intel Optimization for Tensorflow based on dataset MNIST.2. Quantize the frozen PB model file by Intel® Neural Compressor to INT8 model.3. Test and compare the performance of FP32 and INT8 model by same script.  CodePlease refer to [README.md](README.md).  Edit script (Optional)<jupyter_code>%%writefile run_inc_ft_mnist_sample.sh
#!/bin/bash

echo "Enable Conda Env."
source /glob/development-tools/versions/oneapi/2022.1.1/oneapi/intelpython/python3.9/etc/profile.d/conda.sh
#conda activate user_tensorflow
conda activate /data/oneapi_workshop/INC

echo "Train Model by Keras/Tensorflow with MNIST"
python keras_tf_train_mnist.py

FP32_FILE="fp32_frozen.pb"
if [ ! -f $FP32_FILE ]; then
    echo "$FP32_FILE not exists."
    echo "Train AlexNet model is fault, exit!"
    exit 1
else
    echo "Training is finished"
fi

echo "Enable Intel Optimization for Tensorflow by exporting TF_ENABLE_MKL_NATIVE_FORMAT=0"
echo "Intel Optimized TensorFlow 2.5.0 and later require to set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 before running Intel® Neural Compressor quantize Fp32 model or deploying the quantized model."

export TF_ENABLE_MKL_NATIVE_FORMAT=0

echo "Quantize Model by Intel Neural Compressor"
python inc_quantize_model.py

INT8_FILE="alexnet_int8_model.pb"
if [ ! -f $INT8_FILE ]; then
    echo "$INT8_FILE not exists."
    echo "Quantize FP32 model is fault, exit!"
    exit 1
else
    echo "Quantization is finished"
fi

echo "Execute the profiling_inc.py with FP32 model file"
python profiling_inc.py --input-graph=./fp32_frozen.pb --omp-num-threads=4 --num-inter-threads=1 --num-intra-threads=4 --index=32
echo "FP32 performance test is finished"

echo "Execute the profiling_inc.py with INT8 model file"
python profiling_inc.py --input-graph=./alexnet_int8_model.pb --omp-num-threads=4 --num-inter-threads=1 --num-intra-threads=4 --index=8
echo "INT8 performance test is finished"

echo "Compare the Performance of FP32 and INT8 Models"
python compare_perf.py
echo "Please check the PNG files to see the performance!"

if [[ $? -eq 0 ]]
then
  echo "This demo is finished successfully!"
else
  echo "This demo is fault!"
fi

echo "Thank you!"<jupyter_output><empty_output><jupyter_text>Check Script<jupyter_code>!cat run_inc_ft_mnist_sample.sh<jupyter_output><empty_output><jupyter_text>Prepare Running EnvironmentPlease refer to [README.md](README.md).  Remove all old output files (Optional)<jupyter_code>!rm -rf run_inc_ft_mnist_sample.sh.*<jupyter_output><empty_output><jupyter_text>Run in Intel® DevCloudJob submit to compute node with the property 'clx' or 'icx' or 'spr' which support Intel® Deep Learning Boost (avx512_vnni).<jupyter_code>!qsub run_inc_ft_mnist_sample.sh -d `pwd` -l nodes=1:icx:ppn=2<jupyter_output><empty_output><jupyter_text>Check job status<jupyter_code>!qstat<jupyter_output><empty_output><jupyter_text>Check Result Check Result in Log FileCheck the latest created log file with prefix: **run_inc_ft_mnist_sample.sh.o**<jupyter_code>!tail -23 `ls -lAtr run_inc_ft_mnist_sample.sh.o* |  tail -1 | awk '{print $9}'`<jupyter_output><empty_output><jupyter_text>Check any existed log file, for example **run_inc_ft_mnist_sample.sh.o1842343**<jupyter_code>!tail -23 run_inc_ft_mnist_sample.sh.o1842343<jupyter_output><empty_output><jupyter_text>Check Result in PNG file<jupyter_code>from IPython.display import Image, display

listOfImageNames = ['fp32_int8_aboslute.png',
                    'fp32_int8_times.png']

for imageName in listOfImageNames:
    display(Image(filename=imageName))<jupyter_output><empty_output><jupyter_text>Run in Customer Server or CloudNote, it's recommended to use 2nd Generation Intel® Xeon® Scalable Processors or newer to get better performance improvement. Run in Jupyter Notebook<jupyter_code>run_inc_ft_mnist_sample.sh<jupyter_output><empty_output><jupyter_text>Check Result Check Result in Screen Output```...Compare the Performance of FP32 and INT8 ModelsModel            FP32                     INT8                    throughput(fps)  572.4982883964987        3218.52236638019        latency(ms)      2.8339174329018104       1.9863116497896156      accuracy(%)      0.9799                   0.9796                  Save to fp32_int8_aboslute.pngModel            FP32                     INT8                    throughput_times 1                        5.621889936815179       latency_times    1                        0.7009066766478504      accuracy_diff(%) 0                        -0.029999999999986926   Save to fp32_int8_times.pngPlease check the PNG files to see the performance!This demo is finished successfully!Thank you! End of output for job 1842253.v-qsvr-1.aidevcloud Date: Thu 27 Jan 2022 07:05:52 PM PST...``` Check Result in PNG fileThe demo creates figure files: fp32_int8_aboslute.png, fp32_int8_times.png to show performance bar. They could be used in report.<jupyter_code>from IPython.display import Image, display

listOfImageNames = ['fp32_int8_aboslute.png',
                    'fp32_int8_times.png']

for imageName in listOfImageNames:
    display(Image(filename=imageName))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0823045267489712, 'fraction_numerical': 0.050505050505050504, 'mean_word_length': 4.257620452310718, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '598432', 'license': 'MIT', 'n_tokens_mistral': 1935, 'n_tokens_neox': 1734, 'n_words': 492, 'path': 'inc_sample_for_tensorflow.ipynb', 'repo_name': 'NeoZhangJianyu/intel_neural_compressor_sample', 'size': '5346'}