<jupyter_start><jupyter_text>Training Facemask Detection CNN for DPU compilationIn this notebook we show how to train a Convolutional Neural Network (CNN) for deployment on the DPU (ZCU104).Steps:* Loading and pre-processing the imagenet dataset* Training a CNN with Keras and Tensorflow* Freezing the trained model* Quantizing and evaluating the quantized model* Compiling for DPU using the Vitis AI compiler**Note**:* This notebook should be run on a proper X86 machineIn my case:    * Distributor ID: Ubuntu    * Description:	  Ubuntu 18.04.3    * Release:        18.04.3* Please make sure the following has been done in terminal before you start this notebook:conda activate vitis-ai-tensorflowyes | pip install matplotlib keras==2.2.5<jupyter_code>import os
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import random
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
import keras
from keras.layers import Dense, Conv2D, InputLayer, Flatten, MaxPool2D
from keras.preprocessing.image import ImageDataGenerator,img_to_array,load_img
from keras.applications.mobilenet_v2 import preprocess_input
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from keras.utils import to_categorical<jupyter_output>Using TensorFlow backend.<jupyter_text>Load datasetThe Facemask dataset comes with 1916 images with Facemask and 1930 without Facemask. We have to preprocess the images.<jupyter_code>print("The number of images with facemask labelled 'yes':",len(os.listdir('facemask-dataset/yes')))
print("The number of images without facemask labelled 'no':",len(os.listdir('facemask-dataset/no')))<jupyter_output>The number of images with facemask labelled 'yes': 1916
The number of images without facemask labelled 'no': 1930<jupyter_text>Preprocessing DatasetWe are going to preprocess the images, convert them to grayscale, transform their size to 64x64, divide the images by 255 to normalize them, and convert them to numpy float32 arrays.<jupyter_code>import cv2
import os

data = []
labels = []

mylist = os.listdir("facemask-dataset") # Set Dataset Folder

for x in mylist:
    mylist2 = os.listdir("facemask-dataset/"+str(x))
    label = str(x)
    for y in mylist2:
        # extract the class label from the folder name

        # load the input image (64x64) and preprocess it
        image = load_img("facemask-dataset/"+str(x)+"/"+str(y),color_mode="grayscale", target_size=(64, 64))
        image = img_to_array(image)
        image = preprocess_input(image)

        # update the data and labels lists, respectively
        data.append(image)
        labels.append(label)

# convert the data and labels to NumPy arrays
data = np.array(data, dtype="float32")
labels = np.array(labels)

# perform one-hot encoding on the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
labels = to_categorical(labels)

# partition the data into training and testing splits using 80% of
# the data for training and the remaining 20% for testing

(x_train, x_test,y_train, y_test) = train_test_split(data, labels,
                                                        test_size=0.20, stratify=labels, random_state=42)

print('Training data: {}, {}'.format(x_train.shape, y_train.shape))
print('Test data: {}, {}'.format(x_test.shape, y_test.shape))


#The strcuture that we must obtain are the following:
#     For training and test datasets: (#data, 64, 64, 1)
#     For labels: (#labels, 2)
   
#Example:
#Training data: (3091, 64, 64, 1), (3091, 2)
#Test data: (773, 64, 64, 1), (773, 2)<jupyter_output>/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/PIL/Image.py:952: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  "Palette images with Transparency expressed in bytes should be "<jupyter_text>Let's check if the data set and labels are correct with a sample of 6 images<jupyter_code>SampleSize=6
fig, axs = plt.subplots(1, SampleSize, figsize=(10, 10))
plt.tight_layout()

for i in range(SampleSize):
    axs[i].imshow(x_train[i].reshape(64, 64), 'gray')
    if(y_train[i][0]==0):
        tempLabel = "Facemask ON"
    else:
        tempLabel = "Facemask OFF"
    axs[i].set_title('{}'.format(tempLabel))<jupyter_output><empty_output><jupyter_text>Model CreationCreate a sequential model by passing a list of layers. Because the mask detection it is a binary classification problem, we can use 2 Convolutional layers to solve the problem. NOTE: Do not rename the input layer nor the output layer.<jupyter_code>model = keras.models.Sequential([
    InputLayer(input_shape=(64, 64, 1), name='input_data'),
    Conv2D(64, (3,3), activation='relu'),
    MaxPool2D(pool_size=(2,2)),
    Conv2D(128, (3,3), activation='relu'),
    MaxPool2D(pool_size=(2,2)),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(2, activation='softmax', name='output_logits')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
model.summary()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 62, 62, 64)        640       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 31, 31, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 29, 29, 128)       73856     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 14, 14, 128)       0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 25088)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                1605696   
______________________________________________________[...]<jupyter_text>Configure the model for training: choose desired optimizer, loss functionand metrics to observe over the training period.<jupyter_code>model.compile(optimizer='adam',
              loss="binary_crossentropy", # For binary classifiers always use binary_crossentropy
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Early stoping to avoid Overfitting<jupyter_code>from keras.callbacks import EarlyStopping
es = EarlyStopping(monitor='val_loss', mode='auto', verbose=1)<jupyter_output><empty_output><jupyter_text>Now we can train the model.<jupyter_code>history = model.fit(x_train,y_train,validation_data=(x_test, y_test),epochs=30, verbose=1 ,callbacks=[es])<jupyter_output>Train on 3076 samples, validate on 770 samples
Epoch 1/30
3076/3076 [==============================] - 36s 12ms/step - loss: 0.4285 - acc: 0.7845 - val_loss: 0.3658 - val_acc: 0.8273
Epoch 2/30
3076/3076 [==============================] - 36s 12ms/step - loss: 0.2196 - acc: 0.9109 - val_loss: 0.2572 - val_acc: 0.8948
Epoch 3/30
3076/3076 [==============================] - 35s 11ms/step - loss: 0.1436 - acc: 0.9470 - val_loss: 0.1960 - val_acc: 0.9169
Epoch 4/30
3076/3076 [==============================] - 34s 11ms/step - loss: 0.0936 - acc: 0.9652 - val_loss: 0.2057 - val_acc: 0.9234
Epoch 00004: early stopping<jupyter_text>We can inspect the training results by plotting the collected data in the`history` object.<jupyter_code>fig, axs = plt.subplots(1, 2, figsize=(12, 4))

axs[0].plot(history.history['loss'])
axs[0].set_title('Training loss')
axs[0].set(xlabel='Epochs', ylabel='Loss')

axs[1].plot(history.history['acc'])
axs[1].set_title('Training accuracy')
axs[1].set(xlabel='Epochs', ylabel='Accuracy')
plt.show()
loss, accuracy = model.evaluate(x_test,y_test)
print("Test loss: {}".format(loss))
print("Test accuracy: {}".format(accuracy))<jupyter_output>770/770 [==============================] - 2s 3ms/step
Test loss: 0.205728345954573
Test accuracy: 0.9233766233766234<jupyter_text>Evaluate the trained model on the test dataset.  Save checkpoint<jupyter_code>saver = tf.train.Saver()
tf_session = keras.backend.get_session()
input_graph_def = tf_session.graph.as_graph_def()
save_path = saver.save(tf_session, './checkpoint.ckpt')
tf.train.write_graph(input_graph_def,
                     './', 'face_binary.pb', as_text=False)<jupyter_output><empty_output><jupyter_text>As well as saving the checkpoint we also need to make a note of the input and output nodes of the graph for freezing and quantization. We made ourlives a bit easier by naming the input and output layers, which resultsin our input and output nodes being named `input_data` and `output_logits/Softmax` respectively. You can check the node names in the list defined below. DONT RENAME NOTHING<jupyter_code>nodes_names = [node.name for node in 
               tf.get_default_graph().as_graph_def().node]

print(nodes_names)<jupyter_output>['input_data', 'input_data/aquant', 'output_logits/bias', 'output_logits/bias/wquant', 'output_logits/kernel', 'output_logits/kernel/wquant', 'dense_1/bias', 'dense_1/bias/wquant', 'dense_1/kernel', 'dense_1/kernel/wquant', 'flatten_1/stack/0', 'flatten_1/Const', 'flatten_1/strided_slice/stack_2', 'flatten_1/strided_slice/stack_1', 'flatten_1/strided_slice/stack', 'conv2d_2/bias', 'conv2d_2/bias/wquant', 'conv2d_2/kernel', 'conv2d_2/kernel/wquant', 'conv2d_1/bias', 'conv2d_1/bias/wquant', 'conv2d_1/kernel', 'conv2d_1/kernel/wquant', 'conv2d_1/convolution', 'conv2d_1/BiasAdd', 'conv2d_1/Relu', 'conv2d_1/Relu/aquant', 'max_pooling2d_1/MaxPool', 'max_pooling2d_1/MaxPool/aquant', 'conv2d_2/convolution', 'conv2d_2/BiasAdd', 'conv2d_2/Relu', 'conv2d_2/Relu/aquant', 'max_pooling2d_2/MaxPool', 'max_pooling2d_2/MaxPool/aquant', 'flatten_1/Shape', 'flatten_1/strided_slice', 'flatten_1/Prod', 'flatten_1/stack', 'flatten_1/Reshape', 'flatten_1/Reshape/aquant', 'dense_1/MatMul', 'dense_1/BiasAdd', [...]<jupyter_text>Freeze Tensorflow graphThe Vitis AI flow requires a frozen model for quantization. We can obtain a binaryprotobuf file of our frozen model by using the Tensorflow `freeze_graph` utility.<jupyter_code>!freeze_graph \
    --input_graph face_binary.pb \
    --input_checkpoint checkpoint.ckpt \
    --input_binary true \
    --output_graph frozen.pb \
    --output_node_names output_logits/Softmax<jupyter_output>WARNING:tensorflow:From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1122 01:06:11.015468 139705175672640 deprecation.py:323] From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-11-22 01:06:11.275759: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-11-22 01:06:11.288269: I tensorflow/co[...]<jupyter_text>QuantizationWe will save some of our training data as calibration data for`vai_q_tensorflow`, then use that along with the frozen graph to quantize our model.`vai_q_tensorflow inspect` can be used to confirm available input and outputnode names.<jupyter_code>!vai_q_tensorflow inspect --input_frozen_graph=frozen.pb
np.savez('./calib_data.npz', data = x_train[:1000])<jupyter_output><empty_output><jupyter_text>We will save a portion of our training data for quantization.Recommended number is around 100-1000 for images. (1000 in this case)<jupyter_code>%%writefile input_func.py
import numpy as np

data = np.load('calib_data.npz')['data']

batch_size=10 # Images per batch

def calib_input(iter):

    calib_data = data[iter*batch_size:(iter+1)*batch_size]

    return {'input_data': calib_data}
!vai_q_tensorflow quantize \
    --input_frozen_graph frozen.pb \
    --input_fn input_func.calib_input \
    --output_dir quantized \
    --input_nodes input_data \
    --output_nodes output_logits/Softmax \
    --input_shapes ?,64,64,1 \
    --calib_iter 100 # Images per Epoch<jupyter_output>INFO: Checking Float Graph...
INFO: Float Graph Check Done.
INFO: Calibrating for 100 iterations...
100% (100 of 100) |######################| Elapsed Time: 0:01:36 Time:  0:01:36
INFO: Calibration Done.
INFO: Generating Deploy Model...
INFO: Deploy Model Generated.
********************* Quantization Summary *********************      
INFO: Output:       
  quantize_eval_model: quantized/quantize_eval_model.pb       
  deploy_model: quantized/deploy_model.pb<jupyter_text>Evaluate quantized modelThe quantizer produces a special model called `quantize_eval_model.pb`, which we can use to load up like a regular Tensorfow binary graph and evaluate its performance.Because we already have a graph definition in our session, we need toreset the default graph so as to not interfere with the graph we are about to load from the frozen model.<jupyter_code>tf.reset_default_graph()<jupyter_output><empty_output><jupyter_text>In order to evaluate a quantized model we have to import `tensorflow.contrib.decent_q`, otherwise the model evaluation will error out. We will use standard Tensorflow 1, to set up the graph for evaluation.In the next cell, we will read in a frozen binary graph and add the accuracy metric.<jupyter_code>import tensorflow.contrib.decent_q

with tf.gfile.GFile('quantized/quantize_eval_model.pb', "rb") as f:
    graph = tf.GraphDef()
    graph.ParseFromString(f.read())

tf.import_graph_def(graph,name = '')

input_data = tf.get_default_graph().get_tensor_by_name('input_data'+':0')
labels = tf.placeholder(tf.int64, shape=[None,])
logits = tf.get_default_graph().get_tensor_by_name(
    'output_logits/Softmax'+':0')

nn_output = tf.argmax(logits, 1)
correct_prediction = tf.equal(nn_output, labels)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))<jupyter_output><empty_output><jupyter_text>CompilationNow that we are satisfied with our quantized model accuracy, we can compile it and move onto the target.This example targets the ZCU104. You can target a different architecture byspecifying its configuration json file with the `--arch` flag./opt/vitis_ai/compiler/arch/DPUCZDX8G/ZCU104/arch.json ZCU104<jupyter_code>!sudo chmod -R 777 *
!vai_c_tensorflow \
    --frozen_pb quantized/deploy_model.pb \
    --arch /opt/vitis_ai/compiler/arch/DPUCZDX8G/ZCU104/arch.json \
    --output_dir . \
    --net_name face_binary_classifier<jupyter_output>**************************************************
* VITIS_AI Compilation - Xilinx Inc.
**************************************************
[0;33m[VAI_C][Warning] layer [output_logits_Softmax] (type: Softmax) is not supported in DPU, deploy it in CPU instead.
[m
Kernel topology "face_binary_classifier_kernel_graph.jpg" for network "face_binary_classifier"
kernel list info for network "face_binary_classifier"
                               Kernel ID : Name
                                       0 : face_binary_classifier_0
                                       1 : face_binary_classifier_1

                             Kernel Name : face_binary_classifier_0
--------------------------------------------------------------------------------
                             Kernel Type : DPUKernel
                               Code Size : 0.01MB
                              Param Size : 1.60MB
                           Workload MACs : 131.65MOPS
                         IO Memory Space : 0.0[...]
{'fraction_non_alphanumeric': 0.1277851001066968, 'fraction_numerical': 0.04180003765769159, 'mean_word_length': 4.503972366148532, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '38164', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5443, 'n_tokens_neox': 4943, 'n_words': 1340, 'path': 'Setup Notebook and Dataset/train_facemask_model.ipynb', 'repo_name': 'altaga/Facemask-Detector-ZCU104', 'size': '15933'}