<jupyter_start><jupyter_text>Object Detection with Convolution Neural Network (CNN) based on TensorflowApplication: Mask Detection Estimated time needed: **30** minutes  Objectives You will learn how to train a deep neural network for object detection in images using transfer learning with a Mobilenet model.You will detect whether a person is wearing a mask or not. All you need to do is to annotate the provided image dataset and train the model using IBM CV Studio. In practice, very few people train an entire Convolutional Network from scratch (with random initialization) because it needs high computational power such as GPU and TPU and it is relatively rare to have a dataset of sufficient size. Instead, it is common to train a Convolutional Network on a very large dataset in the lab, then use this network to train your model. That is what we will do in this lab.  Import Libraries and Define Auxiliary Functions Import libraries for OS and Cloud:<jupyter_code>%%capture
!pip install --no-deps lvis
!pip install tf_slim
!pip install --no-deps tensorflowjs==1.4.0
!pip install tensorflow==1.15.2
!pip install tensorflow_hub

from IPython.display import display, Javascript, Image
import re
import zipfile
from base64 import b64decode
import sys
import os
import json

import io
import random

import tarfile
from datetime import datetime
from zipfile import ZipFile
import six.moves.urllib as urllib
import PIL.Image
from PIL import Image<jupyter_output><empty_output><jupyter_text>Import libraries for Tensorflow:<jupyter_code>%%capture
import tensorflow as tf
from object_detection.utils import dataset_util, label_map_util, config_util
from object_detection.utils.label_map_util import get_label_map_dict
from skillsnetwork import cvstudio<jupyter_output><empty_output><jupyter_text>Download the files needed to build the model:<jupyter_code>%%capture
import os
if os.path.exists("content-latest.zip"):
    pass
else:
    !wget https://s3.us.cloud-object-storage.appdomain.cloud/cf-courses-data/CognitiveClass/CV0101/content/data/content-latest.zip
    !wget https://s3.us.cloud-object-storage.appdomain.cloud/cf-courses-data/CognitiveClass/CV0101/content/data/tfrecord.py
    
with zipfile.ZipFile('content-latest.zip', 'r') as zip_ref:
    zip_ref.extractall('')<jupyter_output><empty_output><jupyter_text>Set up CV Studio Client and download the images:<jupyter_code># Initialize the CV Studio Client
cvstudioClient = cvstudio.CVStudio()

# Download All Images
cvstudioClient.downloadAll()
# Get the annotations from CV Studio
annotations = cvstudioClient.get_annotations()
labels = annotations['labels']<jupyter_output><empty_output><jupyter_text>Set up the folders paths to the output:<jupyter_code>CHECKPOINT_PATH = os.path.join(os.getcwd(),'content/checkpoint')
OUTPUT_PATH = os.path.join(os.getcwd(),'content/output')
EXPORTED_PATH = os.path.join(os.getcwd(),'content/exported')
DATA_PATH = os.path.join(os.getcwd(),'content/data')
CONFIG_PATH = os.path.join(os.getcwd(),'content/config')
LABEL_MAP_PATH = os.path.join(DATA_PATH, 'label_map.pbtxt')
TRAIN_RECORD_PATH = os.path.join(DATA_PATH, 'train.record')
VAL_RECORD_PATH = os.path.join(DATA_PATH, 'val.record')<jupyter_output><empty_output><jupyter_text>Image Pre-Processing Give an id number for each annotated image as a label map starting from 1:<jupyter_code>os.makedirs(DATA_PATH, exist_ok=True)
with open(LABEL_MAP_PATH, 'w') as f:
    for idx, label in enumerate(labels):
        f.write('item {\n')
        f.write("\tname: '{}'\n".format(label))
        f.write('\tid: {}\n'.format(idx + 1))
        f.write('}\n')<jupyter_output><empty_output><jupyter_text>Generating TFRecord is Tensorflow’s binary storage format. Using a binary file format for storage of your data can have a significant impact on the performance of your import pipeline  as a consequence on the training time of your model. Go to this link [https://www.tensorflow.org/tutorials/load_data/tfrecord](https://www.tensorflow.org/tutorials/load_data/tfrecord?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-cvstudio-2021-01-01) to learn more about it. Then, the images will be split as 70% for the training dataset train_examples and 30% for validation dataset val_examples.<jupyter_code>from tfrecord import create_tf_record, displaydetectedobject
image_files = [image for image in annotations["annotations"].keys()]

label_map = label_map_util.get_label_map_dict(LABEL_MAP_PATH)

random.seed(42)
random.shuffle(image_files)
num_train = int(0.7 * len(image_files))
train_examples = image_files[:num_train]
val_examples = image_files[num_train:]

create_tf_record(train_examples, annotations["annotations"], label_map, os.path.join(os.getcwd(),'images'), TRAIN_RECORD_PATH)
create_tf_record(val_examples, annotations["annotations"], label_map, os.path.join(os.getcwd(),'images'), VAL_RECORD_PATH)<jupyter_output><empty_output><jupyter_text>Selecting the ModelYou will upload your object detection model configuration which is MobileNet V1 from tensorflow.org. There are also many other object detection models available. If you are interested go to this link () and you will find many other models. Training a model from scratch can take long hours and tons of data. So, we helped you in reducing that effort by training the MobileNet model with a checkpoint.The model that will be trained is the SSD MobileNet architecture. SSD MobileNet models have very small file sizes and can execute very quickly, compromising little accuracy, which makes it perfect for running in the browser.<jupyter_code>MODEL_TYPE = 'ssd_mobilenet_v1_quantized_300x300_coco14_sync_2018_07_18'
CONFIG_TYPE = 'ssd_mobilenet_v1_quantized_300x300_coco14_sync'
download_base = 'http://download.tensorflow.org/models/object_detection/'
model = MODEL_TYPE + '.tar.gz'
tmp = '/resources/checkpoint.tar.gz'

if not (os.path.exists(CHECKPOINT_PATH)):
    # Download the checkpoint
    opener = urllib.request.URLopener()
    opener.retrieve(download_base + model, tmp)

    # Extract all the `model.ckpt` files.
    with tarfile.open(tmp) as tar:
        for member in tar.getmembers():
            member.name = os.path.basename(member.name)
            if 'model.ckpt' in member.name:
                tar.extract(member, path=CHECKPOINT_PATH)
            if 'pipeline.config' in member.name:
                tar.extract(member, path=CONFIG_PATH)

    os.remove(tmp)<jupyter_output><empty_output><jupyter_text>Building the Model Training Pipeline This is the the last stage before we start training the model. We need to inject our pipeline with the label map and TFRecord that we created previosuly in this notebook. As mentioned earlier, you won't start training the model from scratch, so we will use a model checkpoint. We set the batch size to be 6. This notebook won't have the sufficient memory power to handle a batch size higher than 6.<jupyter_code>pipeline_skeleton = 'content/models/research/object_detection/samples/configs/' + CONFIG_TYPE + '.config'
configs = config_util.get_configs_from_pipeline_file(pipeline_skeleton)

label_map = label_map_util.get_label_map_dict(LABEL_MAP_PATH)
num_classes = len(label_map.keys())
meta_arch = configs["model"].WhichOneof("model")

override_dict = {
  'model.{}.num_classes'.format(meta_arch): num_classes,
  'train_config.batch_size': 6,
  'train_input_path': TRAIN_RECORD_PATH,
  'eval_input_path': VAL_RECORD_PATH,
  'train_config.fine_tune_checkpoint': os.path.join(CHECKPOINT_PATH, 'model.ckpt'),
  'label_map_path': LABEL_MAP_PATH
}

configs = config_util.merge_external_params_with_configs(configs, kwargs_dict=override_dict)
pipeline_config = config_util.create_pipeline_proto_from_configs(configs)
config_util.save_pipeline_config(pipeline_config, DATA_PATH)<jupyter_output><empty_output><jupyter_text>Start TrainingWe will start the training run by calling the `model_main` script. Then, we will be passing the configuration shown in the above cell as `pipepline.config`. In order to build the model, we have to set different PATH to train MobileNet.<jupyter_code>paths = [
    f'home/jupyterlab/conda/envs/python/lib/python3.6',
    f'content/models/research',
    f'content/models/research/slim'
]
os.environ['PYTHONPATH'] = ':'.join(paths)<jupyter_output><empty_output><jupyter_text>The first step is to set epochs as 40. That will take between 3 to 7 minutes to complete the training. One epoch is when an ENTIRE dataset is passed forward and backward through the neural network only once.<jupyter_code>%%capture
epochs = 40
start_datetime = datetime.now()
!python -m object_detection.model_main \
    --pipeline_config_path=$DATA_PATH/pipeline.config \
    --num_train_steps=$epochs \
    --num_eval_steps=100

regex = re.compile(r"model\.ckpt-([0-9]+)\.index")
numbers = [int(regex.search(f).group(1)) for f in os.listdir(OUTPUT_PATH) if regex.search(f)]
TRAINED_CHECKPOINT_PREFIX = os.path.join(OUTPUT_PATH, 'model.ckpt-{}'.format(max(numbers)))

!python3 -m object_detection.export_inference_graph \
  --pipeline_config_path=$DATA_PATH/pipeline.config \
  --trained_checkpoint_prefix=$TRAINED_CHECKPOINT_PREFIX \
  --output_directory=$EXPORTED_PATH
end_datetime = datetime.now()<jupyter_output><empty_output><jupyter_text>Let's Test the Model and See How Cool It Is! We are going to use the image saved in this notebook. You can also add your own image by providing a URL.<jupyter_code>from PIL import Image

# Here you can specify your own image 
URL = 'https://cdn.cliqueinc.com/posts/289533/kamala-harris-face-mask-289533-1602269219518-square.700x0c.jpg' 

with urllib.request.urlopen(URL) as url:
    with open('test.jpg', 'wb') as f:
        f.write(url.read())
image = Image.open('test.jpg')

image
%matplotlib inline
import numpy as np
n,img,accuracy=displaydetectedobject(image)<jupyter_output><empty_output><jupyter_text>Let's Report the Results<jupyter_code>parameters = {
    'epochs': epochs
}

result = cvstudioClient.report(started=start_datetime, completed=end_datetime, parameters=parameters, accuracy= round(float(accuracy),2)*100)

if result.ok:
    print('Congratulations your results have been reported back to CV Studio!')<jupyter_output><empty_output><jupyter_text>Save the Model for Deployment<jupyter_code>%%capture
!tensorflowjs_converter \
  --input_format=tf_frozen_model \
  --output_format=tfjs_graph_model \
  --output_node_names='Postprocessor/ExpandDims_1,Postprocessor/Slice' \
  --quantization_bytes=1 \
  --skip_op_check \
  $EXPORTED_PATH/frozen_inference_graph.pb \
  .
import json

from object_detection.utils.label_map_util import get_label_map_dict

label_map = get_label_map_dict(LABEL_MAP_PATH)
label_array = [k for k in sorted(label_map, key=label_map.get)]

with open(os.path.join('', 'labels.json'), 'w') as f:
    json.dump(label_array, f)

#!cd model_web 
with ZipFile('model_web.zip','w') as zip:
    zip.write('group1-shard1of2.bin')
    zip.write('group1-shard2of2.bin')
    zip.write('model.json')
    zip.write('labels.json')
    
# Download All Images
cvstudioClient.uploadModel('model_web.zip', {'epochs': epochs })<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09729439168607776, 'fraction_numerical': 0.012632144776921699, 'mean_word_length': 5.6724447101016136, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '588867', 'license': 'MIT', 'n_tokens_mistral': 3680, 'n_tokens_neox': 3531, 'n_words': 959, 'path': '3. Introduction to Computer Vision and Image Processing/4. Object Detection/train-objectdetection-cnn-tensorflow.ipynb', 'repo_name': 'aqafridi/AI-Engineering-Specialization', 'size': '11162'}