<jupyter_start><jupyter_text>**Object Detection with Detectron 2 using a Custom Dataset** **Detectron2** is a research platform and a production library for object detection, built by **Facebook AI Research (FAIR)**.It is completely written on PyTorch and is flexible and extensible, and able to provide fast training on single or multiple GPU servers. Also, it includes high-quality implementations of state-of-the-art object detection algorithms, including **DensePose**, **panoptic feature pyramid network**s, and numerous variants of the pioneering **Mask R-CNN**model family also developed by FAIR. Its extensible design makes it easy to implement cutting-edge research projects without having to fork the entire codebase.**In this notebook, we are going to deal with identifying masked faces using a fine-tuned and saved instance of the Faster RCNN model from the Detectron 2's model zoo.**We identify faces **with_mask**, **without_mask** and with **mask_weared_incorrect**(ly).  Import a few necessary packages.<jupyter_code># Some basic setup:
# Setup detectron2 logger
import detectron2
from detectron2.utils.logger import setup_logger

setup_logger()

# import some common libraries
import os, sys, json, cv2, random, numpy as np
import matplotlib.pyplot as plt

# from google.colab.patches import cv2_imshow

# import some common detectron2 utilities

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog
from detectron2.structures import BoxMode

import pandas as pd
import time
import datetime
import re

import torch, torchvision
# check pytorch installation:
print(
    f"torch version: {torch.__version__}\n\ntorch cuda is available: {torch.cuda.is_available()}"
)
assert torch.__version__.startswith(
    "1.8"
)  # please manually install torch 1.8 if Colab changes its default version
torch.cuda.empty_cache()
print(torch.cuda.memory_summary())<jupyter_output>|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
| Active memory         |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |[...]<jupyter_text>Define global variables<jupyter_code>print(os.getcwd())

base_path = os.getcwd()

masked_face_detector = os.path.join(base_path, "ai", "model", "model_final.pth")
conf_level = 0.5

output_folder = os.path.join(base_path, "data", "output_files")

metadata_folder = os.path.join(base_path, "data", "metadata")
video_folder = os.path.join(base_path, "data", "videos")

broadcast_dict = {}
# PROJECT_PATH = '../'
# DATA_PATH = os.path.join(PROJECT_PATH, 'data')
# MODELS_PATH = os.path.join(PROJECT_PATH, 'results')
# USE_MODEL = 1
# MODEL = None

# if USE_MODEL == 1:
#     MODEL = "MAX-ITER3000_LR0.05_GAMMA0.1-FASTRCNN-ONLY-2-CLASSES"
# elif USE_MODEL == 2:
#      MODEL = "MAX-ITER5000_LR0.05_GAMMA0.2-NMS-0.3-FASTRCNN-ONLY-2-CLASSES"
# elif USE_MODEL == 3:
#      MODEL = "MAX-ITER7000_LR0.05_GAMMA0.2-NMS-0.3-FASTRCNN-ONLY-2-CLASSES"
# elif USE_MODEL == 4:
#      MODEL = None<jupyter_output><empty_output><jupyter_text>**Preparing and registering the Dataset** There are different types for the format of bbox. It must be a member of structures.BoxMode.There are 5 such formats. But, currently it supports: **BoxMode.XYXY_ABS, BoxMode.XYWH_ABS**. We use the second format and mention the same in our dataset. After that, we need to register our datset.<jupyter_code># if your dataset is in COCO format, this cell can be replaced by the following three lines:
# from detectron2.data.datasets import register_coco_instances
# register_coco_instances("my_dataset_train", {}, "json_annotation_train.json", "path/to/image/dir")
# register_coco_instances("my_dataset_val", {}, "json_annotation_val.json", "path/to/image/dir")


def get_masked_face_dicts(coco_format_json_file, img_dir):

    with open(coco_format_json_file) as f:
        dataset_dicts = json.load(f)

    for d in dataset_dicts:
        d["file_name"] = os.path.join(img_dir, d["file_name"])

        for anno in d["annotations"]:
            anno["bbox_mode"] = BoxMode.XYWH_ABS
            anno["category_id"] = int(anno["category_id"])

    return dataset_dicts


for d in ["train", "val", "test"]:
    DatasetCatalog.register(
        "masked_face_" + d,
        lambda d=d: get_masked_face_dicts(
            os.path.join(DATA_PATH, d, d + "-with-only-2-classes.json"),
            os.path.join(DATA_PATH, d),
        ),
    )
    MetadataCatalog.get("masked_face_" + d).set(
        thing_classes=["with_mask", "without_mask"]
    )
masked_face_metadata = MetadataCatalog.get("masked_face_train")<jupyter_output><empty_output><jupyter_text>**Inference from new images using the trained model** An output folder gets saved in the local storage in which the final weights are stored. You can save this folder for inferencing from this model in future.Set the score threshold value for reducing the redundant boxes on the prediction results.<jupyter_code>%%time
from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg

cfg = get_cfg()
cfg.merge_from_file(
    model_zoo.get_config_file("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml")
)
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
# cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = masked_face_detector
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2  # No. of classes = [with_mask, without_mask]

#################################################################################################################################
# add config params
cfg.MODEL.PIXEL_STD = [
    57.375,
    57.120,
    58.395,
]  # Otherwise, you can use [57.375, 57.120, 58.395] (ImageNet std)

cfg.MODEL.ROI_HEADS.IOU_THRESHOLDS = [
    0.5
]  # Overlap threshold for an RoI to be considered foreground (if >= IOU_THRESHOLD), Overlap threshold for an RoI to be considered background (if < IOU_THRESHOLD)
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512  # 128
# cfg.MODEL.ROI_HEADS.POSITIVE_FRACTION = 0.1 # 0.25 Target fraction of RoI minibatch that is labeled foreground (i.e. class > 0)
cfg.MODEL.ROI_BOX_HEAD.NORM = "GN"  # Normalization method for the convolution layers
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.3
# cfg.MODEL.ROI_BOX_HEAD.SMOOTH_L1_BETA = 0.5
cfg.MODEL.ROI_BOX_HEAD.NORM = "GN"

cfg.MODEL.RPN.NMS_THRESH = 0.7  # 0.3
cfg.MODEL.FPN.NORM = "GN"
cfg.MODEL.RPN.IOU_THRESHOLDS = [0.3, 0.7]
cfg.MODEL.RPN.BATCH_SIZE_PER_IMAGE = 512  # 256

##################################################################################################################################

predictor = DefaultPredictor(cfg)<jupyter_output>Wall time: 3.6 s<jupyter_text>**Inference from videos using the trained model**<jupyter_code>from detectron2.utils.visualizer import ColorMode
import glob
import cv2

# from google.colab.patches import cv2_imshow

# cap = cv2.VideoCapture(os.path.join(DATA_PATH, 'images-from-youtube-videos', 'video_5', 'Paris_camera5.mp4'))
cap = cv2.VideoCapture(os.path.join(video_folder, "Paris_camera5.mp4"))
while cap.isOpened():
    ret, frame = cap.read()
    # if frame is read correctly ret is True
    if not ret:
        break

    # frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    frame = cv2.resize(frame, (300, 200))  # cv2.resize(frame, (1800, 1540))
    outputs = predictor(frame)
    v = Visualizer(
        frame[:, :, ::-1],
        metadata=masked_face_metadata,
        scale=1.0,  # 0.8
        instance_mode=ColorMode.IMAGE,
    )
    out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    # cv2.namedWindow( "frame", CV_WINDOW_AUTOSIZE);
    # imshow("Display frame", image);
    cv2.imshow("frame", out.get_image()[:, :, ::-1])
    # plt.figure(figsize = (150, 15))
    # plt.imshow(out.get_image()[:, :, ::-1])
    # plt.show()

    if cv2.waitKey(1) == ord("q"):
        break

# cap.release()
# cv2.destroyAllWindows()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1731005930401701, 'fraction_numerical': 0.0237216068031778, 'mean_word_length': 3.516422435573522, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '611487', 'license': 'Apache-2.0', 'n_tokens_mistral': 3044, 'n_tokens_neox': 2878, 'n_words': 755, 'path': 'notebooks/object-detection/main_demo.ipynb', 'repo_name': 'joohiang/notebooks', 'size': '8937'}