<jupyter_start><jupyter_text>How to train Detectron2 with Custom COCO Datasets[Credits given to Diology](https://www.dlology.com/blog/how-to-train-detectron2-with-custom-coco-datasets/) This notebook will help you get started with  training a Detectron2  instance segmentation model with your custom COCO datasets.  Install detectron2The code below for installation is the most updated as of the time of writing. If any errors occur, please refer to the official Detectron2 Google Colab installation guide [here](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5).<jupyter_code>!pip install pyyaml==5.1 pycocotools>=2.0.1
import torch, torchvision
print(torch.__version__, torch.cuda.is_available())
!gcc --version
assert torch.__version__.startswith("1.6")
!pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.6/index.html<jupyter_output>Looking in links: https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.6/index.html
Collecting detectron2
[?25l  Downloading https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.6/detectron2-0.2.1%2Bcu101-cp36-cp36m-linux_x86_64.whl (6.6MB)
[K     |████████████████████████████████| 6.6MB 4.1MB/s 
[?25hRequirement already satisfied: tqdm>4.29.0 in /usr/local/lib/python3.6/dist-packages (from detectron2) (4.41.1)
Collecting Pillow>=7.1
[?25l  Downloading https://files.pythonhosted.org/packages/30/bf/92385b4262178ca22b34f82e0e09c2922eb351fe39f3cc7b8ba9ea555b41/Pillow-7.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.2MB)
[K     |████████████████████████████████| 2.2MB 6.6MB/s 
[?25hRequirement already satisfied: cloudpickle in /usr/local/lib/python3.6/dist-packages (from detectron2) (1.3.0)
Requirement already satisfied: pycocotools>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from detectron2) (2.0.1)
Requirement already satisfied: tabulate in /usr/local/lib/python3.6[...]<jupyter_text>Import Relevant Libraries<jupyter_code># You may need to restart your runtime prior to this, to let your installation take effect
# Some basic setup
# Setup detectron2 logger
import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

# import some common libraries
import matplotlib.pyplot as plt
import numpy as np
import cv2
from google.colab.patches import cv2_imshow

# import some common detectron2 utilities
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog
import torch, torchvision<jupyter_output><empty_output><jupyter_text>Prepare  a custom COCO datasetResizing the datasetDetectron2 seems to have a bug where the SizeMisMatch error is called onto random images, stating that its dimensions are flipped when its actually not.  To avoid this error altogether, resizing the image into a square is a good option. There are many free online softwares that can help resize images in bulk.Annotating the datasetAs far as I am aware, Detectron2 only supports **COCO format datasets**, hence we will have to prepare annotations in that format.You can annotate your dataset with the [VGG Annotator online](http://www.robots.ox.ac.uk/~vgg/software/via/via_demo.html). It is intuitive and easy to use. However, do take note of the following precautions when annotating your images:1. Draw only **polygons**, as rectangles are will not produce X-Y coordinates needed during training.2. If your objects are close to the edge of the image, please take precaution to draw your boundaries within the image, otherwise an error will be produced during training.3. Save both coco and json formats of your annotations just in case. When you want to load your annotations into VGG again, using the json format file will allow it to be more accurate, as coco formats do not preserve the names of the objects within your images.Afterwards, we will be making some edits to the coco file. When annotating your images, you would have realised that the categories that can be chosen are limited. In order to edit classes and categories of your images, scroll to the bottom of your coco file, and edit according to the example below:<jupyter_code>#example of how to edit last paragrah of coco file
"categories": [
    {"supercategory": "person","id": 1,"name": "person"},
    {"supercategory": "vehicle","id": 2,"name": "bicycle"},
    {"supercategory": "vehicle","id": 3,"name": "car"},
    {"supercategory": "vehicle","id": 4,"name": "motorcycle"},
    {"supercategory": "vehicle","id": 5,"name": "airplane"},
    ...
    {"supercategory": "indoor","id": 89,"name": "hair drier"},   
    {"supercategory": "indoor","id": 90,"name": "toothbrush"}
]<jupyter_output><empty_output><jupyter_text>I will be using the [doorway dataset](https://github.com/saltedpotato/custom-data-with-mcrnn)which  has 4 classes: door, lift, escalator, and stairs. You may look at the dataset for examples or clues.We'll train a segmentation model from an existing model pre-trained on the COCO dataset, available in detectron2's model zoo.**Note that the COCO dataset does not have the "door", "lift" , "escalator" or "stairs" categories.***Hint:** As it may take a long time to upload your hundreds of datasets into Google Drive or Google Colab, I have found that directly git cloning your files into Google Colab is the quickest and most effecient method of file transfer.<jupyter_code># download data from GitHub
!git clone --quiet https://github.com/saltedpotato/custom-data-with-mcrnn<jupyter_output><empty_output><jupyter_text>Register the doorway dataset to detectron2, following the [detectron2 custom dataset tutorial](https://detectron2.readthedocs.io/tutorials/datasets.html).<jupyter_code>from detectron2.data.datasets import register_coco_instances
register_coco_instances("door_train", {}, "/content/custom-data-with-mcrnn/dataset/train_detectron2/annotations_detectron2.json", "/content/custom-data-with-mcrnn/dataset/train_detectron2/images")
register_coco_instances("door_val", {}, "/content/custom-data-with-mcrnn/dataset/val/annotations_detectron2.json", "/content/custom-data-with-mcrnn/dataset/val/images")
register_coco_instances("door_test", {}, "/content/custom-data-with-mcrnn/dataset/predict_detectron2/annotations_detectron2.json", "/content/custom-data-with-mcrnn/dataset/predict_detectron2/images")<jupyter_output><empty_output><jupyter_text>To verify the data loading is correct, let's visualize the annotations of randomly selected samples in the training set:<jupyter_code>import random
from detectron2.utils.visualizer import Visualizer

#visualize training data
my_dataset_train_metadata = MetadataCatalog.get("door_train")
dataset_dicts = DatasetCatalog.get("door_train")

for d in random.sample(dataset_dicts, 1):
    img = cv2.imread(d["file_name"])
    visualizer = Visualizer(img[:, :, ::-1], metadata=my_dataset_train_metadata, scale=0.5)
    vis = visualizer.draw_dataset_dict(d)
    cv2_imshow(vis.get_image()[:, :, ::-1])<jupyter_output>[32m[08/12 10:00:27 d2.data.datasets.coco]: [0mLoaded 344 images in COCO format from /content/custom-data-with-mcrnn/dataset/train_detectron2/annotations_detectron2.json<jupyter_text>Configure & Train the modelBefore we start training the model, there are a few configurations that must be done. Firstly, edit the *MODEL.ROI_HEADS.NUM_CLASSES* variable to the same number of classes that the dataset has. The variables that can be changed can be found in the *configs folder* in the Detectron2 folder that you have cloned earlier.  The config files houses all the default configurations for your model, and making changes in the *directly in the cell below will overwrite the default configuration.The configurations of your model depends greatly on the kind of data you would like to train your model on. Hence, I am unable to give you a definite answer on what is the "best" configuration for your model. Lastly, we will fine-tune a coco-pretrained R50-FPN Mask R-CNN model on the dataset. It takes ~30 minutes to 1.5 hours to train 2000+ iterations on Colab's GPU.<jupyter_code>#We are importing our own Trainer Module here to use the COCO validation evaluation during training. Otherwise no validation eval occurs.
from detectron2 import model_zoo
from detectron2.engine import DefaultTrainer
from detectron2.evaluation import COCOEvaluator

class CocoTrainer(DefaultTrainer):

  @classmethod
  def build_evaluator(cls, cfg, dataset_name, output_folder=None):

    if output_folder is None:
        os.makedirs("coco_eval", exist_ok=True)
        output_folder = "coco_eval"

    return COCOEvaluator(dataset_name, cfg, False, output_folder)


from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
import os

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("door_train",)
cfg.DATASETS.VAL = ("door_val",)
cfg.DATASETS.TEST = ("door_test",)  
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")


cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4 # 4 classes (door, lift, stairs, escalator)
cfg.SOLVER.MAX_ITER = 1800 #adjust up if val mAP is still rising, adjust down if overfit


cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.001
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128

cfg.TEST.EVAL_PERIOD = 500

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = CocoTrainer(cfg)
trainer.resume_or_load(resume=True) #continue training on top of coco checkpoint 
#trainer.train()<jupyter_output>[32m[08/04 11:08:51 d2.engine.defaults]: [0mModel:
GeneralizedRCNN(
  (backbone): FPN(
    (fpn_lateral2): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1))
    (fpn_output2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (fpn_lateral3): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
    (fpn_output3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (fpn_lateral4): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1))
    (fpn_output4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (fpn_lateral5): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1))
    (fpn_output5): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (top_block): LastLevelMaxPool()
    (bottom_up): ResNet(
      (stem): BasicStem(
        (conv1): Conv2d(
          3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
          (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
        )
      )
 [...]<jupyter_text>Save the Model<jupyter_code>from detectron2.modeling import build_model
import torch
model = build_model(cfg)
#torch.save(model.state_dict(), '/content/output/mrcnn.pth')<jupyter_output><empty_output><jupyter_text>Load the ModelTo use the model for inference,<jupyter_code>model.load_state_dict(torch.load( '/content/output/mrcnn.pth', map_location='cpu'))<jupyter_output><empty_output><jupyter_text>Evaluating the ModelNow, we perform inference with the trained model on the doorway dataset. First, let's create a predictor using the model we just trained.  Average Precision (AP) Values on Test DatasetThese values are important in showing how well the model is doing overall, and for each individual classes. This will be crucial in spotting any overfitting tendencies within the model, and how well it fairs with different IoU levels. You can read more about AP values and intersections over unions (IoU) [here](https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173). Usually, a score of around 80% for AP50 would be good enough for me. To improve on your accuracy, consider tweaking the configurations, or adding new data to your dataset.<jupyter_code>#test evaluation
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader
from detectron2.evaluation import COCOEvaluator, inference_on_dataset

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.85
predictor = DefaultPredictor(cfg)
evaluator = COCOEvaluator("door_test", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "door_test")
inference_on_dataset(trainer.model, val_loader, evaluator)<jupyter_output>[32m[08/04 11:09:42 d2.data.datasets.coco]: [0mLoaded 27 images in COCO format from /content/custom-data-with-mcrnn/dataset/predict_detectron2/annotations_detectron2.json
[32m[08/04 11:09:42 d2.data.build]: [0mDistribution of instances among all 4 categories:
[36m|  category  | #instances   |  category  | #instances   |  category  | #instances   |
|:----------:|:-------------|:----------:|:-------------|:----------:|:-------------|
| escalator  | 13           |   stairs   | 9            |    door    | 9            |
|    lift    | 11           |            |              |            |              |
|   total    | 42           |            |              |            |              |[0m
[32m[08/04 11:09:42 d2.data.common]: [0mSerializing 27 elements to byte tensors and concatenating them all ...
[32m[08/04 11:09:42 d2.data.common]: [0mSerialized dataset takes 0.01 MiB
[32m[08/04 11:09:42 d2.data.dataset_mapper]: [0mAugmentations used in training: [ResizeShortestEdge(short_[...]<jupyter_text>Visual Evaluation on Test DatasetHere, we randomly select several samples to visualize the prediction results.<jupyter_code>from detectron2.utils.visualizer import ColorMode

#switch to test dataset
my_dataset_test_metadata = MetadataCatalog.get("door_test")
dataset_dicts = DatasetCatalog.get("door_test")

for d in random.sample(dataset_dicts, 3):    
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1],
                   metadata=my_dataset_test_metadata, 
                   scale=0.8, 
                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels
    )
    v = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    cv2_imshow(v.get_image()[:, :, ::-1])
from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive<jupyter_text>Benchmark inference speed<jupyter_code>import time
times = []
for i in range(20):
    start_time = time.time()
    outputs = predictor(im)
    delta = time.time() - start_time
    times.append(delta)
mean_delta = np.array(times).mean()
fps = 1 / mean_delta
print("Average(sec):{:.2f},fps:{:.2f}".format(mean_delta, fps))<jupyter_output>Average(sec):0.39,fps:2.59
{'fraction_non_alphanumeric': 0.11666336175556878, 'fraction_numerical': 0.03899795095511931, 'mean_word_length': 5.025487853444843, 'pattern_counts': {'":': 22, '<': 43, '<?xml version=': 0, '>': 47, 'https://': 11, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '273091', 'license': 'MIT', 'n_tokens_mistral': 5328, 'n_tokens_neox': 4929, 'n_words': 1357, 'path': 'ipynb guides/Detectron2_MRCNN_custom_dataset_coco.ipynb', 'repo_name': 'saltedpotato/custom-data-with-mcrnn', 'size': '15129'}