<jupyter_start><jupyter_text>Exploring Object Detection using RetinaNet in Torch Vision 0.8 and Pytorch LightningInstead of using higher level library such as IceVision, I am gonna try using what's in Torch Vision, see [example](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html).To force some learning, I shall attempt to port the example to [Pytorch-Lightning](https://github.com/PyTorchLightning/pytorch-lightning) as it removes a lot of boiler plate code and standardized Pytorch usage. It also makes advanced features like gradient accumulation and multi-GPU multi-node training simple.<jupyter_code>#export
import json, os, requests, sys, tarfile
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import numpy as np
import pickle
import random

from collections import defaultdict
from functools import reduce
from IPython.utils import io
from pathlib import Path
from PIL import Image
from PIL import ImageStat

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
from tqdm import tqdm
from typing import Hashable, List, Tuple, Union
#hide

IN_COLAB = 'google.colab' in sys.modules

if IN_COLAB:
    from google.colab import drive
    drive.mount('/content/drive/', force_remount=True)
    os.chdir('/content/drive/My Drive/ColabData')
    !wget -O 'mcbbox/subcoco_utils.py' https://raw.githubusercontent.com/bguan/mcbbox/master/mcbbox/subcoco_utils.py
    !wget -O 'mcbbox/subcoco_lightning_utils.py' https://raw.githubusercontent.com/bguan/mcbbox/master/mcbbox/subcoco_lightning_utils.py
    !pip install torch torchvision pytorch_lightning effdet gpumonitor albumentations==0.4.6

print(f'Current Directory is now {os.getcwd()}')
#export
import albumentations as A
import torch, torchvision
import pytorch_lightning as pl
import torch.nn.functional as F
import torch.multiprocessing
from torch import nn
from torch.nn.modules import module
from torch import optim
from torch.utils.data import DataLoader, random_split
from torchvision.models.detection import RetinaNet, retinanet_resnet50_fpn
from mcbbox.subcoco_utils import *
from mcbbox.subcoco_lightning_utils import *

torch.multiprocessing.set_sharing_strategy('file_system')
print(f"Python ver {sys.version}, torch {torch.__version__}, torchvision {torchvision.__version__}, pytorch_lightning {pl.__version__}")

if is_notebook():
    from nbdev.showdoc import *<jupyter_output>Python ver 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0], torch 1.7.0, torchvision 0.8.1, pytorch_lightning 1.0.4, Albumentation 0.5.0
Python ver 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0], torch 1.7.0, torchvision 0.8.1, pytorch_lightning 1.0.4<jupyter_text>Setup the RunCentralize test run vs prod time parameters<jupyter_code>#hide
test = True
backbone_name = 'retinanet_resnet50_fpn'
if test:
    img_sz, head_runs, full_runs, save_top, lr, patience, calc_metrics = 128, 1, 1, 1, 0.001, 1, True
    bs, acc, workers = 4, 8, 4 
    datadir, url, img_subdir = '/tmp', 'http://files.fast.ai/data/examples/coco_tiny.tgz', 'train'
else:
    img_sz, head_runs, full_runs, save_top, lr, patience, calc_metrics = 384, 10, 100, -1, 0.001, 10, True
    bs, acc, workers =  (16, 2, 8) if IN_COLAB else (6, 6, 4) 
    datadir, url, img_subdir = 'workspace', 'https://s3.amazonaws.com/fast-ai-coco/coco_sample.tgz', 'train_sample'
    
stem = Path(url).stem
img_dir=f'{datadir}/{stem}/{img_subdir}'<jupyter_output>/home/brian/.local/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)<jupyter_text>Download a Sample of COCO DataThe full COCO Dataset is huge (~50GB?). For my self education exploring object detection, with the intention of using pretrained model in transfer learning, it is not practical to deal with dataset this big as my first project.  Luckily, the kind folks at [FastAI](https://fast.ai) have prepared some convenient subsets, the medium size 3GB https://s3.amazonaws.com/fast-ai-coco/coco_sample.tgz seems like a good candidate.  The 800KB "http://files.fast.ai/data/examples/coco_tiny.tgz" on the other hand seems way too small, thus may not have enough data for adequate training.Let's download and inspect the annotation file that comes with the coco tiny dataset...<jupyter_code>#hide
train_json = fetch_subcoco(datadir=datadir, url=url, img_subdir=img_subdir)
train_json['categories'], train_json['images'][0], [a for a in train_json['annotations'] if a['image_id']==train_json['images'][0]['id'] ]<jupyter_output><empty_output><jupyter_text>Digest the Dataset for useful StatsDo some basic analysis of the data to get numbers like total images, boxes, and average box count per image... Print out number of boxes per category to see if the distribution is not too unbalanced...<jupyter_code>#hide
stats = load_stats(train_json, img_dir=img_dir, force_reload=False)
print(
    f"Categories {stats.num_cats}, Images {stats.num_imgs}, Boxes {stats.num_bboxs}, avg (w,h) {(stats.avg_width, stats.avg_height)}"
    f"avg cats/img {stats.avg_ncats_per_img:.1f}, avg boxs/img {stats.avg_nboxs_per_img:.1f}, avg boxs/cat {stats.avg_nboxs_per_cat:.1f}.")

print(f"Image means by channel {stats.chn_means}, std.dev by channel {stats.chn_stds}")
{ stats.cat2name[stats.lbl2cat[lid]]: len(ibs) for lid, ibs in stats.l2ibs.items()}<jupyter_output>Categories 6, Images 200, Boxes 688, avg (w,h) (128.0, 128.0)avg cats/img 7.0, avg boxs/img 3.4, avg boxs/cat 114.7.
Image means by channel [126.2909314  109.02263367 100.61073242], std.dev by channel [65.60744282 65.09696176 60.4631069 ]<jupyter_text>Look at ImagesLet's look at an image annotation.<jupyter_code>best_img_pos = None
max_boxs = 0
for img_pos in range(len(train_json['images'])):  
    img_id = train_json['images'][img_pos]['id']
    l2bs = stats.img2l2bs.get(img_id, {})
    n_boxs =  reduce((lambda x, y: x + y), [1 if len(bs) > 0 else 0 for bs in l2bs.values()], 0)
    if n_boxs >= max_boxs:
        best_img_pos = img_pos
        max_boxs = n_boxs

print(f'best_img_pos {best_img_pos}')
best_img_id = train_json['images'][best_img_pos]['id']
l2bs = stats.img2l2bs[best_img_id]
best_img_fname = stats.img2fname[best_img_id]
best_img_id, best_img_fname, l2bs<jupyter_output>best_img_pos 89<jupyter_text>Overlay Boxes and Labels from AnnotationLet's overlay bounding boxes and labels over the image to confirm our understanding of the data.Pick some random image to test drive the box overlay code...<jupyter_code>img = Image.open(Path(img_dir)/stats.img2fname[best_img_id])
print(f"img = {img}")
overlay_img_bbox(img, stats.img2l2bs[best_img_id], stats.lbl2name)<jupyter_output>img = <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=128x128 at 0x7FDA2CD53C70><jupyter_text>Packaging Model and Metric Into A Lightning Module<jupyter_code># export
class RetinaNetModule(AbstractDetectorLightningModule):
    def __init__(self, **kwargs):
        AbstractDetectorLightningModule.__init__(self, **kwargs)
    
    def create_model(self, backbone_name, num_classes=1, **kwargs): 
        model = retinanet_resnet50_fpn(pretrained=False, num_classes=num_classes+1, pretrained_backbone=True)
        
        # Hacked to avoid model builtin call to GeneralizedRCNNTransform.normalize() as done in augmentation
        def noop_normalize(image): return image 

        # Hacked to avoid model builtin call to GeneralizedRCNNTransform.resize() as done in augmentation
        def noop_resize(image, target): return image, target  

        # HACK!! IceVision does this too!
        model.transform.normalize = noop_normalize
        model.transform.resize = noop_resize
        
        return model
        
    def get_main_model(self): return self.model
        
    def get_head(self): return self.model.head
        
    def get_backbone(self): return self.model.backbone<jupyter_output><empty_output><jupyter_text>Neural Network Architecture Model On my Dell XPS15 GTX-1050 GPU w/ 4GB VRAM,  I can average 1 epoc every 120 min or so, at bs=2 acc=16, img 384x384, with unfrozen backbone.Using my GCP w/ V100 of 16GB VRAM, I can fit bs=12 for sample_coco, acc=16, img 384x384, with unfrozen backbone, 1 epoc every ~25 minutes.<jupyter_code>if torch.cuda.is_available():
    retnet_model, last_model_path = run_training(RetinaNetModule, backbone_name, stats, img_dir, 
        resume_ckpt_fname=f'{backbone_name}-{img_sz}-last.ckpt', 
        img_sz=img_sz, lr=lr, bs=bs, acc=acc, workers=workers, head_runs=head_runs, full_runs=full_runs,
        save_top=save_top, test=test, calc_metrics=calc_metrics, monitor='val_acc', mode='max', patience=patience
    )

    print(f"last_model_path saved = {last_model_path}")<jupyter_output>Training with image size 128, learning rate 0.001, patience = 1, for 1+1 epochs.
Loading previously saved model: models/retinanet_resnet50_fpn-128-last.ckpt...<jupyter_text>InferenceTime to see how well the model can perform.<jupyter_code>test_set = SubCocoDataset(img_dir, stats, img_ids=[best_img_id])
    
if torch.cuda.is_available():
    x, y = test_set[0] 
    pimg = torchvision.transforms.ToPILImage()(x)
    tgt_l2bs = defaultdict(empty_list)
    for l, b in zip(list(y['labels']), y['boxes']):
        tgt_l2bs[int(l)].append( (int(b[0]), int(b[1]), int(b[2]-b[0]), int(b[3]-b[1])) ) 

    overlay_img_bbox(pimg, tgt_l2bs, stats.lbl2name)

    retnet_model.eval()
    pred = retnet_model.forward([x])
    pred_l2bs = digest_pred(stats.lbl2name, pred[0], cutoff=0.5, img_sz=x.shape[1])
    print(f"pred labels to boxes: { {k:len(v) for k, v in pred_l2bs.items()} }")
    overlay_img_bbox(pimg, pred_l2bs, stats.lbl2name)<jupyter_output>Entering forward, training = False
Exiting forward, returning [{'boxes': tensor([], size=(0, 4)), 'scores': tensor([]), 'labels': tensor([], dtype=torch.int64)}]
pred labels to boxes: {}<jupyter_text>Compare with Original Image before transformations incl. Normalization and Resize.<jupyter_code>if torch.cuda.is_available():
    img_id = int(y['image_id'])
    img = Image.open(Path(img_dir)/stats.img2fname[img_id])
    overlay_img_bbox(img, stats.img2l2bs[img_id], stats.lbl2name)<jupyter_output><empty_output><jupyter_text>Reloading Saved Checkpoint using Pytorch Lightning<jupyter_code>#hide
if torch.cuda.is_available():
    pretrained_model = RetinaNetModule.load_from_checkpoint(last_model_path, backbone_name=backbone_name, bs=bs, steps_per_epoch=0, num_classes=len(stats.lbl2name), img_sz=img_sz)
    pretrained_model.freeze()
    x, y = test_set[0] 
    pred = pretrained_model([x])
    pred_l2bs = digest_pred(stats.lbl2name, pred[0], cutoff=0.1, img_sz=x.shape[1])
    print(f"pred labels to boxes: { {k:len(v) for k, v in pred_l2bs.items()} }")
    pimg = torchvision.transforms.ToPILImage()(x)
    overlay_img_bbox(pimg, pred_l2bs, stats.lbl2name)<jupyter_output>pred labels to boxes: {}<jupyter_text>Saving and Reloading using Underlying Pytorch<jupyter_code># export
def save_final(retnet_model, model_save_path):
    torch.save(retnet_model.model.state_dict(), model_save_path)
if torch.cuda.is_available():
    model_save_path = f'models/{backbone_name}-{img_sz}-final.pth'
    save_final(retnet_model, model_save_path)
    pretrained_model = RetinaNetModule(backbone_name=backbone_name, bs=bs, steps_per_epoch=0, num_classes=len(stats.lbl2name), img_sz=img_sz)
    pretrained_model.model.load_state_dict(torch.load(model_save_path))
    pretrained_model.freeze()
if torch.cuda.is_available():
    x, y = test_set[0] 
    pred = pretrained_model([x])
    pred_l2bs = digest_pred(stats.lbl2name, pred[0], cutoff=0.1, img_sz=x.shape[1])
    print(f"pred labels to boxes: { {k:len(v) for k, v in pred_l2bs.items()} }")
    pimg = torchvision.transforms.ToPILImage()(x)
    overlay_img_bbox(pimg, pred_l2bs, stats.lbl2name)<jupyter_output>pred labels to boxes: {}
{'fraction_non_alphanumeric': 0.1104559421974919, 'fraction_numerical': 0.029233452371065526, 'mean_word_length': 5.2751433038040645, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '468191', 'license': 'Apache-2.0', 'n_tokens_mistral': 4536, 'n_tokens_neox': 4282, 'n_words': 1059, 'path': '50_subcoco_retinanet_lightning.ipynb.ipynb', 'repo_name': 'bguan/plmcbbox', 'size': '12041'}