<jupyter_start><jupyter_text>DE⫶TR -- Extending Object Detection to Panoptic Segmentation> An Extension to Object Detection using Transformers to Predict Pixel wise mask of each class.- toc: true - badges: true- comments: true- categories: [Custom Dataset, Object Detection, Panoptic Segmentation, COCO ,DETR]- image: images/PanopticSegmentation/DETR.gif  ![SuperCool](../images/PanopticSegmentation/DETR.gif )  Creating Dataset class for Panoptic Segmentation<jupyter_code>#collapse-hide

class ConstructionPanoptic:
    def __init__(self, img_folder, ann_folder, ann_file, transforms=None, return_masks=True):
        with open(ann_file, 'r') as f:
            self.coco = json.load(f) # Readig the json file

        # sort 'images' field so that they are aligned with 'annotations'
        # i.e., in alphabetical order
        self.coco['images'] = sorted(self.coco['images'], key=lambda x: x['id'])
        self.coco['annotations'] = sorted(self.coco['annotations'], key=lambda x: x['image_id'])
        # sanity check, image names in images is same as in annotations masks.
        if "annotations" in self.coco:
            for img, ann in zip(self.coco['images'], self.coco['annotations']):
                #print(img['file_name'], ann['file_name'])
                assert img['file_name'].split('.')[:-1] == ann['file_name'].split('.')[:-1]

        self.img_folder = img_folder
        self.ann_folder = ann_folder
        self.ann_file = ann_file
        self.transforms = transforms
        self.return_masks = return_masks

    def __getitem__(self, idx):
        ann_info = self.coco['annotations'][idx] if "annotations" in self.coco else self.coco['images'][idx]
        img_ext = Path(self.coco['images'][idx]['file_name']).suffix
        img_path = Path(self.img_folder) / ann_info['file_name'].replace('.png', img_ext)
        ann_path = Path(self.ann_folder) / ann_info['file_name']

        img = Image.open(img_path).convert('RGB')
        w, h = img.size
        if "segments_info" in ann_info:
            masks = np.asarray(Image.open(ann_path), dtype=np.uint32) # Read the mask file
            masks = rgb2id(masks) # Convert the mask to the id format

            ids = np.array([ann['id'] for ann in ann_info['segments_info']]) # Get the unique ids of classes in mask
            masks = masks == ids[:, None, None]

            masks = torch.as_tensor(masks, dtype=torch.uint8)
            labels = torch.tensor([ann['category_id'] for ann in ann_info['segments_info']], dtype=torch.int64)

        target = {}
        target['image_id'] = torch.tensor([ann_info['image_id'] if "image_id" in ann_info else ann_info["id"]])
        if self.return_masks:
            target['masks'] = masks
        target['labels'] = labels

        # Calculating BBox using mask, we already have BBox in annotations, we could have directly call ann_info['bbox']
        target["boxes"] = masks_to_boxes(masks) #  target["boxes"] = ann_info['bbox']

        target['size'] = torch.as_tensor([int(h), int(w)])
        target['orig_size'] = torch.as_tensor([int(h), int(w)])
        if "segments_info" in ann_info:
            for name in ['iscrowd', 'area']:
                target[name] = torch.tensor([ann[name] for ann in ann_info['segments_info']])

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.coco['images'])

    def get_height_and_width(self, idx):
        img_info = self.coco['images'][idx]
        height = img_info['height']
        width = img_info['width']
        return height, width<jupyter_output><empty_output><jupyter_text>For panoptic segmentation, we would want the dataset to return the image and the segmentation mask when we call the `__getitem__` method. We also return additoinal details like class label, bounding boxes, orignal size of the image as part of target along with the mask itself.Also, if there are any transformations given, we apply those transformations on both the image and the mask.As discussed in the `Creating Custom Dataset for DETR` post, the mask of the corresponding image should have the same name as the image with the suffix `.png` in the annotations folder.  Transformations<jupyter_code>#collapse-hide

# detr/datasets/construction.py

import datasets.transforms as T

normalize = T.Compose([
        T.ToTensor(),
        T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

scales = [480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800]

if image_set == 'train':
    return T.Compose([
        T.RandomHorizontalFlip(),
        T.RandomSelect(
            T.RandomResize(scales, max_size=1333),
            T.Compose([
                T.RandomResize([400, 500, 600]),
                T.RandomSizeCrop(384, 600),
                T.RandomResize(scales, max_size=1333),
            ])
        ),
        normalize,
    ])

if image_set == 'val':
    return T.Compose([
        T.RandomResize([800], max_size=1333),
        normalize,
    ])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12386766443481685, 'fraction_numerical': 0.01870815281606932, 'mean_word_length': 3.030952380952381, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '594683', 'license': 'Apache-2.0', 'n_tokens_mistral': 1654, 'n_tokens_neox': 1539, 'n_words': 432, 'path': '_notebooks/2021-10-03-PanopticSegmentation-DETR.ipynb', 'repo_name': 'abdksyed/blog', 'size': '5078'}