<jupyter_start><jupyter_text>Label Propagaton HelpersThis notebook has helpers for the following:*  Pre label propagation  * Takes in a COCO json and converts *.npy segmentation maps for each image collected from Replica to *.png to feed to label propagation techniques.*  Insert your own label propagation technique here, to take in segmentation maps in png and output png*  Post label propagation  * Converts *.png segmentation maps output by label propagation into *.npy segmentation maps  * Outputs metrics (MSE) to evaluate the quality of label propagation, including visualizations<jupyter_code>import numpy as np
import sys
if '/opt/ros/kinetic/lib/python2.7/dist-packages' in sys.path:
    sys.path.remove('/opt/ros/kinetic/lib/python2.7/dist-packages')

import cv2
import random
import os
import numpy as np
import json
from pycocotools.coco import COCO
from PIL import Image
%matplotlib inline
import matplotlib.pyplot as plt
import png<jupyter_output><empty_output><jupyter_text>Pre label propagationconvert *.npy to *.png<jupyter_code>train_json = '/home/locobotm/AVD/habitat_data_with_seg/train.json'
gt_seg_dir =  '/home/locobotm/AVD/habitat_data_with_seg/seg/' # folder with GT segmentation maps collected from Replica
out_seg_dir = '/home/locobotm/AVD/habitat_data_with_seg/prop_3/' # dumps *.pngs here
coco = COCO(train_json)
to_propogate = coco.getImgIds()
print("{} images to prepare".format(len(to_propogate)))

if not os.path.exists(out_seg_dir):
    print("Creating directory {}".format(out_seg_dir))
    os.makedirs(out_seg_dir)

# Copy *.npy to *.png
for x in to_propogate:
    filename = "{:05d}".format(x)
    png_f = os.path.join(out_seg_dir + filename + '.png')
    semantic_obs = np.load(os.path.join(gt_seg_dir, filename + '.npy')).astype(np.uint8)    
    semantic_img = png.from_array(semantic_obs, 'L').save(png_f)<jupyter_output>loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
120 images to prepare<jupyter_text>Do label propagation on `out_seg_dir` images  Post label propagation  * Converts *.png segmentation maps output by label propagation into *.npy segmentation maps  * Outputs metrics (MSE) to evaluate the quality of label propagation, including visualization<jupyter_code>prop_seg_dir =  '/home/locobotm/semantic-segmentation/masks/demoVideo/rgb/' # Label propagated segmentation maps
out_npy_dir = '/home/locobotm/AVD/habitat_data_with_seg/seg_p/' # dump *.npy segmentation maps here
if not os.path.exists(out_npy_dir):
    print("Creating folder {}".format(out_npy_dir))
    os.makedirs(out_npy_dir)

prop_fs = [x for x in os.listdir(prop_seg_dir)]
print("Found {} segmentation maps.".format(len(aug_fs)))

for x in prop_fs:
    img = Image.open( os.path.join(prop_seg_dir, x))
    filename = x.split('.')[0] +'.npy'
    data = np.array(img, dtype='uint8')
    np.save(os.path.join(out_npy_dir, filename), data)

print("Dumped npy segmentation maps at {}".format(out_npy_dir))<jupyter_output>Found 240 segmentation maps.
Dumped npy segmentation maps at /home/locobotm/AVD/habitat_data_with_seg/seg_p/<jupyter_text>Evaluate label propagated segmentation mapsMSE (`out_npy_dir`, `gt_seg_dir`)<jupyter_code>prop_seg = os.listdir(out_npy_dir)
mse = {}
cnt = 0
for x in prop_seg:
    img = int(x.split('.')[0])
    if img not in to_propogate:
        cnt += 1
        prop_data = np.load(os.path.join(out_npy_dir) + x).astype(np.uint8)
        gt_data = np.load(os.path.join(gt_seg_dir) + x).astype(np.uint8)
        mse[img] = (np.square(prop_data-gt_data)).mean(axis=None)

print("MSE {}, over {} frames".format(sum(mse.values()), cnt))

print("Display highest 5 MSE segm maps ... ")
for k, v in sorted(mse.items(), key = lambda item: item[1], reverse=True)[:5]:
    filename = "{:05d}.npy".format(k)
    prop_data = np.load(os.path.join(out_npy_dir) + filename).astype(np.uint8)
    gt_data = np.load(os.path.join(gt_seg_dir) + filename).astype(np.uint8)
    
    plt.figure(figsize=(12 , 8))
    ax = plt.subplot(1, 2, 1)
    ax.axis('off')
    ax.set_title('propagated')
    plt.imshow(prop_data)

    ax = plt.subplot(1, 2, 2)
    ax.axis('off')
    ax.set_title("gt")
    plt.imshow(gt_data)

    plt.show()<jupyter_output>MSE 374.5859680175781, over 120 frames
Display highest 5 MSE segm maps ...
{'fraction_non_alphanumeric': 0.1252336448598131, 'fraction_numerical': 0.01425233644859813, 'mean_word_length': 4.647757255936676, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '384260', 'license': 'MIT', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1521, 'n_words': 366, 'path': 'examples_and_tutorials/notebooks/active_vision/label_prop_helpers.ipynb', 'repo_name': 'bharadwaj1098/droidlet', 'size': '4280'}