<jupyter_start><jupyter_text>Semantic Segmentation DemoThis is a notebook for running the benchmark semantic segmentation network from the the [ADE20K MIT Scene Parsing Benchchmark](http://sceneparsing.csail.mit.edu/).The code for this notebook is available herehttps://github.com/CSAILVision/semantic-segmentation-pytorch/tree/master/notebooksIt can be run on Colab at this URL https://colab.research.google.com/github/CSAILVision/semantic-segmentation-pytorch/blob/master/notebooks/DemoSegmenter.ipynb  Environment SetupFirst, download the code and pretrained models if we are on colab.<jupyter_code>%%bash
# Colab-specific setup
!(stat -t /usr/local/lib/*/dist-packages/google/colab > /dev/null 2>&1) && exit 
pip install yacs 2>&1 >> install.log
git init 2>&1 >> install.log
git remote add origin https://github.com/CSAILVision/semantic-segmentation-pytorch.git 2>> install.log
git pull origin master 2>&1 >> install.log
DOWNLOAD_ONLY=1 ./demo_test.sh 2>> install.log<jupyter_output><empty_output><jupyter_text>Imports and utility functionsWe need pytorch, numpy, and the code for the segmentation model.  And some utilities for visualizing the data.<jupyter_code># System libs
import os, csv, torch, numpy, scipy.io, PIL.Image, torchvision.transforms
# Our libs
from mit_semseg.models import ModelBuilder, SegmentationModule
from mit_semseg.utils import colorEncode

colors = scipy.io.loadmat('data/color150.mat')['colors']
names = {}
with open('data/object150_info.csv') as f:
    reader = csv.reader(f)
    next(reader)
    for row in reader:
        names[int(row[0])] = row[5].split(";")[0]

def visualize_result(img, pred, index=None):
    # filter prediction class if requested
    if index is not None:
        pred = pred.copy()
        pred[pred != index] = -1
        print(f'{names[index+1]}:')
        
    # colorize prediction
    pred_color = colorEncode(pred, colors).astype(numpy.uint8)

    # aggregate images and save
    im_vis = numpy.concatenate((img, pred_color), axis=1)
    display(PIL.Image.fromarray(im_vis))<jupyter_output><empty_output><jupyter_text>Loading the segmentation modelHere we load a pretrained segmentation model.  Like any pytorch model, we can call it like a function, or examine the parameters in all the layers.After loading, we put it on the GPU.  And since we are doing inference, not training, we put the model in eval mode.<jupyter_code># Network Builders
net_encoder = ModelBuilder.build_encoder(
    arch='resnet50dilated',
    fc_dim=2048,
    weights='ckpt/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth')
net_decoder = ModelBuilder.build_decoder(
    arch='ppm_deepsup',
    fc_dim=2048,
    num_class=150,
    weights='ckpt/ade20k-resnet50dilated-ppm_deepsup/decoder_epoch_20.pth',
    use_softmax=True)

crit = torch.nn.NLLLoss(ignore_index=-1)
segmentation_module = SegmentationModule(net_encoder, net_decoder, crit)
segmentation_module.eval()
segmentation_module.cuda()<jupyter_output><empty_output><jupyter_text>Load test dataNow we load and normalize a single test image.  Here we use the commonplace convention of normalizing the image to a scale for which the RGB values of a large photo dataset would have zero mean and unit standard deviation.  (These numbers come from the imagenet dataset.)  With this normalization, the limiiting ranges of RGB values are within about (-2.2 to +2.7).<jupyter_code># Load and normalize one image as a singleton tensor batch
pil_to_tensor = torchvision.transforms.Compose([
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(
        mean=[0.485, 0.456, 0.406], # These are RGB mean+std values
        std=[0.229, 0.224, 0.225])  # across a large photo dataset.
])
pil_image = PIL.Image.open('/home/jarvis1121/AI/semantic_segmentation/semantic-segmentation-pytorch/119211376_318640699588996_1291935836100922877_n.jpg').convert('RGB')
img_original = numpy.array(pil_image)
img_data = pil_to_tensor(pil_image)
singleton_batch = {'img_data': img_data[None].cuda()}
output_size = img_data.shape[1:]<jupyter_output><empty_output><jupyter_text>Run the ModelFinally we just pass the test image to the segmentation model.The segmentation model is coded as a function that takes a dictionary as input, because it wants to know both the input batch image data as well as the desired output segmentation resolution.  We ask for full resolution output.Then we use the previously-defined visualize_result function to render the segmentation map.<jupyter_code># Run the segmentation at the highest resolution.
with torch.no_grad():
    scores = segmentation_module(singleton_batch, segSize=output_size)
    
# Get the predicted scores for each pixel
_, pred = torch.max(scores, dim=1)
pred = pred.cpu()[0].numpy()
visualize_result(img_original, pred)<jupyter_output><empty_output><jupyter_text>Showing classes individuallyTo see which colors are which, here we visualize individual classes, one at a time.<jupyter_code># Top classes in answer
predicted_classes = numpy.bincount(pred.flatten()).argsort()[::-1]
for c in predicted_classes[:15]:
    visualize_result(img_original, pred, c)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09612765129402608, 'fraction_numerical': 0.02588052150223779, 'mean_word_length': 5.237864077669903, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 40, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '104869', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1715, 'n_tokens_neox': 1600, 'n_words': 467, 'path': 'notebooks/DemoSegmenter.ipynb', 'repo_name': 'RicoSuaveGuapo/semantic-segmentation-pytorch', 'size': '5139'}