<jupyter_start><jupyter_text>Mask R-CNN Image Segmentation DemoThis Colab enables you to use a Mask R-CNN model that was trained on Cloud TPU to perform instance segmentation on a sample input image. The resulting predictions are overlayed on the sample image as boxes, instance masks, and labels. You can also experiment with your own images by editing the input image URL. About Mask R-CNNThe Mask R-CNN model addresses one of the most difficult computer vision challenges: image segmentation. Image segmentation is the task of detecting and distinguishing multiple objects within a single image. In particular, Mask R-CNN performs "instance segmentation," which means that different instances of the same type of object in the input image, for example, car, should be assigned distinct labels.  Instructions  &nbsp;&nbsp;Use a free Cloud TPU    1. On the main menu, click Runtime and select **Change runtime type**. Set "TPU" as the hardware accelerator.   2. Click Runtime again and select **Runtime > Run All**. You can also run the cells manually with Shift-ENTER.  Download the source codeDownload the source code of the Mask R-CNN model.<jupyter_code>!git clone https://github.com/tensorflow/tpu/<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>from IPython import display
from PIL import Image
import numpy as np
%tensorflow_version 1.x
import tensorflow as tf
import sys
sys.path.insert(0, 'tpu/models/official')
sys.path.insert(0, 'tpu/models/official/mask_rcnn')
import coco_metric
from mask_rcnn.object_detection import visualization_utils<jupyter_output><empty_output><jupyter_text>Load the COCO index mappingThis Colab uses a pretrained checkpoint of the Mask R-CNN model that is trained using the COCO dataset. Here is the mapping between the indices that the model predicts and the categories in text.<jupyter_code>ID_MAPPING = {
    1: 'person',
    2: 'bicycle',
    3: 'car',
    4: 'motorcycle',
    5: 'airplane',
    6: 'bus',
    7: 'train',
    8: 'truck',
    9: 'boat',
    10: 'traffic light',
    11: 'fire hydrant',
    13: 'stop sign',
    14: 'parking meter',
    15: 'bench',
    16: 'bird',
    17: 'cat',
    18: 'dog',
    19: 'horse',
    20: 'sheep',
    21: 'cow',
    22: 'elephant',
    23: 'bear',
    24: 'zebra',
    25: 'giraffe',
    27: 'backpack',
    28: 'umbrella',
    31: 'handbag',
    32: 'tie',
    33: 'suitcase',
    34: 'frisbee',
    35: 'skis',
    36: 'snowboard',
    37: 'sports ball',
    38: 'kite',
    39: 'baseball bat',
    40: 'baseball glove',
    41: 'skateboard',
    42: 'surfboard',
    43: 'tennis racket',
    44: 'bottle',
    46: 'wine glass',
    47: 'cup',
    48: 'fork',
    49: 'knife',
    50: 'spoon',
    51: 'bowl',
    52: 'banana',
    53: 'apple',
    54: 'sandwich',
    55: 'orange',
    56: 'broccoli',
    57: 'carrot',
    58: 'hot dog',
    59: 'pizza',
    60: 'donut',
    61: 'cake',
    62: 'chair',
    63: 'couch',
    64: 'potted plant',
    65: 'bed',
    67: 'dining table',
    70: 'toilet',
    72: 'tv',
    73: 'laptop',
    74: 'mouse',
    75: 'remote',
    76: 'keyboard',
    77: 'cell phone',
    78: 'microwave',
    79: 'oven',
    80: 'toaster',
    81: 'sink',
    82: 'refrigerator',
    84: 'book',
    85: 'clock',
    86: 'vase',
    87: 'scissors',
    88: 'teddy bear',
    89: 'hair drier',
    90: 'toothbrush',
}
category_index = {k: {'id': k, 'name': ID_MAPPING[k]} for k in ID_MAPPING}<jupyter_output><empty_output><jupyter_text>Load an imageNow, you can load an image. Use either the sample image included here, or update the field with an image of your choice.<jupyter_code>!wget https://upload.wikimedia.org/wikipedia/commons/thumb/0/08/Kitano_Street_Kobe01s5s4110.jpg/2560px-Kitano_Street_Kobe01s5s4110.jpg -O test.jpg
!wget http://farm8.staticflickr.com/7023/6462387023_0bf07efcfd_z.jpg -O test2.jpg
!wget https://farm4.staticflickr.com/3219/3109776113_fe37d76785_z.jpg -O test3.jpg
image_path1 = 'test.jpg'
image_path2 = 'test2.jpg'
image_path3 = 'test3.jpg'

with open(image_path1, 'rb') as f:
  np_image_string = np.array([f.read()])
  
image1 = Image.open(image_path1)
width, height = image1.size
np_image1 = np.array(image1.getdata()).reshape(height, width, 3).astype(np.uint8)

display.display(display.Image(image_path1, width=1024))
with open(image_path2, 'rb') as f:
  np_image_string = np.array([f.read()])
  
image = Image.open(image_path2)
width, height = image.size
np_image2 = np.array(image.getdata()).reshape(height, width, 3).astype(np.uint8)

display.display(display.Image(image_path2, width=1024))
with open(image_path3, 'rb') as f:
  np_image_string = np.array([f.read()])
  
image = Image.open(image_path3)
width, height = image.size
np_image3 = np.array(image.getdata()).reshape(height, width, 3).astype(np.uint8)

display.display(display.Image(image_path3, width=1024))<jupyter_output><empty_output><jupyter_text>Create a Tensorflow sessionNow let us create a Tensorflow session to run the inference. You can either connect to a TPU or a normal CPU backend.<jupyter_code>use_tpu = True #@param {type:"boolean"}
if use_tpu:
  import os
  import pprint

  assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'
  TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']
  print('TPU address is', TPU_ADDRESS)

  session = tf.Session(TPU_ADDRESS, graph=tf.Graph())
  print('TPU devices:')
  pprint.pprint(session.list_devices())
else:
  session = tf.Session(graph=tf.Graph())<jupyter_output><empty_output><jupyter_text>Load the pretrained modelLoading the COCO pretrained saved model from the public GCS bucket.  saved_model_dir = 'gs://cloud-tpu-checkpoints/mask-rcnn/1555659850' @param {type:"string"}_ = tf.saved_model.loader.load(session, ['serve'], saved_model_dir)<jupyter_code>saved_model_dir = 'gs://cloud-tpu-checkpoints/mask-rcnn/1555659850' #@param {type:"string"}
_ = tf.saved_model.loader.load(session, ['serve'], saved_model_dir)<jupyter_output><empty_output><jupyter_text>Perform instance segmentation and retrieve the predictionsNow let's run the inference and process the predictions from the model.<jupyter_code>num_detections, detection_boxes, detection_classes, detection_scores, detection_masks, image_info = session.run(
    ['NumDetections:0', 'DetectionBoxes:0', 'DetectionClasses:0', 'DetectionScores:0', 'DetectionMasks:0', 'ImageInfo:0'],
    feed_dict={'Placeholder:0': np_image_string})

num_detections = np.squeeze(num_detections.astype(np.int32), axis=(0,))
detection_boxes = np.squeeze(detection_boxes * image_info[0, 2], axis=(0,))[0:num_detections]
detection_scores = np.squeeze(detection_scores, axis=(0,))[0:num_detections]
detection_classes = np.squeeze(detection_classes.astype(np.int32), axis=(0,))[0:num_detections]
instance_masks = np.squeeze(detection_masks, axis=(0,))[0:num_detections]
ymin, xmin, ymax, xmax = np.split(detection_boxes, 4, axis=-1)
processed_boxes = np.concatenate([xmin, ymin, xmax - xmin, ymax - ymin], axis=-1)
segmentations = coco_metric.generate_segmentation_from_masks(instance_masks, processed_boxes, height, width)<jupyter_output><empty_output><jupyter_text>Visualize the detection resultsTime to check out the result!<jupyter_code>max_boxes_to_draw = 20   #@param {type:"integer"}
min_score_thresh = 0.05    #@param {type:"slider", min:0, max:1, step:0.01}

image_with_detections = visualization_utils.visualize_boxes_and_labels_on_image_array(
    np_image1,
    detection_boxes,
    detection_classes,
    detection_scores,
    category_index,
    instance_masks=segmentations,
    use_normalized_coordinates=False,
    max_boxes_to_draw=max_boxes_to_draw,
    min_score_thresh=min_score_thresh)
output_image_path1 = 'test_results.jpg'

Image.fromarray(image_with_detections.astype(np.uint8)).save(output_image_path1)
display.display(display.Image(output_image_path1, width=1024))
max_boxes_to_draw = 50   #@param {type:"integer"}
min_score_thresh = 0.05    #@param {type:"slider", min:0, max:1, step:0.01}

image_with_detections = visualization_utils.visualize_boxes_and_labels_on_image_array(
    np_image2,
    detection_boxes,
    detection_classes,
    detection_scores,
    category_index,
    instance_masks=segmentations,
    use_normalized_coordinates=False,
    max_boxes_to_draw=max_boxes_to_draw,
    min_score_thresh=min_score_thresh)
output_image_path2 = 'test_results1.jpg'
Image.fromarray(image_with_detections.astype(np.uint8)).save(output_image_path2)
display.display(display.Image(output_image_path2, width=1024))
max_boxes_to_draw = 50   #@param {type:"integer"}
min_score_thresh = 0.05    #@param {type:"slider", min:0, max:1, step:0.01}

image_with_detections = visualization_utils.visualize_boxes_and_labels_on_image_array(
    np_image3,
    detection_boxes,
    detection_classes,
    detection_scores,
    category_index,
    instance_masks=segmentations,
    use_normalized_coordinates=False,
    max_boxes_to_draw=max_boxes_to_draw,
    min_score_thresh=min_score_thresh)
output_image_path3 = 'test_results2.jpg'
Image.fromarray(image_with_detections.astype(np.uint8)).save(output_image_path)
display.display(display.Image(output_image_path3, width=1024))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12272923875432526, 'fraction_numerical': 0.03892733564013841, 'mean_word_length': 5.056974459724951, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '386310', 'license': 'MIT', 'n_tokens_mistral': 3478, 'n_tokens_neox': 3238, 'n_words': 776, 'path': 'Edurekha_tf_course/Class_3/7th_oct_Mask_R_CNN_Image_Segmentation_7xdqq.ipynb', 'repo_name': 'novoforce/Exploring-Tensorflow', 'size': '9248'}