<jupyter_start><jupyter_text>Mask R-CNNApplying a mask that will outline car damage.Mask_RCNN GitHub: https://github.com/matterport/Mask_RCNNDownload Coco at: https://github.com/matterport/Mask_RCNN/releasesThis code will work on this docker container: https://hub.docker.com/r/waleedka/modern-deep-learning/NOTE: the logs folder and mask_rcnn_coco.h5 files were not added to GitHub because they are too big. If you want to run this code, you must download coco and run the training in the cell below "Model Setup".  IMPORTANT:To train this model, I had to make the updates to model.py detailed here:https://github.com/keras-team/keras/issues/10311<jupyter_code>import json
import math
import numpy as np
import os
import random
import tensorflow as tf

import matplotlib.pyplot as plt

from mrcnn import utils
from mrcnn import visualize
from mrcnn.visualize import display_images
from mrcnn import model
import mrcnn.model as modellib
from mrcnn.model import log
import custom
# Load dataset
DATA_DIR = os.getcwd() + "/data/"
dataset = custom.CustomDataset()
dataset.load_custom(DATA_DIR, "train")

# Must call before using the dataset
dataset.prepare()

print("Image Count: {}".format(len(dataset.image_ids)))
print("Class Count: {}".format(dataset.num_classes))
for i, info in enumerate(dataset.class_info):
    print("{:3}. {:50}".format(i, info['name']))<jupyter_output>Image Count: 641
Class Count: 2
  0. BG                                                
  1. damage<jupyter_text>Visualize some car damage images and associated masksThe second column are the masks we annotated for the cars shown on the left. If there is more than one area with damage, then the other annotations would show up in theplots to the left.<jupyter_code># Load and display random samples
image_ids = np.random.choice(dataset.image_ids, 5)
for image_id in image_ids:
    image = dataset.load_image(image_id)
    mask, class_ids = dataset.load_mask(image_id)
    visualize.display_top_masks(image, mask, class_ids, dataset.class_names)<jupyter_output><empty_output><jupyter_text>Check the Bounding Box with annotated damage mask for a typical car imageHere is an example image with our annotation and bounding box around it.<jupyter_code>image_id = random.choice(dataset.image_ids)
image = dataset.load_image(image_id)
mask, class_ids = dataset.load_mask(image_id)
# Compute Bounding box
bbox = utils.extract_bboxes(mask)

# Display image and additional stats
print("image_id ", image_id, dataset.image_reference(image_id))
log("image", image)
log("mask", mask)
log("class_ids", class_ids)
log("bbox", bbox)
# Display image and instances
visualize.display_instances(image, bbox, mask, class_ids, dataset.class_names)
#Annotation file load
annotations = list(json.load(open(os.path.join(DATA_DIR + "/train/via_region_data.json"),encoding="utf8")).values())
annotations = [a for a in annotations if a['regions']]
annotations[0]<jupyter_output><empty_output><jupyter_text>Model Setup  Train the model<jupyter_code>## this will generate 50 .h5 files. We will use them to load weights below. 
# !python custom.py train --dataset=/Users/patriciadegner/Desktop/car_damage_upload/unsupervised_learning/data/ --weights=coco
config = custom.CustomConfig()
class InferenceConfig(config.__class__):
    # Run detection on one image at a time
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

config = InferenceConfig()
config.display()

# Device to load the neural network on
DEVICE = "/cpu:0"  # /cpu:0 or /gpu:0

# Inspect the model in training or inference modes
# values: 'inference' or 'training'
TEST_MODE = "inference"<jupyter_output>Configurations:
BACKBONE                       resnet101
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE         None
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.9
DETECTION_NMS_THRESHOLD        0.3
FPN_CLASSIF_FC_LAYERS_SIZE     1024
GPU_COUNT                      1
GRADIENT_CLIP_NORM             5.0
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  1024
IMAGE_META_SIZE                14
IMAGE_MIN_DIM                  800
IMAGE_MIN_SCALE                0
IMAGE_RESIZE_MODE              square
IMAGE_SHAPE                    [1024 1024    3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTA[...]<jupyter_text>Load model weights<jupyter_code>def get_ax(rows=1, cols=1, size=16):
    """Return a Matplotlib Axes array to be used in
    all visualizations in the notebook. Provide a
    central point to control graph sizes.
    
    Adjust the size attribute to control how big to render images
    """
    _, ax = plt.subplots(rows, cols, figsize=(size*cols, size*rows))
    return ax

from importlib import reload # was constantly changing the visualization, so I decided to reload it instead of notebook
reload(visualize)

# Create model in inference mode
MODEL_DIR = "/Users/patriciadegner/Desktop/car_damage_upload/unsupervised_learning/logs/damage20201021T1457"
custom_WEIGHTS_PATH = MODEL_DIR + "/mask_rcnn_damage_0050.h5"
with tf.device(DEVICE):
    model = modellib.MaskRCNN(mode="inference", model_dir=MODEL_DIR, config=config)

# load the last best model you trained
# weights_path = model.find_last()[1]

# Load weights
print("Loading weights ", custom_WEIGHTS_PATH)
load_weights(model, custom_WEIGHTS_PATH, by_name=True)   
visualize.display_weight_stats(model)<jupyter_output><empty_output><jupyter_text>Predict on All Validation ImagesThen view a random example<jupyter_code>import glob
import skimage.draw

results=[]
val_dir = DATA_DIR + 'val/'
filelist = glob.glob(os.path.join(val_dir, '*.JPEG')) 
filelist.sort()

for i, image_path in enumerate(filelist):
    image = skimage.io.imread(image_path)
    
    # Run object detection
    result = model.detect([image])
    results.append(result)

pos_preds = 0
for i in range(len(results)):
    print((filelist[i][-9:]), ": Class = ",results[i][0].get("class_ids", 0), "  ", "Score = ",results[i][0].get("scores", 0))
    if len(results[i][0].get("class_ids", 0)) >0: pos_preds+=1
        
print ("number of images validated =", len(results))        
print ("accuracy =", pos_preds/len(results))
test_dir = DATA_DIR + 'test/'
filelist = glob.glob(os.path.join(test_dir, '*.JPEG'))

image_id = random.choice(filelist)
image = skimage.io.imread(image_id)

# Run object detection
results = model.detect([image], verbose=1)

# Display results
ax = get_ax(1)
r = results[0]
visualize.display_instances(image, r['rois'], r['masks'], r['class_ids'], 
                            'damage', r['scores'], ax=ax,
                            title="Predictions")<jupyter_output>Processing 1 images
image                    shape: (225, 225, 3)         min:    0.00000  max:  255.00000  uint8
molded_images            shape: (1, 1024, 1024, 3)    min: -123.70000  max:  149.10000  float64
image_metas              shape: (1, 14)               min:    0.00000  max: 1024.00000  float64
anchors                  shape: (1, 261888, 4)        min:   -0.35390  max:    1.29134  float32
{'fraction_non_alphanumeric': 0.1000408329930584, 'fraction_numerical': 0.03130529467809991, 'mean_word_length': 3.6097867001254706, 'pattern_counts': {'":': 1, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '338756', 'license': 'Unlicense', 'n_tokens_mistral': 2555, 'n_tokens_neox': 2356, 'n_words': 658, 'path': 'Machine_Learning/car_damage/unsupervised_learning/Mask_rcnn.ipynb', 'repo_name': 'pdegner/pdegner.github.io', 'size': '7347'}