<jupyter_start><jupyter_text>Project : Kangaroo Detection (Custom Object Detector)  Select tensorflow version 1.x (Version 2.x is not supported)<jupyter_code>%tensorflow_version 1.x<jupyter_output>TensorFlow 1.x selected.<jupyter_text>Clone the Mask_RCNN official repository from git to get the dependencies for Mask_RCNN<jupyter_code>!git clone https://github.com/matterport/Mask_RCNN.git<jupyter_output>Cloning into 'Mask_RCNN'...
remote: Enumerating objects: 956, done.[K
remote: Total 956 (delta 0), reused 0 (delta 0), pack-reused 956[K
Receiving objects: 100% (956/956), 116.75 MiB | 33.80 MiB/s, done.
Resolving deltas: 100% (567/567), done.<jupyter_text>Change the working directory to Mask_RCNN directory<jupyter_code>cd Mask_RCNN/<jupyter_output>/content/Mask_RCNN<jupyter_text>Install all the requirements and dependencies for working with Mask-RCNN<jupyter_code>!python setup.py install<jupyter_output>WARNING:root:Fail load requirements file, so using default ones.
running install
running bdist_egg
running egg_info
creating mask_rcnn.egg-info
writing mask_rcnn.egg-info/PKG-INFO
writing dependency_links to mask_rcnn.egg-info/dependency_links.txt
writing top-level names to mask_rcnn.egg-info/top_level.txt
writing manifest file 'mask_rcnn.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'mask_rcnn.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/mrcnn
copying mrcnn/utils.py -> build/lib/mrcnn
copying mrcnn/parallel_model.py -> build/lib/mrcnn
copying mrcnn/__init__.py -> build/lib/mrcnn
copying mrcnn/config.py -> build/lib/mrcnn
copying mrcnn/visualize.py -> build/lib/mrcnn
copying mrcnn/model.py -> build/lib/mrcnn
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/mrcnn
copyi[...]<jupyter_text>Checking whether mask-rcnn is installed or not<jupyter_code>!pip show mask-rcnn<jupyter_output>Name: mask-rcnn
Version: 2.1
Summary: Mask R-CNN for object detection and instance segmentation
Home-page: https://github.com/matterport/Mask_RCNN
Author: Matterport
Author-email: <EMAIL>
License: MIT
Location: /usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg
Requires: 
Required-by:<jupyter_text>Downloading the dataset For downloading the images and annotation files we have to clone the given repository. The given repository contains the images and annotation files<jupyter_code>!git clone https://github.com/experiencor/kangaroo.git<jupyter_output>Cloning into 'kangaroo'...
remote: Enumerating objects: 334, done.[K
Receiving objects:   0% (1/334)   
Receiving objects:   1% (4/334)   
Receiving objects:   2% (7/334)   
Receiving objects:   3% (11/334)   
Receiving objects:   4% (14/334)   
Receiving objects:   5% (17/334)   
Receiving objects:   6% (21/334)   
Receiving objects:   7% (24/334)   
Receiving objects:   8% (27/334)   
Receiving objects:   9% (31/334)   
Receiving objects:  10% (34/334)   
Receiving objects:  11% (37/334)   
Receiving objects:  12% (41/334)   
Receiving objects:  13% (44/334)   
Receiving objects:  14% (47/334)   
Receiving objects:  15% (51/334)   
Receiving objects:  16% (54/334)   
Receiving objects:  17% (57/334)   
Receiving objects:  18% (61/334)   
Receiving objects:  19% (64/334)   
Receiving objects:  20% (67/334)   
Receiving objects:  21% (71/334)   
Receiving objects:  22% (74/334)   
Receiving objects:  23% (77/334)   
Receiving objects:  24% (81/334)   
Receiving objects:  25% (84/334) [...]<jupyter_text>Downloading Pretrained weights Since we are using transfer learning in our project,we download the pretrained weights of mask_rcnn algorithm trained on famous COCO datasets<jupyter_code>!wget https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5<jupyter_output>--2020-05-06 13:21:16--  https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5
Resolving github.com (github.com)... 172.16.17.32
Connecting to github.com (github.com)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107595270/872d3234-d21f-11e7-9a51-7b4bc8075835?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200506%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200506T132116Z&X-Amz-Expires=300&X-Amz-Signature=5a1e1c9c3070e7314dba6c5c1976aff15a6e242354989ee6ff80081513184903&X-Amz-SignedHeaders=host&actor_id=0&repo_id=107595270&response-content-disposition=attachment%3B%20filename%3Dmask_rcnn_coco.h5&response-content-type=application%2Foctet-stream [following]
--2020-05-06 13:21:16--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107595270/872d3234-d21f-11e7-9a51-7b4bc8075835?X-Amz-Algorithm=AWS4-HMAC-SHA256&[...]<jupyter_text>Set the path to the images and annotation files<jupyter_code>kangaroo = "/content/Mask_RCNN/kangaroo"<jupyter_output><empty_output><jupyter_text>Data Preprocessing And Training 1) Firstly, importing necessary packages that will be requiring to do the object detection task.  2) Create the child class KangarooDataset which will inherit the properties from the Dataset class. The Datasett class is inside the utils.py file. Check the utils.py file and find the Dataset class.We inherit the properties from Dataset Class in KangarooDataset class 3) The load_dataset function helps to load the images and annotation files from the dataset. The image with id '0090' is bad so we donot load that image as it can arise eroors in the program. The images with id greater than 150 are splitted as test images and remaining as train images.So, 131 train images and 32 test images are loaded from the load_dataset function 4) The extract_boxes function helps to get the bounding box coordinates and the shapes of all images from the annotation files. 5) The load_mask function helps to create a mask for the predicted image. 6) The image_reference function helps to load an image reference. 7) The KangarooConfig is an inherited class from Config class which helps to set the configuration for running the project. 8) An object is created and the methods of  class are accessed by an object. And finally, the model is loaded with the pretrained weights and trained on the custom kangaroo datasets for 5 epochs with the configurations mentioned in the KangarooConfig class.<jupyter_code># fit a mask rcnn on the kangaroo dataset
from os import listdir
from xml.etree import ElementTree
from numpy import zeros
from numpy import asarray
from mrcnn.utils import Dataset
from mrcnn.config import Config
from mrcnn.model import MaskRCNN
 
# class that defines and loads the kangaroo dataset
class KangarooDataset(Dataset):
	# load the dataset definitions
	def load_dataset(self, dataset_dir, is_train=True):
		# define one class
		self.add_class("dataset", 1, "kangaroo")
		# define data locations
		images_dir = dataset_dir + '/images/'
		annotations_dir = dataset_dir + '/annots/'
		# find all images
		for filename in listdir(images_dir):
			# extract image id
			image_id = filename[:-4]
			# skip bad images
			if image_id in ['00090']:
				continue
			# skip all images after 150 if we are building the train set
			if is_train and int(image_id) >= 150:
				continue
			# skip all images before 150 if we are building the test/val set
			if not is_train and int(image_id) < 150:
				continue
			img_path = images_dir + filename
			ann_path = annotations_dir + image_id + '.xml'
			# add to dataset
			self.add_image('dataset', image_id=image_id, path=img_path, annotation=ann_path)
 
	# extract bounding boxes from an annotation file
	def extract_boxes(self, filename):
		# load and parse the file
		tree = ElementTree.parse(filename)
		# get the root of the document
		root = tree.getroot()
		# extract each bounding box
		boxes = list()
		for box in root.findall('.//bndbox'):
			xmin = int(box.find('xmin').text)
			ymin = int(box.find('ymin').text)
			xmax = int(box.find('xmax').text)
			ymax = int(box.find('ymax').text)
			coors = [xmin, ymin, xmax, ymax]
			boxes.append(coors)
		# extract image dimensions
		width = int(root.find('.//size/width').text)
		height = int(root.find('.//size/height').text)
		return boxes, width, height
 
	# load the masks for an image
	def load_mask(self, image_id):
		# get details of image
		info = self.image_info[image_id]
		# define box file location
		path = info['annotation']
		# load XML
		boxes, w, h = self.extract_boxes(path)
		# create one array for all masks, each on a different channel
		masks = zeros([h, w, len(boxes)], dtype='uint8')
		# create masks
		class_ids = list()
		for i in range(len(boxes)):
			box = boxes[i]
			row_s, row_e = box[1], box[3]
			col_s, col_e = box[0], box[2]
			masks[row_s:row_e, col_s:col_e, i] = 1
			class_ids.append(self.class_names.index('kangaroo'))
		return masks, asarray(class_ids, dtype='int32')
 
	# load an image reference
	def image_reference(self, image_id):
		info = self.image_info[image_id]
		return info['path']
 
# define a configuration for the model
class KangarooConfig(Config):
	# define the name of the configuration
	NAME = "kangaroo_cfg"
	# number of classes (background + kangaroo)
	NUM_CLASSES = 1 + 1
	# number of training steps per epoch
	STEPS_PER_EPOCH = 131
 
# prepare train set
train_set = KangarooDataset()
train_set.load_dataset('kangaroo', is_train=True)
train_set.prepare()
print('Train: %d' % len(train_set.image_ids))
# prepare test/val set
test_set = KangarooDataset()
test_set.load_dataset('kangaroo', is_train=False)
test_set.prepare()
print('Test: %d' % len(test_set.image_ids))
# prepare config
config = KangarooConfig()
config.display()
# define the model
model = MaskRCNN(mode='training', model_dir='./', config=config)
# load weights (mscoco) and exclude the output layers
model.load_weights('/content/Mask_RCNN/mask_rcnn_coco.h5', by_name=True, exclude=["mrcnn_class_logits", "mrcnn_bbox_fc",  "mrcnn_bbox", "mrcnn_mask"])
# train weights (output layers or 'heads')
model.train(train_set, test_set, learning_rate=config.LEARNING_RATE, epochs=5, layers='heads')<jupyter_output>Train: 131
Test: 32

Configurations:
BACKBONE                       resnet101
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     2
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE         None
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.7
DETECTION_NMS_THRESHOLD        0.3
FPN_CLASSIF_FC_LAYERS_SIZE     1024
GPU_COUNT                      1
GRADIENT_CLIP_NORM             5.0
IMAGES_PER_GPU                 2
IMAGE_CHANNEL_COUNT            3
IMAGE_MAX_DIM                  1024
IMAGE_META_SIZE                14
IMAGE_MIN_DIM                  800
IMAGE_MIN_SCALE                0
IMAGE_RESIZE_MODE              square
IMAGE_SHAPE                    [1024 1024    3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14[...]<jupyter_text>Importing the necessary packages for evaluating the mask rcnn model on the kangaroo dataset<jupyter_code># evaluate the mask rcnn model on the kangaroo dataset
from os import listdir
from xml.etree import ElementTree
from numpy import zeros
from numpy import asarray
from numpy import expand_dims
from numpy import mean
from mrcnn.config import Config
from mrcnn.model import MaskRCNN
from mrcnn.utils import Dataset
from mrcnn.utils import compute_ap
from mrcnn.model import load_image_gt
from mrcnn.model import mold_image<jupyter_output><empty_output><jupyter_text>Evaluating the MAP(mean average precision) for real bounding boxes and the bounding boxes predicted. Set the configuration for prediction. The prediction config class is expected to use GPU for inferencing. You can avoid it if you donot have GPU installed on your system. The average precision is calculated for each image by comparing the bounding box localization of real images and predicted image and finally calculating the mean average precision for all the images.  Finally we calculate the training and testing mean average precision to know how the model is performing.<jupyter_code># define the prediction configuration
class PredictionConfig(Config):
	# define the name of the configuration
	NAME = "kangaroo_cfg"
	# number of classes (background + kangaroo)
	NUM_CLASSES = 1 + 1
	# simplify GPU config
	GPU_COUNT = 1
	IMAGES_PER_GPU = 1
 
# calculate the mAP for a model on a given dataset
def evaluate_model(dataset, model, cfg):
	APs = list()
	for image_id in dataset.image_ids:
		# load image, bounding boxes and masks for the image id
		image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(dataset, cfg, image_id, use_mini_mask=False)
		# convert pixel values (e.g. center)
		scaled_image = mold_image(image, cfg)
		# convert image into one sample
		sample = expand_dims(scaled_image, 0)
		# make prediction
		yhat = model.detect(sample, verbose=0)
		# extract results for first sample
		r = yhat[0]
		# calculate statistics, including AP
		AP, _, _, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
		# store
		APs.append(AP)
	# calculate the mean AP across all images
	mAP = mean(APs)
	return mAP
 
# load the train dataset
train_set = KangarooDataset()
train_set.load_dataset('kangaroo', is_train=True)
train_set.prepare()
print('Train: %d' % len(train_set.image_ids))
# load the test dataset
test_set = KangarooDataset()
test_set.load_dataset('kangaroo', is_train=False)
test_set.prepare()
print('Test: %d' % len(test_set.image_ids))
# create config
cfg = PredictionConfig()
# define the model
model = MaskRCNN(mode='inference', model_dir='./', config=cfg)
# load model weights
model.load_weights('/content/Mask_RCNN/kangaroo_cfg20200506T1321/mask_rcnn_kangaroo_cfg_0005.h5', by_name=True)
# evaluate model on training dataset
train_mAP = evaluate_model(train_set, model, cfg)
print("Train mAP: %.3f" % train_mAP)
# evaluate model on test dataset
test_mAP = evaluate_model(test_set, model, cfg)
print("Test mAP: %.3f" % test_mAP)<jupyter_output>Train: 131
Test: 32
WARNING:tensorflow:From /content/Mask_RCNN/mrcnn/model.py:720: The name tf.sets.set_intersection is deprecated. Please use tf.sets.intersection instead.

WARNING:tensorflow:From /content/Mask_RCNN/mrcnn/model.py:722: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.

WARNING:tensorflow:From /content/Mask_RCNN/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Re-starting from epoch 5
Train mAP: 0.882
Test mAP: 0.896<jupyter_text>Visualizing the outputs  Importing necessary libraries for visualization (Matplotlib is used)<jupyter_code>from matplotlib import pyplot
from matplotlib.patches import Rectangle<jupyter_output><empty_output><jupyter_text>Visualizing Actual location of bounding box in an image Vs Predicted bounding box in an image.  The bounding box is locating the kangaroo object in an image<jupyter_code># plot a number of photos with ground truth and predictions
def plot_actual_vs_predicted(dataset, model, cfg, n_images=5):
	# load image and mask
	for i in range(n_images):
		# load the image and mask
		image = dataset.load_image(i)
		mask, _ = dataset.load_mask(i)
		# convert pixel values (e.g. center)
		scaled_image = mold_image(image, cfg)
		# convert image into one sample
		sample = expand_dims(scaled_image, 0)
		# make prediction
		yhat = model.detect(sample, verbose=0)[0]
		# define subplot
		pyplot.subplot(n_images, 4, i*4+1)
		# plot raw pixel data
		pyplot.imshow(image)
		pyplot.title('Actual')
		# plot masks
		for j in range(mask.shape[2]):
			pyplot.imshow(mask[:, :, j], cmap='gray', alpha=0.3)
		# get the context for drawing boxes
		pyplot.subplot(n_images, 2, i*2+2)
		# plot raw pixel data
		pyplot.imshow(image)
		pyplot.title('Predicted')
		ax = pyplot.gca()
		# plot each box
		for box in yhat['rois']:
			# get coordinates
			y1, x1, y2, x2 = box
			# calculate width and height of the box
			width, height = x2 - x1, y2 - y1
			# create the shape
			rect = Rectangle((x1, y1), width, height, fill=False, color='red')
			# draw the box
			ax.add_patch(rect)
	# show the figure
	pyplot.show()
 
# load the train dataset
train_set = KangarooDataset()
train_set.load_dataset('kangaroo', is_train=True)
train_set.prepare()
print('Train: %d' % len(train_set.image_ids))
# load the test dataset
test_set = KangarooDataset()
test_set.load_dataset('kangaroo', is_train=False)
test_set.prepare()
print('Test: %d' % len(test_set.image_ids))
# create config
cfg = PredictionConfig()
# define the model
model = MaskRCNN(mode='inference', model_dir='./', config=cfg)
# load model weights
model_path = '/content/Mask_RCNN/kangaroo_cfg20200506T1321/mask_rcnn_kangaroo_cfg_0005.h5'
model.load_weights(model_path, by_name=True)
# plot predictions for train dataset
plot_actual_vs_predicted(train_set, model, cfg)
# plot predictions for test dataset
plot_actual_vs_predicted(test_set, model, cfg)<jupyter_output>Train: 131
Test: 32
Re-starting from epoch 5
{'fraction_non_alphanumeric': 0.09451850149390945, 'fraction_numerical': 0.04039301310043668, 'mean_word_length': 4.585686777920411, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 51, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '632319', 'license': 'MIT', 'n_tokens_mistral': 6519, 'n_tokens_neox': 5836, 'n_words': 1707, 'path': 'kangaroo_detector.ipynb', 'repo_name': 'prabhat-123/Custom_object_detector-Kangaroo-Detection-', 'size': '17425'}