<jupyter_start><jupyter_text>**1. Operations on New Virtual Machines**  1.1 Upgrade to Tensorflow 2.0, Install dependencies<jupyter_code># !pip install tensorflow-gpu
!pip install --no-dependencies image-classifiers<jupyter_output>Collecting image-classifiers
  Downloading https://files.pythonhosted.org/packages/81/98/6f84720e299a4942ab80df5f76ab97b7828b24d1de5e9b2cbbe6073228b7/image_classifiers-1.0.0-py3-none-any.whl
Installing collected packages: image-classifiers
Successfully installed image-classifiers-1.0.0<jupyter_text>1.2 Test if GPU is available<jupyter_code>import tensorflow as tf

# print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))

# tf.debugging.set_log_device_placement(True)

# # Create some tensors
# a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
# b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
# c = tf.matmul(a, b)
# print(c)

if tf.test.gpu_device_name():
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
else:
    print("Please install GPU version of TF")

# device_name = tf.test.gpu_device_name()
# if device_name != '/device:GPU:0':
#     raise SystemError('GPU device not found')
# print('Found GPU at: {}'.format(device_name))
# !mkdir dataset
# !cp -r "/content/drive/My Drive/Colab Notebooks/MARDI-dataset/labelled/" ./dataset
# !ls
# !pip freeze
import tensorflow.keras.backend as K
K.clear_session()
!nvidia-smi<jupyter_output>Mon Dec  9 07:38:37 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P0    32W / 250W |    265MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU  [...]<jupyter_text>1.3 Mount with Google Drive<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
%cd "drive/My Drive/Colab Notebooks"<jupyter_output><empty_output><jupyter_text>1.4 Speak!<jupyter_code>def speak(text):
    from IPython.display import Javascript as js, clear_output
    # Escape single quotes
    text = text.replace("'", r"\'")
    display(js(f'''
    if(window.speechSynthesis) {{
        var synth = window.speechSynthesis;
        synth.speak(new window.SpeechSynthesisUtterance('{text}'));
    }}
    '''))
    # Clear the JS so that the notebook doesn't speak again when reopened/refreshed
#     clear_output(False)
speak('Hi boss, welcome back.')<jupyter_output><empty_output><jupyter_text>**2. Program Codes Start Here**  2.1 Generate Anchors based on Training Sets<jupyter_code>from k_means_anchors import generate_anchors
generate_anchors('MARDI-dataset/labelled/annotations/', 'MARDI-dataset/labelled/images/', ['bee-A', 'bee-B'], 1056, 7)

speak('Anchors Generated!')<jupyter_output>iteration 1: dists = 62685.49901184575
iteration 2: dists = 6750.775055113117
iteration 3: dists = 4095.776469789351
iteration 4: dists = 2229.83897316567
iteration 5: dists = 1089.0635161198704
iteration 6: dists = 611.8322284194287
iteration 7: dists = 454.9602448992751
iteration 8: dists = 485.82496732916866
iteration 9: dists = 513.4432020862406
iteration 10: dists = 551.7475972262221
iteration 11: dists = 443.3411064632691
iteration 12: dists = 343.5092799149165
iteration 13: dists = 386.49068982316476
iteration 14: dists = 331.80329426121347
iteration 15: dists = 348.89090773262006
iteration 16: dists = 338.7117428111689
iteration 17: dists = 325.01915770192267
iteration 18: dists = 237.70602588307165
iteration 19: dists = 274.64705696509645
iteration 20: dists = 222.40900200485953
iteration 21: dists = 275.66417061059246
iteration 22: dists = 197.91091432790023
iteration 23: dists = 249.92101480129475
iteration 24: dists = 215.74414927001882
iteration 25: dists = 177.55861335671[...]<jupyter_text>2.2 Initialize network model<jupyter_code>from architecture import VGG16Net
from architecture import TinyYoloNet
from architecture import YoloNet
from architecture import YoloMobileNet
from architecture import YoloResNet50
from architecture import YoloResNet18
anchors = [0.64,1.13, 0.65,1.82, 0.91,1.51, 0.95,2.23, 1.05,1.10, 1.30,1.58, 1.39,2.32]

# base_net = TinyYoloNet(net_input_size=1056, 
#                        anchors=anchors, 
#                        n_class=2, 
#                        weights_dir=None, 
#                        labels=['bee-A', 'bee-B'])

# base_net = YoloMobileNet(net_input_size=1056, 
#                        anchors=anchors, 
#                        n_class=2, 
#                        weights_dir=None, 
#                        labels=['bee-A', 'bee-B'])

# base_net = YoloNet(net_input_size=1056, 
#                        anchors=anchors, 
#                         n_class=2, 
#                         weights_dir=None, 
#                         labels=['bee-A', 'bee-B'])

# base_net = VGG16Net(net_input_size=1056, 
#                        anchors=anchors, 
#                        n_class=2, 
#                        weights_dir=None,
#                        labels=['bee-A', 'bee-B'])

# base_net = YoloResNet50(net_input_size=1056, 
#                        anchors=anchors, 
#                        n_class=2, 
#                        weights_dir=None, 
#                        labels=['bee-A', 'bee-B'])

# base_net = YoloResNet18(net_input_size=1056, 
#                       anchors=anchors, 
#                       n_class=2, 
#                       weights_dir=None, 
#                       labels=['bee-A', 'bee-B'])

# base_net = YoloResNet18(net_input_size=1056, 
#                       anchors=anchors, 
#                       n_class=1, 
#                       weights_dir=None, 
#                       labels=['bee-A'])


feature_extractor = base_net.create_base_network(transfer_learning=False)
feature_extractor.summary()<jupyter_output>Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 1056, 1056, 3)]   0         
_________________________________________________________________
conv_1 (Conv2D)              (None, 1056, 1056, 16)    432       
_________________________________________________________________
norm_1 (BatchNormalization)  (None, 1056, 1056, 16)    64        
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 1056, 1056, 16)    0         
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 528, 528, 16)      0         
_________________________________________________________________
conv_2 (Conv2D)              (None, 528, 528, 32)      4608      
_____________________________________________________________[...]<jupyter_text>2.2.1 Other options to initialize the model (not required to run all)  If choose to load pretrained network model:<jupyter_code>from tensorflow.keras.models import load_model
# model = load_model('weights_models/resne18-1056-mardi-labelled-onetype461-bee-model.h5', compile=False)

base_net.create_model(feature_extractor)

model = load_model('Final Model Weights/resnet18-aug-1056-mardi-labelled-twotypes200-bee-model.h5', custom_objects={'custom_loss': base_net.custom_loss})<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer ins[...]<jupyter_text>If choose to load pretrained weights:<jupyter_code>model = base_net.create_model(feature_extractor)
model.load_weights('Final Model Weights/resnet18-aug-1056-mardi-labelled-twotypes200-bee-weights.h5')
model.summary()<jupyter_output><empty_output><jupyter_text>If choose to load pretrained weights that does not match fully with self network:<jupyter_code>from tensorflow.keras.models import load_model

model_test = load_model('Final Model Weights/resnet18-aug-1056-mardi-labelled-twotypes200-bee-model', compile=False)
# model_test.summary()

pretrained_layers = model_test.layers[-3].layers

for i in range(len(feature_extractor.layers)):
    feature_extractor.layers[i].set_weights(pretrained_layers[i].get_weights())<jupyter_output><empty_output><jupyter_text>2.3 Else, continue without pretrained weights<jupyter_code>model = base_net.create_model(feature_extractor)
model.summary()<jupyter_output>Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 1056, 1056, 3)]   0         
_________________________________________________________________
model (Model)                (None, 33, 33, 512)       14714688  
_________________________________________________________________
DetectionLayer (Conv2D)      (None, 33, 33, 49)        25137     
_________________________________________________________________
reshape (Reshape)            (None, 33, 33, 7, 7)      0         
=================================================================
Total params: 14,739,825
Trainable params: 14,739,825
Non-trainable params: 0
_________________________________________________________________<jupyter_text>**3. Load Dataset**  3.1.1 If presaved pickle is available<jupyter_code>import pickle

with open('imgs.pkl', 'rb') as imgs_file:
    imgs = pickle.load(imgs_file)<jupyter_output><empty_output><jupyter_text>3.1.2 Else, continue to load all xml annotations, save the instance with pickle is encouraged<jupyter_code>from utils import parse_annotation
# imgs, _ = parse_annotation('MARDI-dataset/train-test-split/annotations/', 'MARDI-dataset/train-test-split/images/', ['bee-A', 'bee-B'])
# test_imgs, _ = parse_annotation('MARDI-dataset/train-test-split/test-annotations/', 'MARDI-dataset/train-test-split/test-images/', ['bee-A', 'bee-B'])

# imgs, _ = parse_annotation('MARDI-dataset/labelled/annotations/', 'MARDI-dataset/labelled/images/', ['bee-A', 'bee-B'])
imgs, _ = parse_annotation('MARDI-dataset/two-types-training/annotations/', 'MARDI-dataset/two-types-training/images/', ['bee-A', 'bee-B'])
import pickle

with open('imgs.pkl', 'wb') as imgs_file:
	pickle.dump(imgs, imgs_file)<jupyter_output><empty_output><jupyter_text>3.2 Operations on datasets<jupyter_code>import random
random.Random(2).shuffle(imgs)
# random.shuffle(imgs)
test_imgs = imgs[:60]
imgs = imgs[60:]
# Train-Test-Val split
import math

n_test_set = math.floor(len(imgs) * 0.2)

test_imgs = imgs[:n_test_set]
imgs = imgs[n_test_set:]

n_val_set = math.floor(len(imgs) * 0.1)
val_imgs = imgs[:n_val_set]
imgs = imgs[n_val_set:]
print(len(imgs))
print(len(val_imgs))
print(len(test_imgs))<jupyter_output>722
80
200<jupyter_text>3.3 If generating pure Xy train-test set is needed (not required to run all)<jupyter_code># X_train, y_train = generate_Xy(imgs=imgs, 
#                                labels=['raccoon'], 
#                                anchors=anchors, 
#                                n_grid=7, 
#                                net_input_size=224, 
#                                n_class=1, 
#                                normalize=base_net.normalize)

# X_test, y_test = generate_Xy(imgs=test_imgs, 
#                              labels=['bee-A'], 
#                              anchors=anchors, 
#                              n_grid=19, 
#                              net_input_size=608, 
#                              n_class=1, 
#                              normalize=base_net.normalize)
# print(X_train.shape)
# print(y_train.shape)<jupyter_output><empty_output><jupyter_text>**4. Train Model**  4.1 Optional Configurations (Tensorboard, etc)<jupyter_code># %load_ext tensorboard

# import datetime, os
# from keras.callbacks import TensorBoard

# logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
# tensorboard_callback = TensorBoard(logdir, histogram_freq=1)
# %tensorboard --logdir logs
# from tensorboard import notebook
# notebook.list()<jupyter_output><empty_output><jupyter_text>4.2 Compile Model<jupyter_code>from tensorflow.keras.optimizers import Adam, SGD
learning_rate = 1e-4
EPOCHS = 100
decay = learning_rate / EPOCH

optimizer = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
BATCH_SIZE = 2
VAL_BATCH_SIZE = 2

train_loss = []
val_loss = []
model = base_net.compile_model(model, optimizer)<jupyter_output><empty_output><jupyter_text>4.3 Callbacks - Custom Model Evaluation/Early Stopping<jupyter_code>model_save_path = 'Final Model Weights/vgg16-noaug-1056-mardi-labelled-twotypes200-bee-model.h5'
weights_save_path = 'Final Model Weights/vgg16-noaug-1056-mardi-labelled-twotypes200-bee-weights.h5'
from tensorflow.keras.callbacks import Callback
from tensorflow.keras.callbacks import EarlyStopping

class EvaluationMAP(Callback):
    def __init__(self, base_net, test_imgs, obj_threshold, nms_threshold, iou_threshold, patience=10):
        super(Callback, self).__init__()
        self.base_net = base_net
        self.test_imgs = test_imgs
        self.obj_threshold = obj_threshold
        self.nms_threshold = nms_threshold
        self.iou_threshold = iou_threshold
        
        self.maximum_map = float('-inf')
        self.patience = patience
        self.wait = 0
        self.best_weights = None
        self.best_epoch = 0
        self.all_logs = {'val_loss': [], 'train_loss': [], 'mAP': [], 'aIOU': []}
    
    def on_epoch_end(self, epoch, logs=None):
        # base_net.evaluate(self.model, self.test_imgs, self.obj_threshold, self.nms_threshold)[0]
        self.all_logs['val_loss'].append(logs['val_loss'])
        self.all_logs['train_loss'].append(logs['loss'])
        if(logs['loss'] < 0.15):
            print()
            print('Evaluating...')
            current_map = base_net.evaluate(self.model, self.test_imgs, self.obj_threshold, self.nms_threshold, self.iou_threshold)
            self.all_logs['mAP'].append(current_map[1])
            self.all_logs['aIOU'].append(current_map[2])
            print()
            if self.maximum_map <= current_map[0]:
                self.maximum_map = current_map[0]
                self.wait = 0
                self.best_weights = self.model.get_weights()
                self.best_epoch = epoch
                speak('current average precision: ' + str(round(self.maximum_map * 100, 2)) + "%")
                model.save(model_save_path)
                model.save_weights(weights_save_path)

            else:
                self.wait += 1
                if self.wait == self.patience:
                    print()
                    speak('Patience reached, restore best model from epoch ' + str(self.best_epoch))
                    print('Patience reached, restore best model from epoch ' + str(self.best_epoch))
                    self.model.stop_training = True
                    self.model.set_weights(self.best_weights)
                else:
                    print('model not improving since epoch ' + str(self.best_epoch))
                    speak('model not improving since epoch ' + str(self.best_epoch))
            print()
        else:
            print('\nmodel not improving since epoch ' + str(self.best_epoch))
            speak('model not improving since epoch ' + str(self.best_epoch))
            # self.wait +=1
            self.all_logs['mAP'].append(0)
            self.all_logs['aIOU'].append(0)
            if self.wait == self.patience:
                print()
                speak('Patience reached, restore best model from epoch ' + str(self.best_epoch))
                print('Patience reached, restore best model from epoch ' + str(self.best_epoch))
                self.model.stop_training = True
                self.model.set_weights(self.best_weights)
map_callback = EvaluationMAP(base_net, test_imgs, obj_threshold=0.025, nms_threshold=0.2, iou_threshold=0.3)
# early_stopping = EarlyStopping(monitor='val_loss', patience=8, verbose=1)<jupyter_output><empty_output><jupyter_text>4.4.1 Train with fit_generator (Encouraged)<jupyter_code>train_gen = base_net.set_generator(imgs=imgs, batch_size=BATCH_SIZE, aug=False)
val_gen = base_net.set_generator(imgs=test_imgs, batch_size=VAL_BATCH_SIZE, aug=False)
# test_gen = base_net.set_generator(imgs=test_imgs, batch_size=VAL_BATCH_SIZE, aug=False)
history = model.fit_generator(generator=train_gen, 
                    steps_per_epoch=len(imgs)//BATCH_SIZE,
                    epochs=EPOCHS,
                    validation_data=val_gen,
                    validation_steps=len(test_imgs)//VAL_BATCH_SIZE,
                    # initial_epoch=2,
                    callbacks=[map_callback]
                    # validation_data=(X_test, y_test), 
                   )
speak('Training Completed.')<jupyter_output>Epoch 1/100
119/120 [============================>.] - ETA: 0s - loss: 0.1164Epoch 1/100
 30/120 [======>.......................] - ETA: 14s - loss: 0.1536
Evaluating...
bee-A 0.0037
bee-B 0.0002
mAP: 0.0019
average IOU: 0.0178

model not improving since epoch 3<jupyter_text>4.4.2 Or train with pure fit function<jupyter_code>model.fit(x=X_train, 
          y=y_train, 
          validation_data=(X_test, y_test), 
          epochs=EPOCHS, 
          batch_size=BATCH_SIZE, 
          callbacks=[map_callback]
          verbose=1)

speak('Training Completed.')<jupyter_output><empty_output><jupyter_text>4.5 All logs for the training session<jupyter_code>map_callback.all_logs<jupyter_output><empty_output><jupyter_text>**5. Evaluate Model**<jupyter_code>base_net.evaluate(model, test_imgs, obj_threshold=0.025, nms_threshold=0.2, iou_threshold=0.1)
# map_callback.average_map<jupyter_output>bee-A 0.6630
bee-B 0.5594
mAP: 0.6112
average IOU: 0.1851<jupyter_text>**6. Test on Images**<jupyter_code>import cv2
from google.colab.patches import cv2_imshow
import numpy as np
image = cv2.imread('frame-315.jpg')
predicted_image = base_net.predict_boxes(model, image, obj_threshold=0.035, nms_threshold=0.4)

cv2_imshow(image)
model.save('Final Model Weights/resnet18-aug-1056-mardi-labelled-twotypes200-bee-model.h5')
model.save_weights('Final Model Weights/resnet18-aug-1056-mardi-labelled-twotypes200-bee-weights.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1399143792495593, 'fraction_numerical': 0.05726517250062956, 'mean_word_length': 3.0129345189975747, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '264794', 'license': 'MIT', 'n_tokens_mistral': 7080, 'n_tokens_neox': 6259, 'n_words': 1170, 'path': 'main.ipynb', 'repo_name': 'shinw97/DL-based-bees-counting-classification-and-analytical-model', 'size': '19855'}