<jupyter_start><jupyter_text>Face Mask SegmentationObjective:In this problem, we use "Transfer Learning" of an Image Segmentation model todetect any object according to the problem in hand.  DatasetFaces in images marked with bounding boxes. Have around 500 images with around 1100 faces manually tagged via bounding box.  Mount Google drive if you are using google colab- We recommend using Google Colab as you can face memory issues and longer runtimes while running on local<jupyter_code>%tensorflow_version 2.x
import tensorflow
tensorflow.__version__
from google.colab import drive
drive.mount('/content/drive')
# Initialize the random number generator
import random
random.seed(0)

# Ignore the warnings
import warnings
warnings.filterwarnings("ignore")
import os
project_path = '/content/drive/My Drive/CNN/Assignment-FaceMask-CV/'<jupyter_output><empty_output><jupyter_text>Load the "images.npy" file - This file contains images with details of bounding boxes<jupyter_code>npy_img = project_path + 'images.npy'
print(npy_img)
import numpy as np
data = np.load(npy_img,allow_pickle = True)
print(data[0][1])
print(data.shape)
IMAGE_WIDTH = 224
IMAGE_HEIGHT = 224<jupyter_output><empty_output><jupyter_text>Create features and labels- Here feature is the image- The label is the mask- Images will be stored in "X_train" array- Masks will be stored in "masks" array<jupyter_code>import cv2
from tensorflow.keras.applications.mobilenet import preprocess_input

masks = np.zeros((int(data.shape[0]), IMAGE_HEIGHT, IMAGE_WIDTH))
X_train = np.zeros((int(data.shape[0]), IMAGE_HEIGHT, IMAGE_WIDTH, 3))
for index in range(data.shape[0]):
    img = data[index][0]
    img = cv2.resize(img, dsize=(IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_CUBIC)
    try:
        img = img[:, :, :3]
    except:
        continue
    X_train[index] = preprocess_input(np.array(img, dtype=np.float32))
    for i in data[index][1]:
        x1 = int(i["points"][0]['x'] * IMAGE_WIDTH)
        x2 = int(i["points"][1]['x'] * IMAGE_WIDTH)
        y1 = int(i["points"][0]['y'] * IMAGE_HEIGHT)
        y2 = int(i["points"][1]['y'] * IMAGE_HEIGHT)
        masks[index][y1:y2, x1:x2] = 1
print(X_train.shape)
print(masks.shape)<jupyter_output>(409, 224, 224)<jupyter_text>Print a sample image and image array<jupyter_code>from matplotlib import pyplot
n = 10
print(X_train[n])
pyplot.imshow(X_train[n])
pyplot.imshow(masks[n])<jupyter_output><empty_output><jupyter_text>Create the model- Add MobileNet as model with below parameter values  - input_shape: IMAGE_HEIGHT, IMAGE_WIDTH, 3  - include_top: False  - alpha: 1.0  - weights: "imagenet"- Add UNET architecture layers  - This is the trickiest part of the project<jupyter_code>from tensorflow.keras.applications.mobilenet import MobileNet
from tensorflow.keras.layers import Concatenate, UpSampling2D, Conv2D, Reshape
from tensorflow.keras.models import Model

ALPHA = 1.0 # Width hyper parameter for MobileNet (0.25, 0.5, 0.75, 1.0). Higher width means more accurate but slower
HEIGHT_CELLS = 28
WIDTH_CELLS = 28
IMAGE_HEIGHT = 224
IMAGE_WIDTH = 224

def create_model(trainable=True):
    model = MobileNet(input_shape=(IMAGE_HEIGHT, IMAGE_WIDTH, 3), include_top=False, alpha=ALPHA, weights="imagenet") # Load pre-trained mobilenet
    for layer in model.layers:
        layer.trainable = trainable

     
    block0 = model.get_layer("conv_pw_1_relu").output
    block1 = model.get_layer("conv_pw_3_relu").output
    block2 = model.get_layer("conv_pw_5_relu").output
    block3 = model.get_layer("conv_pw_11_relu").output
    block4 = model.get_layer("conv_pw_13_relu").output

    x = Concatenate()([UpSampling2D()(block4), block3])
    x = Concatenate()([UpSampling2D()(x), block2])
    x = Concatenate()([UpSampling2D()(x), block1])
    x = Concatenate()([UpSampling2D()(x), block0])
    x = UpSampling2D()(x)
    x = Conv2D(1, kernel_size=1, activation="sigmoid")(x)

    x = Reshape((IMAGE_HEIGHT, IMAGE_HEIGHT))(x)

    return Model(inputs=model.input, outputs=x)<jupyter_output><empty_output><jupyter_text>Call the create_model function<jupyter_code># Give trainable=False as argument, if you want to freeze lower layers for fast training (but low accuracy)
model = create_model(trainable=True)

# Print summary
print(model.summary())<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet/mobilenet_1_0_224_tf_no_top.h5
17227776/17225924 [==============================] - 0s 0us/step
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 225, 225, 3)  0           input_1[0][0]                    
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 112, 112, 32) 864         conv1_pad[0][0]                  
____[...]<jupyter_text>Define dice coefficient function - Create a function to calculate dice coefficient<jupyter_code>def dice_coefficient(y_true, y_pred):
    numerator = 2 * tensorflow.reduce_sum(y_true * y_pred)
    denominator = tensorflow.reduce_sum(y_true + y_pred)
    return numerator / (denominator + tensorflow.keras.backend.epsilon())<jupyter_output><empty_output><jupyter_text>Define loss<jupyter_code>from tensorflow.keras.losses import binary_crossentropy
from tensorflow.keras.backend import log, epsilon
def loss(y_true, y_pred):
    return binary_crossentropy(y_true, y_pred) - log(dice_coefficient(y_true, y_pred) + epsilon())<jupyter_output><empty_output><jupyter_text>Compile the model - Complie the model using below parameters  - loss: use the loss function defined above  - optimizers: use Adam optimizer  - metrics: use dice_coefficient function defined above<jupyter_code>from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import binary_crossentropy

optimizer = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(loss=loss, optimizer=optimizer, metrics=[dice_coefficient])<jupyter_output><empty_output><jupyter_text>Define checkpoint and earlystopping<jupyter_code>from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
checkpoint = ModelCheckpoint("model-{loss:.2f}.h5", monitor="loss", verbose=1, save_best_only=True,
                             save_weights_only=True, mode="min", period=1)
stop = EarlyStopping(monitor="loss", patience=5, mode="min")
reduce_lr = ReduceLROnPlateau(monitor="loss", factor=0.2, patience=5, min_lr=1e-6, verbose=1, mode="min")<jupyter_output>WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.<jupyter_text>Fit the model - Fit the model using below parameters  - epochs: you can decide  - batch_size: 1  - callbacks: checkpoint, reduce_lr, stop<jupyter_code>model.fit(X_train, masks, epochs=50, batch_size=1, verbose=1,callbacks=[checkpoint,reduce_lr,stop])<jupyter_output>Epoch 1/50
409/409 [==============================] - ETA: 0s - loss: 1.3244 - dice_coefficient: 0.4488
Epoch 00001: loss improved from inf to 1.32440, saving model to model-1.32.h5
409/409 [==============================] - 42s 103ms/step - loss: 1.3244 - dice_coefficient: 0.4488 - lr: 1.0000e-04
Epoch 2/50
409/409 [==============================] - ETA: 0s - loss: 0.8113 - dice_coefficient: 0.6090
Epoch 00002: loss improved from 1.32440 to 0.81131, saving model to model-0.81.h5
409/409 [==============================] - 42s 103ms/step - loss: 0.8113 - dice_coefficient: 0.6090 - lr: 1.0000e-04
Epoch 3/50
409/409 [==============================] - ETA: 0s - loss: 0.6315 - dice_coefficient: 0.6740
Epoch 00003: loss improved from 0.81131 to 0.63149, saving model to model-0.63.h5
409/409 [==============================] - 42s 103ms/step - loss: 0.6315 - dice_coefficient: 0.6740 - lr: 1.0000e-04
Epoch 4/50
409/409 [==============================] - ETA: 0s - loss: 0.5628 - dice_coefficient[...]<jupyter_text>Get the predicted mask for a sample image<jupyter_code># original image
n = 10
sample_image = X_train[n]

pyplot.imshow(sample_image)<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).<jupyter_text>Impose the mask on the image<jupyter_code>pred_mask = cv2.resize(1.0*(model.predict(x=np.array([sample_image]))[0] > 0.5), (IMAGE_WIDTH,IMAGE_HEIGHT))
image2 = sample_image
image2[:,:,0] = pred_mask*sample_image[:,:,0]
image2[:,:,1] = pred_mask*sample_image[:,:,1]
image2[:,:,2] = pred_mask*sample_image[:,:,2]

out_image = image2

pyplot.imshow(out_image)
n = 80
sample_image = X_train[n]

pyplot.imshow(sample_image)
pred_mask = cv2.resize(1.0*(model.predict(x=np.array([sample_image]))[0] > 0.5), (IMAGE_WIDTH,IMAGE_HEIGHT))
image2 = sample_image
image2[:,:,0] = pred_mask*sample_image[:,:,0]
image2[:,:,1] = pred_mask*sample_image[:,:,1]
image2[:,:,2] = pred_mask*sample_image[:,:,2]

out_image = image2

pyplot.imshow(out_image)<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).<jupyter_text>Original Image<jupyter_code>n = 400
sample_image = X_train[n]

pyplot.imshow(sample_image)
pred_mask = cv2.resize(1.0*(model.predict(x=np.array([sample_image]))[0] > 0.5), (IMAGE_WIDTH,IMAGE_HEIGHT))<jupyter_output><empty_output><jupyter_text>Image after masking<jupyter_code>image2 = sample_image
image2[:,:,0] = pred_mask*sample_image[:,:,0]
image2[:,:,1] = pred_mask*sample_image[:,:,1]
image2[:,:,2] = pred_mask*sample_image[:,:,2]

out_image = image2

pyplot.imshow(out_image)<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
{'fraction_non_alphanumeric': 0.14868718654734978, 'fraction_numerical': 0.04808732422067066, 'mean_word_length': 5.39622641509434, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '374070', 'license': 'MIT', 'n_tokens_mistral': 3718, 'n_tokens_neox': 3399, 'n_words': 784, 'path': 'Face_mask_segmentation_Solution_Project_CV_AIML_Online.ipynb', 'repo_name': 'Anupama7/Face-Mask-Segmentation', 'size': '10169'}