<jupyter_start><jupyter_text>Machine learning assignment for BITS Pillani M. Tech in Data Science and Engineering  Group 215 **Group members:** - SRINATH.M . - <NAME> - <NAME>   Steps in assignment1. Moving the file to google drive  - Importing the data into colab local environment by unzipping the file  - Doing exploratory data analysis  - Deleting the files that is not needed2. Verifying the image files3. EDA on the files4. Building a deep learning model  - VGG16 with pretrained weights  - Unfreezing only last few layers and adding dense layers to it5. Building a model with fast.ai---  Building three models1. Keras model with VGG16 architecture with frozen weights2. Keras model with VGG16 architecture with unfrozen weights3. Fast AI resnet based architecture Validation is done on the fast AI architecture  Step 1: Google drive to colab connectivityI have put the zip file in the google drive and load the drive on the colab notebook and extract all the files to local notebook environment<jupyter_code>import pandas as pd
import numpy as np

from tensorflow.keras import applications
from tensorflow.keras.layers import Flatten, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import ModelCheckpoint, CSVLogger, TensorBoard, EarlyStopping

import os.path
from google.colab import drive
drive.mount('/content/gdrive')<jupyter_output><empty_output><jupyter_text>!unzip "/content/gdrive/MyDrive/ML/assignment/data/Img/img_align_celeba.zip"<jupyter_code>width, height = 178//2, 218//2
mini_batch_size = 512//2
train_file = "/content/gdrive/MyDrive/ML/assignment/data/Anno/list_attr_celeba.txt"
epoch = 50<jupyter_output><empty_output><jupyter_text>Performing minor cleaning like replacing -1 in labels with 0. And renaming the index which has the file names into the name image<jupyter_code>#Deleting the files which does not have the three attributes
data = pd.read_csv(train_file, delim_whitespace=True, header=1).reset_index()
data = data[["index", "Eyeglasses", "Wearing_Hat", "Mustache"]]
data["Eyeglasses"] = np.where(data["Eyeglasses"] < 0, 0, 1)
data["Wearing_Hat"] = np.where(data["Wearing_Hat"] < 0, 0, 1)
data["Mustache"] = np.where(data["Mustache"] < 0, 0, 1)
data = data.rename(columns = {"index": "image"})
data.head()
#Number of files with label eyeglasses:
print("Number of images with Eyeglasses == 1 is: ", str((data[data['Eyeglasses'] !=0]).size))
print("Number of images with Wearing_Hat == 1 is: ", str((data[data['Wearing_Hat'] !=0]).size))
print("Number of images with Mustache == 1 is: ", str((data[data['Mustache'] !=0]).size))

print("Number of images with no labels = ", str((data[(data['Eyeglasses'] ==0) & (data['Wearing_Hat'] ==0) & (data['Mustache'] ==0)]).size))<jupyter_output><empty_output><jupyter_text>There are 699520 files in the excel annotation file. But in the path there is only about 200,000+ files for the images that is extracted. But there are too many files with the labels that is 0. So we first balance the dataset to have less numebr of images and good class balance so that we will be able to train the mode.For the images with all the three classes == 0, we assign a 0.8 probability to delete the images. We keep only if the probebility falls under 0.2. By this we will eliminate 80% of the images with no target classes present<jupyter_code>import os

paths = ["img_align_celeba/no_label/",
         "img_align_celeba/Mustache/",
         "img_align_celeba/Wearing_Hat/",
         "img_align_celeba/Eyeglasses/",
         "img_align_celeba/all3/"
         ]

for p in paths:
  if not os.path.exists(p):
      os.makedirs(p)<jupyter_output><empty_output><jupyter_text>import osdata_selected = pd.DataFrame()deleted = not_deleted = label_match = 0for i in range(0, len(data)):  if (data["Eyeglasses"].iloc[i] == 0) and (data["Wearing_Hat"].iloc[i] == 0) and (data["Mustache"].iloc[i] == 0):    if np.random.rand() < 0.80:      os.remove("img_align_celeba/" + data["image"].iloc[i])      deleted += 1    else:      not_deleted += 1      os.rename("img_align_celeba/" + data["image"].iloc[i], "img_align_celeba/no_label/" + data["image"].iloc[i])      data_selected = data_selected.append(data.iloc[i])  elif (data["Eyeglasses"].iloc[i] == 0) and (data["Wearing_Hat"].iloc[i] == 0) and (data["Mustache"].iloc[i] == 1):     os.rename("img_align_celeba/" + data["image"].iloc[i], "img_align_celeba/Mustache/" + data["image"].iloc[i])    label_match += 1    data_selected = data_selected.append(data.iloc[i])  elif (data["Eyeglasses"].iloc[i] == 0) and (data["Wearing_Hat"].iloc[i] == 1) and (data["Mustache"].iloc[i] == 0):     os.rename("img_align_celeba/" + data["image"].iloc[i], "img_align_celeba/Wearing_Hat/" + data["image"].iloc[i])    label_match += 1    data_selected = data_selected.append(data.iloc[i])  elif (data["Eyeglasses"].iloc[i] == 1) and (data["Wearing_Hat"].iloc[i] == 0) and (data["Mustache"].iloc[i] == 0):     os.rename("img_align_celeba/" + data["image"].iloc[i], "img_align_celeba/Eyeglasses/" + data["image"].iloc[i])    label_match += 1    data_selected = data_selected.append(data.iloc[i])  elif (data["Eyeglasses"].iloc[i] == 1) and (data["Wearing_Hat"].iloc[i] == 1) and (data["Mustache"].iloc[i] == 1):     os.rename("img_align_celeba/" + data["image"].iloc[i], "img_align_celeba/all3/" + data["image"].iloc[i])    label_match += 1    data_selected = data_selected.append(data.iloc[i])  if i % 2500 == 0: print("Processed: ", i, "Deleted: ", deleted, "Not deleted: ", not_deleted, "Label match: ", label_match)<jupyter_code>data_selected.shape
data_selected.columns<jupyter_output><empty_output><jupyter_text>Displaying random images from all the classes:<jupyter_code>from IPython.display import Image, display
for cols in ['Eyeglasses', 'Mustache', 'Wearing_Hat']:
    print("Checking for " + cols)
    annot_sample_5 = data_selected[data_selected[cols] == 1].sample(5)
    
    for i in range(0, len(annot_sample_5)):
        print(annot_sample_5.iloc[i])
        display(Image('img_align_celeba/' + cols + "/" + annot_sample_5.iloc[i].image))
import cv2
import numpy as np
from sklearn.preprocessing import OneHotEncoder<jupyter_output><empty_output><jupyter_text>First I am going to try to build a model with VGG16 and pretrained weights of imagenet. Freezing all he layers except for the last 2 Convolution layers<jupyter_code>model = applications.VGG16(weights = "imagenet", include_top=False, input_shape = (width, height, 3))

for layer in model.layers[:16]:
    layer.trainable = False

model.summary()<jupyter_output><empty_output><jupyter_text>Adding my own dense layers and then give 3 labels as the output layer.By reference of many documents, for multi class learning, if all the classes are binary, then we can use activation of softmax with binary cross entropy for each of the output to be trained on.<jupyter_code>x = model.output
x = Flatten()(x)
x = Dense(32, activation="relu")(x)
x = Dropout(0.5)(x)
x = Dense(32, activation="relu")(x)
predictions = Dense(3, activation="softmax", name='label')(x)
model_final = Model(inputs = model.input, outputs = predictions)
model_final.summary()
# clipnorm seems to speeds up convergence
#opt = SGD(lr=0.02, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5)
opt = Adam(learning_rate = 0.001)

# the loss calc occurs elsewhere, so use a dummy lambda func for the loss
model_final.compile(loss="binary_crossentropy", optimizer=opt, metrics=['accuracy'])

#model.load_weights("gdrive/My Drive/Colab/Ship/Checkpoints/td.weights.last.hdf5")
import numpy as np

data = data_selected.copy()
#data = pd.read_csv(train_file, delim_whitespace=True, header=1).reset_index()
data = data[["image", "Eyeglasses", "Wearing_Hat", "Mustache"]]
data["Eyeglasses"] = np.where(data["Eyeglasses"] < 0, 0, 1)
data["Wearing_Hat"] = np.where(data["Wearing_Hat"] < 0, 0, 1)
data["Mustache"] = np.where(data["Mustache"] < 0, 0, 1)

training, validation = train_test_split(data, test_size = 0.10)

training = training.reset_index().drop(columns = ["index"])
validation = validation.reset_index().drop(columns = ["index"])

roundofffortraining = (len(training) // mini_batch_size) * mini_batch_size
roundoffforvalidation = (len(training) // mini_batch_size) * mini_batch_size

training = training[:roundofffortraining]
validation = validation[:roundoffforvalidation]

print("Training on ", str(len(training)), " samples")
print("Validating on ", str(len(validation)), " samples")

print(training.head())
import cv2
import numpy as np
from sklearn.preprocessing import OneHotEncoder

#%%
def read_one_image(filename):
    image = cv2.imread("img_align_celeba/" + filename)
    h, w, f = np.shape(image)
    
    image = cv2.resize(image, (width, height), interpolation = cv2.INTER_AREA)
    h, w, f = np.shape(image)
    
    return image


#%%
def generate_arrays_from_file(infile):
    while True:
        for record in range(0, len(infile), mini_batch_size):
            mini_batch = 0
            image_out = np.zeros((mini_batch_size, height, width, 3))
            label_out = np.zeros((mini_batch_size, 3))
                
            while mini_batch < mini_batch_size:
                filename = infile["image"][record]
    
                image_out[mini_batch] = read_one_image(filename)
                label_out[mini_batch] = infile[["Eyeglasses", "Wearing_Hat", "Mustache"]].iloc[record].values
                
                mini_batch = mini_batch + 1
                record = record + 1
                
            inputs = {
                    'input_2': image_out     #Batch Size, h, w, no of channels
                    }
            outputs = {'label': label_out}    #Batch size, 1
            
            yield(inputs, outputs)
            
#%%
def generate_val_arrays_from_file(infile):
    while True:
        for record in range(0, len(infile)):
            image_out = np.zeros((1, height, width, 3))
            label_out = np.zeros((1, 3))      
            filename = infile["image"][record]
            image_out[0] = read_one_image(filename)
            label_out[0] = infile[["Eyeglasses", "Wearing_Hat", "Mustache"]].iloc[record].values
                
            inputs = {
                    'input_2': image_out     #Batch Size, h, w, no of channels
                    }
            outputs = {'label': label_out}    #Batch size, 1
            
            yield(inputs, outputs)            
filepath="gdrive/My Drive/Machine learning/assignment/Checkpoints/weights.{accuracy:.3f}a-{loss:.3f}l.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='accuracy', verbose=1, save_best_only=True, mode='max')

checkpoint_all = ModelCheckpoint("gdrive/My Drive/Machine learning/assignment/Checkpoints/td.weights.last.hdf5", monitor='loss', verbose=1, save_best_only=False)

traininglog = CSVLogger("gdrive/My Drive/Machine learning/assignment/Checkpoints/logs.csv", separator=',', append=True)

tensorboard = TensorBoard(log_dir="gdrive/My Drive/Machine learning/assignment/Checkpoints")

early_stop = EarlyStopping(monitor='loss', patience=5)

callbacks_list = [checkpoint, checkpoint_all, traininglog, tensorboard]

history = model_final.fit(x = generate_arrays_from_file(training),
          steps_per_epoch=len(training) // mini_batch_size, 
          epochs=50, 
          callbacks=callbacks_list,
          validation_data = generate_val_arrays_from_file(validation),
          validation_steps=100
          )
from matplotlib import pyplot as plt
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>--- Model 2 using fast AI<jupyter_code>!pip install scrapbook
!pip install pyyaml h5py
!pip install bqplot
!pip install fastai==1.0.61
!pip install utils_cv
# fastai and torch
import fastai
from fastai.metrics import accuracy
from fastai.vision import (
    models, ImageList, imagenet_stats, partial, cnn_learner, ClassificationInterpretation, to_np,
)
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
import sys
import os
sys.path.insert(0,'/content/gdrive/MyDrive/ML/assignment')
py_file_location = "/content/gdrive/MyDrive/ML/assignment"
sys.path.append(os.path.abspath(py_file_location))


import numpy as np
from pathlib import Path
import scrapbook as sb
%cd /content/gdrive/MyDrive/ML/assignment
%pwd
# local modules
from utils_cv.classification.model import TrainMetricsRecorder
from utils_cv.classification.plot import plot_pr_roc_curves
from utils_cv.classification.widget import ResultsWidget
from utils_cv.classification.data import Urls
from utils_cv.common.data import unzip_url
from utils_cv.common.gpu import db_num_workers, which_processor
print(f"Fast.ai version = {fastai.__version__}")
which_processor()
DATA_PATH     = '/content/img_align_celeba'
EPOCHS        = 5
LEARNING_RATE = 1e-4
IM_SIZE       = 178//2

BATCH_SIZE    = 64
ARCHITECTURE  = models.resnet34
path = Path(DATA_PATH)
path.ls()
data = (
    ImageList.from_folder(path)
    .split_by_rand_pct(valid_pct=0.2, seed=10)
    .label_from_folder()
    .transform(size=IM_SIZE)
    .databunch(bs=BATCH_SIZE, num_workers = db_num_workers())
    .normalize(imagenet_stats)
)
data.show_batch(rows=3, figsize=(15,11))
learn = cnn_learner(
    data,
    ARCHITECTURE,
    metrics=[accuracy],
    callback_fns=[partial(TrainMetricsRecorder, show_graph=True)]
)
learn.fit(EPOCHS, LEARNING_RATE)
learn.save('face recog.h5')
learn.recorder.plot_losses()
_, validation_accuracy = learn.validate(learn.data.valid_dl, metrics=[accuracy])
print(f'Accuracy on validation set: {100*float(validation_accuracy):3.2f}')
interp = ClassificationInterpretation.from_learner(learn)
# Get prediction scores. We convert tensors to numpy array to plot them later.
pred_scores = to_np(interp.preds)
w_results = ResultsWidget(
    dataset=learn.data.valid_ds,
    y_score=pred_scores,
    y_label=[data.classes[x] for x in np.argmax(pred_scores, axis=1)],
)
display(w_results.show())
# True labels of the validation set. We convert to numpy array for plotting.
true_labels = to_np(interp.y_true)
plot_pr_roc_curves(true_labels, pred_scores, data.classes)
interp.plot_confusion_matrix()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12603291438094577, 'fraction_numerical': 0.015485035761405457, 'mean_word_length': 4.520122652357225, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '232966', 'license': 'MIT', 'n_tokens_mistral': 4951, 'n_tokens_neox': 4718, 'n_words': 1170, 'path': '_notebooks/2021-09-15-celebrity-face-cnn.ipynb', 'repo_name': 'GireeshS22/gireeshs22.github.io', 'size': '14419'}