<jupyter_start><jupyter_text>Importing essential libraries<jupyter_code>import numpy as np
import pandas as pd
import cv2
import math
import matplotlib.pyplot as plt    
# for preprocessing the images
from keras.preprocessing import image   
from keras.utils import np_utils
# for resizing images
from skimage.transform import resize<jupyter_output>Using TensorFlow backend.<jupyter_text>Step 1: Extracting the static images from the video fileWe'll be using a mp4 file of duration 04:58 minutes. When converted to seconds it'll be equivalent to 298 seconds and if we constrict the frame size to 1, we will have 298 images in total.<jupyter_code>count = 0
videoFile = "Tom and jerry.mp4"
videoCapture = cv2.VideoCapture(videoFile)   # capturing the video from the given path
frameRate = videoCapture.get(5) # frame rate
x=1
while(videoCapture.isOpened()):
    frameId = videoCapture.get(1) #current frame number
    ret, frame = videoCapture.read()
    if (ret != True):
        break
    if (frameId % math.floor(frameRate) == 0):
        filename ="frame%d.jpg" % count;count+=1
        cv2.imwrite(filename, frame)
videoCapture.release()<jupyter_output><empty_output><jupyter_text>Step- 2: Label the images to encode them according to TOM/ JERRY's presence in the pic for training.We'll encode the images as 0 for TOM, 1 for JERRY and -1 for neither in the frames. The labelled data is stored in mapping.csv.<jupyter_code>data = pd.read_csv('mapping.csv') 
data.head()<jupyter_output><empty_output><jupyter_text>You can view whether the mapping is correct or not by viewing the corresponding frame.<jupyter_code>img = cv2.imread('frame2.jpg')
plt.imshow(img)
plt.title('label: 0') # Zero indicates TOM should be in the image
img = cv2.imread('frame0.jpg')
plt.imshow(img)
plt.title('label: 1') # Zero indicates TOM should be in the imagee<jupyter_output><empty_output><jupyter_text>Step 3: Convert and stack the images as numpy arraysFor input to the neural nets, we convert the '.jpg' images to numpy arrays and one- hot encoding the classes. Further we reduce the size of images so as to have their size equivalent to the one we are supposed to provide as input to the pretrained networks.<jupyter_code>images = []
for image in data.Image_ID:
    img = cv2.imread(image)
    X = np.array(img)
    images.append(X)
images = np.array(images)
# one hot encoding Classes
y = np_utils.to_categorical(data.Class, num_classes = 3)<jupyter_output><empty_output><jupyter_text>Pretrained models are a wonderful source of help for people looking to learn an algorithm or try out an existing framework. Due to time restrictions or computational restraints, it’s not always possible to build a model from scratch which is why pretrained models exist! You can use a pretrained model as a benchmark to either improve the existing model, or test your own model against it. The potential and possibilities are vast.<jupyter_code># Reducing the size of images to 224*224*3
img = []
for i in range(0,images.shape[0]):
    a = resize(images[i], preserve_range=True, output_shape=(224,224)).astype(int)      
    img.append(a)
X = np.array(img)
from keras.applications.vgg16 import preprocess_input
X = preprocess_input(X, mode='tf')      # preprocessing the input data<jupyter_output><empty_output><jupyter_text>Step 5: Creating training, validation and test sets.We also need a validation set to check the performance of the model on unseen images. We will make use of the train_test_split() function of the sklearn.model_selection module to randomly divide images into training and validation set.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_eval, y_train, y_eval = train_test_split(X, y, test_size=0.3, random_state=42)<jupyter_output><empty_output><jupyter_text>Step 6: Building the modelsWe will be using the predictions from the pretrained VGG16 model as input to the base CNN which we'll develop later<jupyter_code>from keras.models import Sequential
from keras.applications.vgg16 import VGG16
from keras.layers import Dense, InputLayer, Dropout<jupyter_output><empty_output><jupyter_text>VGG16 model for prediction over Training and Validation sets<jupyter_code>base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))    
X_train = base_model.predict(X_train)
X_eval = base_model.predict(X_eval)

# The shapes of output are (208,7,7,512)
X_train.shape, X_eval.shape
# Reshaping to 1-D array
X_train = X_train.reshape(208, -1)      
X_eval = X_eval.reshape(90,-1)
# Centering the data
train = X_train/X_train.max()     
X_eval = X_eval/X_train.max()<jupyter_output><empty_output><jupyter_text>Training the base CNN<jupyter_code>net = Sequential()
net.add(InputLayer((7*7*512,)))    # input layer
net.add(Dense(units=1024, activation='sigmoid')) # hidden layer
net.add(Dense(3, activation='softmax'))    # output layer
net.summary()
# O/P has 3 neurons because we've 3 classes to predict
# Setting the optimizer, type of loss and metric for assessment of the model
net.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Training the model
net.fit(train, y_train, epochs=200, validation_data=(X_eval, y_eval))<jupyter_output>Train on 208 samples, validate on 90 samples
Epoch 1/200
208/208 [==============================] - 0s 860us/step - loss: 0.0070 - acc: 0.9952 - val_loss: 0.7095 - val_acc: 0.7111
Epoch 2/200
208/208 [==============================] - 0s 818us/step - loss: 0.0070 - acc: 0.9952 - val_loss: 0.6899 - val_acc: 0.7778
Epoch 3/200
208/208 [==============================] - 0s 817us/step - loss: 0.0068 - acc: 0.9952 - val_loss: 0.6847 - val_acc: 0.8444
Epoch 4/200
208/208 [==============================] - 0s 796us/step - loss: 0.0075 - acc: 0.9904 - val_loss: 0.6666 - val_acc: 0.8778
Epoch 5/200
208/208 [==============================] - 0s 792us/step - loss: 0.0076 - acc: 0.9904 - val_loss: 0.7077 - val_acc: 0.7222
Epoch 6/200
208/208 [==============================] - 0s 795us/step - loss: 0.0070 - acc: 0.9952 - val_loss: 0.7058 - val_acc: 0.7333
Epoch 7/200
208/208 [==============================] - 0s 781us/step - loss: 0.0072 - acc: 0.9952 - val_loss: 0.7117 - val_acc: 0.7111
Epoch 8/20[...]<jupyter_text>At the end of 200 epochs we could acheive a validation accuracy of 85.66%  Step 7: EVALUATING THE MODEL OVER TEST DATAFor this phase, we once again will extract the frames and preprocess the images obtained from the test .mp4 file before we evaluate it using the net. We have a subpar performance on the test set (probably due to the lack of training data)<jupyter_code># Similar to earlier extract the preprocess the video file and corresponding images
count = 0
videoFile = "Tom and Jerry 2.mp4"
cap = cv2.VideoCapture(videoFile)
frameRate = cap.get(5) #frame rate
x=1
while(cap.isOpened()):
    frameId = cap.get(1) #current frame number
    ret, frame = cap.read()
    if (ret != True):
        break
    if (frameId % math.floor(frameRate) == 0):
        filename ="test%d.jpg" % count;count+=1
        cv2.imwrite(filename, frame)
cap.release()

test = pd.read_csv('test.csv')
test.head()
test_images = []
for img_name in test.Image_ID:
    img = plt.imread('' + img_name)
    test_images.append(img)
test_images = np.array(test_images)

test_img = []
for i in range(0,test_images.shape[0]):
    a = resize(test_images[i], preserve_range=True, output_shape=(224,224)).astype(int)
    test_img.append(a)
test_img = np.array(test_img)
# preprocessing the images
test_img = preprocess_input(test_img, mode='tf')

# extracting features from the images using VGG16
test_img = base_model.predict(test_img)

# converting the images to 1-D array
test_img = test_img.reshape(186, -1)

# zero centered images
test_img = test_img/test_img.max()
test_preds= net.predict_classes(test_img)
print("The screen time of JERRY is ", test_preds[test_preds==1].shape[0], "seconds")
print("The screen time of TOM is ", test_preds[test_preds==0].shape[0], "seconds")<jupyter_output>The screen time of JERRY is 5 seconds
The screen time of TOM is 116 seconds
{'fraction_non_alphanumeric': 0.12332672285572632, 'fraction_numerical': 0.04895884977689638, 'mean_word_length': 4.928728875826598, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '298579', 'license': 'MIT', 'n_tokens_mistral': 2905, 'n_tokens_neox': 2599, 'n_words': 866, 'path': 'video_data_analysis_using_CNN_with_Keras.ipynb', 'repo_name': 'Basileus1995/Introduction-to-Video-Data-Analysis-using-CNNs', 'size': '8068'}