<jupyter_start><jupyter_text>Automated Smile Detection Part 2: Preprocess & ModelingThis second notebook will walk through data preparation and initial modeling. The original dataset was downloaded from this [repo](https://github.com/hromi/SMILEsmileD). It consists of 3,690 images of smiling faces and 9,476 images of non-smiling faces.   Imports<jupyter_code>import os
import warnings

import cv2
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.keras import models, layers, optimizers
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.tools import freeze_graph

%load_ext blackcellmagic

warnings.simplefilter("ignore")
tf.disable_eager_execution()
tf.keras.backend.clear_session()<jupyter_output><empty_output><jupyter_text>EDA  Read in some sample images from smiles and non-smiles.<jupyter_code>pos_imgs = os.listdir("../data/train/positive")
pos_imgs = [os.path.join("../data/train/positive", img) for img in pos_imgs]
pos_subset = pos_imgs[0:9]
pos_subset[0]
neg_imgs = os.listdir("../data/train/negative")
neg_imgs = [os.path.join("../data/train/negative", img) for img in neg_imgs]
neg_subset = neg_imgs[0:9]
neg_subset[0]<jupyter_output><empty_output><jupyter_text>Process Images with OpenCV<jupyter_code>def process_imgs(imgs: list) -> list:
    processed_imgs = [cv2.imread(img) for img in imgs]
    return processed_imgs<jupyter_output><empty_output><jupyter_text>Positive Samples  Below is a 3x3 grid of sample images from the positive training set.<jupyter_code>pos_subset_processed = process_imgs(pos_subset)

fig, axs = plt.subplots(nrows=3, ncols=3)
axs = axs.flatten()

for i, ax in enumerate(axs):
    ax.imshow(pos_subset_processed[i])
    
plt.show()<jupyter_output><empty_output><jupyter_text>Negative Samples  Below is a 3x3 grid of sample images from the negative training set.<jupyter_code>neg_subset_processed = process_imgs(neg_subset)

fig, axs = plt.subplots(nrows=3, ncols=3)
axs = axs.flatten()

for i, ax in enumerate(axs):
    ax.imshow(neg_subset_processed[i])
    
plt.show()<jupyter_output><empty_output><jupyter_text>Image Shape  All of the images are grey-scale and consist of the following shape:<jupyter_code>pos_subset_processed[0].shape<jupyter_output><empty_output><jupyter_text>Based on the shape of the images, I won't have to perform any size reduction during model preprocessing.  Modeling  The first modeling attempt will be made with the keras module embedded in tensforflow. Depending on model performance, AutoML tools may be tested (i.e. GCP AutoML Vision & AWS Sagemaker)<jupyter_code>train_dir = "../data/train"
test_dir = "../data/test"
val_dir = "../data/validation"<jupyter_output><empty_output><jupyter_text>Training & Validation Generators<jupyter_code>train_datagen = ImageDataGenerator(rescale=1.0 / 255)
val_datagen = ImageDataGenerator(rescale=1.0 / 255)

train_generator = train_datagen.flow_from_directory(
    train_dir, batch_size=20, target_size=(150, 150), class_mode="binary"
)

val_generator = val_datagen.flow_from_directory(
    val_dir, batch_size=20, target_size=(150, 150), class_mode="binary"
)<jupyter_output>Found 5380 images belonging to 2 classes.
Found 1000 images belonging to 2 classes.<jupyter_text>Build Model 1  The first model is a very straightforward CNN with 4 convolutional layers, max pooling and two fully connected layers.<jupyter_code>def build_model():
    model = models.Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(150, 150, 3)))
    model.add(layers.MaxPool2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Reshape((6272,)))
    model.add(layers.Dense(512, activation="relu"))
    model.add(layers.Dense(1, activation="sigmoid"))

    model.compile(
        loss="binary_crossentropy",
        optimizer=optimizers.RMSprop(lr=1e-4),
        metrics=["acc"],
    )
    return model
model = build_model()<jupyter_output>WARNING:tensorflow:From /Users/jasonadam/.local/share/virtualenvs/smile-detection-ISm4InNp/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /Users/jasonadam/.local/share/virtualenvs/smile-detection-ISm4InNp/lib/python3.7/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>Model Summary<jupyter_code>model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 148, 148, 32)      896       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 74, 74, 32)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 72, 72, 64)        18496     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 36, 36, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 34, 34, 128)       73856     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 17, 17, 128)       0         
________________________________________________________[...]<jupyter_text>Train Model<jupyter_code>history = model.fit_generator(
    train_generator,
    steps_per_epoch=100,
    epochs=30,
    validation_data=val_generator,
    validation_steps=50,
)
model.save_weights("../models/weights.h5")
model.save("../models/smile_model.h5")
tf.keras.backend.clear_session()
tf.keras.backend.set_learning_phase(0)
model = build_model()
model.load_weights("../models/weights.h5")

save_dir = "frozen_models"

tf.saved_model.simple_save(
    tf.keras.backend.get_session(),
    save_dir,
    inputs={"input": model.inputs[0]},
    outputs={"output": model.outputs[0]},
)

freeze_graph.freeze_graph(
    None,
    None,
    None,
    None,
    model.outputs[0].op.name,
    None,
    None,
    os.path.join(save_dir, "frozen_model.pb"),
    False,
    "",
    input_saved_model_dir=save_dir
);
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Build Model 2  Model two institutes the same architecture as the first mode, but it implements dropout and data augmentation.   Data Augmentation<jupyter_code>train_datagen = ImageDataGenerator(
    rescale=1.0 / 255,
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
)

# Validation generator doesn't need augmentation
val_datagen = ImageDataGenerator(rescale=1.0 / 255)

train_generator = train_datagen.flow_from_directory(
    train_dir, batch_size=20, target_size=(150, 150), class_mode="binary"
)

val_generator = val_datagen.flow_from_directory(
    val_dir, batch_size=20, target_size=(150, 150), class_mode="binary"
)<jupyter_output>Found 5380 images belonging to 2 classes.
Found 1000 images belonging to 2 classes.<jupyter_text>Model with Dropout<jupyter_code>def model_dropout():
    model = models.Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(150, 150, 3)))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Flatten())
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(512, activation="relu"))
    model.add(layers.Dense(1, activation="sigmoid"))
    model.compile(
        loss="binary_crossentropy",
        optimizer=optimizers.RMSprop(lr=1e-4),
        metrics=["acc"],
    )
    return model
model_dropout = model_dropout()
model_dropout.summary()
history_drp = model_dropout.fit_generator(
    train_generator,
    steps_per_epoch=100,
    epochs=50,
    validation_data=val_generator,
    validation_steps=50,
)
acc = history_drp.history['acc']
val_acc = history_drp.history['val_acc']
loss = history_drp.history['loss']
val_loss = history_drp.history['val_loss']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1263316582914573, 'fraction_numerical': 0.03246231155778895, 'mean_word_length': 5.338216560509554, 'pattern_counts': {'":': 2, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '298918', 'license': 'Unlicense', 'n_tokens_mistral': 3512, 'n_tokens_neox': 3271, 'n_words': 695, 'path': 'notebooks/smile_detection_02.ipynb', 'repo_name': 'Jason-Adam/smile-detection', 'size': '9950'}