<jupyter_start><jupyter_text>Assignment 7: Image Processing With a CNN  This assignment follows the same structure as Assignment 6. We will employ at least a 2x2 completely crossed experimental design. We will again use a simple training-and-test regimen. The factors in the design may include numbers of layers and/or nodes within layers, types of layers (convolutional or pooling), and/or other hyperparameters. You will utilize convolutional neural networks (CNNs) within Python TensorFlow. This week, you will compete in the Dogs vs. Cats kernels Edition Kaggle.com competition, https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition (Links to an external site.). Specifically, you will build models using the training set to forecast the test set. The images are in .jpg format, so you will need to research how to handle that.  You are required to submit no fewer than four models for evaluation by Kaggle.com, and you must provide your Kaggle.com scores and user ID for validation.   Management Problem  Assume that we are providing advice to a website provider who is looking for tools to automatically label images provided by end users. As we look across the factors in the study, making recommendations to management about image classification, we are most concerned about achieving the highest possible accuracy in image classification. That is, we should be willing to sacrifice training time for model accuracy. What type of machine learning model works best? If it is a convolutional neural network, what type of network should we use? Part of this recommendation may concern information about the initial images themselves (input data for the classification task). What types of images work best?  Setup  & Data Ingest  Initial Imports<jupyter_code># General purpose libraries
import json
import os, cv2, re, random
import numpy as np
import pandas as pd

# Keras
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing.image import img_to_array, load_img
from keras import layers, models, optimizers
from keras import backend as K

# Scikit-learn
from sklearn.model_selection import train_test_split<jupyter_output>Using TensorFlow backend.<jupyter_text>Load Data<jupyter_code># Create directory for images
!mkdir ~/.kaggle
!mkdir /content/.kaggle

# User token
token = {"username":"YourUsername","key":"YourKey"}

# Save token
with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(token, file)
!cp /content/.kaggle/kaggle.json ~/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
!kaggle config set -n path -v{/content}
# Pull the images from the kaggle api
!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition -p /content
!unzip -q \*.zip<jupyter_output>Downloading test.zip to /content
 92% 250M/271M [00:01<00:00, 186MB/s]
100% 271M/271M [00:01<00:00, 198MB/s]
Downloading train.zip to /content
 98% 531M/544M [00:04<00:00, 64.0MB/s]
100% 544M/544M [00:04<00:00, 127MB/s] 
Downloading sample_submission.csv to /content
  0% 0.00/111k [00:00<?, ?B/s]
100% 111k/111k [00:00<00:00, 108MB/s]

2 archives were successfully processed.<jupyter_text>Data Cleaning<jupyter_code># Create Train & test sets
TRAIN_DIR = '/content/train/'
TEST_DIR = '/content/test/'

#img_width = 150
#img_height = 150

img_width = 224
img_height = 224

train_images_dogs_cats = [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR)] # use this for full dataset
test_images_dogs_cats = [TEST_DIR+i for i in os.listdir(TEST_DIR)]
# Helper functions
def atoi(text):
    return int(text) if text.isdigit() else text

def natural_keys(text):
    return [ atoi(c) for c in re.split('(\d+)', text) ]
# Reduce the training size to avoid out of memory error
train_images_dogs_cats.sort(key=natural_keys)
train_images_dogs_cats = train_images_dogs_cats[0:1300] + train_images_dogs_cats[12500:13800] 

test_images_dogs_cats.sort(key=natural_keys)
# Data prep function
def prepare_data(list_of_images):
    """
    Returns two arrays: 
        x is an array of resized images
        y is an array of labels
    """
    x = [] # images as arrays
    y = [] # labels
    
    for image in list_of_images:
        x.append(cv2.resize(cv2.imread(image), (img_width,img_height), interpolation=cv2.INTER_CUBIC))
    
    for i in list_of_images:
        if 'dog' in i:
            y.append(1)
        elif 'cat' in i:
            y.append(0)
        #else:
            #print('neither cat nor dog name present in images')
            
    return x, y
# Prep the data
X, Y = prepare_data(train_images_dogs_cats)

# Check backend format for how tensors needs to be fed into model
print(K.image_data_format())<jupyter_output>channels_last<jupyter_text>Train / Validation Split<jupyter_code># First split the data in two sets, 80% for training, 20% for Val/Test)
X_train, X_val, Y_train, Y_val = train_test_split(X,Y, test_size=0.2, random_state=1)<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code># Review tensor shape of traning data
np.array(X).shape
# Plot a few images
def show_cats_and_dogs(idx):
  picture = X_train[idx]
  plt.figure(figsize=(10,5))
  plt.imshow(picture)
  plt.show()
    
for idx in range(0,3):
    show_cats_and_dogs(idx)<jupyter_output><empty_output><jupyter_text>Modeling  Convolutional Neural Network (CNN) 1:  Training<jupyter_code># Create variables for models
nb_train_samples = len(X_train)
nb_validation_samples = len(X_val)
batch_size = 10
# Build layers for the CNN

model = models.Sequential()

model.add(layers.Conv2D(32, (3, 3), input_shape=(img_width, img_height, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Conv2D(32, (3, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Conv2D(64, (3, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Flatten())
model.add(layers.Dense(64))
model.add(layers.Activation('relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1))
model.add(layers.Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

model.summary()
# Image pre-processing; ImageDataGenerator will incrementall process images and
# feed them to the model to avoid out of memory errors
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

val_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)
# Create generators to pass to model
train_generator = train_datagen.flow(np.array(X_train), Y_train, batch_size=batch_size)
validation_generator = val_datagen.flow(np.array(X_val), Y_val, batch_size=batch_size)
# Run the model (10 epochs)
history = model.fit_generator(
    train_generator, 
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=10,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size
)<jupyter_output>Epoch 1/10
208/208 [==============================] - 19s 91ms/step - loss: 0.5093 - acc: 0.7707 - val_loss: 0.5456 - val_acc: 0.7414
Epoch 2/10
208/208 [==============================] - 17s 80ms/step - loss: 0.4999 - acc: 0.7674 - val_loss: 0.5756 - val_acc: 0.7330
Epoch 3/10
208/208 [==============================] - 17s 80ms/step - loss: 0.4745 - acc: 0.7924 - val_loss: 0.5407 - val_acc: 0.7194
Epoch 4/10
208/208 [==============================] - 17s 83ms/step - loss: 0.4697 - acc: 0.7924 - val_loss: 0.5520 - val_acc: 0.7255
Epoch 5/10
208/208 [==============================] - 19s 90ms/step - loss: 0.4572 - acc: 0.7984 - val_loss: 0.5470 - val_acc: 0.7500
Epoch 6/10
208/208 [==============================] - 18s 85ms/step - loss: 0.4439 - acc: 0.7939 - val_loss: 0.7838 - val_acc: 0.7145
Epoch 7/10
208/208 [==============================] - 17s 81ms/step - loss: 0.4375 - acc: 0.8041 - val_loss: 0.4955 - val_acc: 0.7840
Epoch 8/10
208/208 [==============================] - 17s 81ms[...]<jupyter_text>Testing<jupyter_code># Mount google drive and write the dataframe to csv
from google.colab import drive
drive.mount('/content/gdrive')
# Write model and weights to files in case need runtime crashes
model.save_weights('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_one_weights.h5')
model.save('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_one_keras.h5')
# Prep test data
X_test, Y_test = prepare_data(test_images_dogs_cats)
# Create generator for test data; only rescale test data
test_datagen = ImageDataGenerator(rescale=1. / 255)
# Instantiate the generator and pass iterative test data to model for predictions
# output of predictions is a probability that the image is a dog
test_generator = test_datagen.flow(np.array(X_test), batch_size=batch_size)
prediction_probabilities = model.predict_generator(test_generator, verbose=1, steps=len(X_test) // batch_size)
# Create file for submission
counter = range(1, len(test_images_dogs_cats) + 1)
solution = pd.DataFrame({"id": counter, "label":list(prediction_probabilities)})
cols = ['label']

for col in cols:
    solution[col] = solution[col].map(lambda x: str(x).lstrip('[').rstrip(']')).astype(float)

solution.to_csv("/content/gdrive/My Drive/northwestern/422-msds/week-07/dogsVScats_one.csv", index = False)<jupyter_output><empty_output><jupyter_text>**Kaggle Score**  Log Loss = 1.90651  Convolutional Neural Network (CNN) 2:  Training<jupyter_code># Build layers for the CNN

model_two = models.Sequential()

model_two.add(layers.Conv2D(32, (3, 3), input_shape=(img_width, img_height, 3)))
model_two.add(layers.Activation('relu'))
model_two.add(layers.MaxPooling2D(pool_size=(2, 2)))

model_two.add(layers.Conv2D(32, (3, 3)))
model_two.add(layers.Activation('relu'))
model_two.add(layers.MaxPooling2D(pool_size=(2, 2)))

model_two.add(layers.Conv2D(64, (3, 3)))
model_two.add(layers.Activation('relu'))
model_two.add(layers.MaxPooling2D(pool_size=(2, 2)))

model_two.add(layers.Conv2D(64, (3, 3)))
model_two.add(layers.Activation('relu'))
model_two.add(layers.MaxPooling2D(pool_size=(2, 2)))

model_two.add(layers.Conv2D(128, (3, 3)))
model_two.add(layers.Activation('relu'))
model_two.add(layers.MaxPooling2D(pool_size=(2, 2)))

model_two.add(layers.Flatten())
model_two.add(layers.Dense(128))
model_two.add(layers.Activation('relu'))
model_two.add(layers.Dropout(0.5))

model_two.add(layers.Dense(128))
model_two.add(layers.Activation('relu'))
model_two.add(layers.Dropout(0.5))

model_two.add(layers.Dense(2))
model_two.add(layers.Activation('sigmoid'))

model_two.compile(loss='binary_crossentropy',optimizer='rmsprop',metrics=['accuracy'])

model_two.summary()
# Run the model (10 epochs)
history_two = model_two.fit_generator(
    train_generator, 
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=10,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size
)
# Write model and weights to files in case need runtime crashes
model.save_weights('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_two_weights.h5')
model.save('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_two_keras.h5')<jupyter_output><empty_output><jupyter_text>Testing<jupyter_code>prediction_probabilities = model_two.predict_generator(test_generator, verbose=1, steps=len(X_test) // batch_size)
solution = pd.DataFrame({"id": counter, "label":list(prediction_probabilities)})
cols = ['label']

for col in cols:
    solution[col] = solution[col].map(lambda x: str(x).lstrip('[').rstrip(']')).astype(float)

solution.to_csv("/content/gdrive/My Drive/northwestern/422-msds/week-07/dogsVScats_two.csv", index = False)<jupyter_output><empty_output><jupyter_text>**Kaggle Score**  Log Loss = 1.26014  Convolutional Neural Network (CNN) 3:  Training<jupyter_code># Run the model (50 epochs)
history = model.fit_generator(
    train_generator, 
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size
)
# Write model and weights to files in case need runtime crashes
model.save_weights('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_three_weights.h5')
model.save('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_three_keras.h5')<jupyter_output><empty_output><jupyter_text>Testing<jupyter_code>prediction_probabilities = model.predict_generator(test_generator, verbose=1, steps=len(X_test) // batch_size)
solution = pd.DataFrame({"id": counter, "label":list(prediction_probabilities)})
cols = ['label']

for col in cols:
    solution[col] = solution[col].map(lambda x: str(x).lstrip('[').rstrip(']')).astype(float)

solution.to_csv("/content/gdrive/My Drive/northwestern/422-msds/week-07/dogsVScats_three.csv", index = False)<jupyter_output><empty_output><jupyter_text>**Kaggle Score**  Log Loss = 3.63742  *The model overfit the training data. Having more epochs isn't necessarily a good thing.  We can try adjusting the learning rate and use a different optimizer.*  Convolutional Neural Network (CNN) 4:  Training<jupyter_code># Build layers for the CNN

optimizer = optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)

model = models.Sequential()

model.add(layers.Conv2D(32, (3, 3), input_shape=(img_width, img_height, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Conv2D(32, (3, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Conv2D(64, (3, 3)))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D(pool_size=(2, 2)))

model.add(layers.Flatten())
model.add(layers.Dense(64))
model.add(layers.Activation('relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1))
model.add(layers.Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

model.summary()
# Run the model (20 epochs)
history = model.fit_generator(
    train_generator, 
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size
)
# Write model and weights to files in case need runtime crashes
model.save_weights('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_four_weights.h5')
model.save('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_four_keras.h5')<jupyter_output><empty_output><jupyter_text>Testing<jupyter_code># Load model after memory error
from keras.models import load_model
# Load model from file (due to memory error)
model = load_model('/content/gdrive/My Drive/northwestern/422-msds/week-07/model_four_keras.h5')
# Instantiate the generator and pass iterative test data to model for predictions
# output of predictions is a probability that the image is a dog
test_generator = test_datagen.flow(np.array(X_test), batch_size=batch_size)
prediction_probabilities = model.predict_generator(test_generator, verbose=1, steps=len(X_test) // batch_size)
counter = range(1, len(test_images_dogs_cats) + 1)
solution = pd.DataFrame({"id": counter, "label":list(prediction_probabilities)})
cols = ['label']

for col in cols:
    solution[col] = solution[col].map(lambda x: str(x).lstrip('[').rstrip(']')).astype(float)

solution.to_csv("/content/gdrive/My Drive/northwestern/422-msds/week-07/dogsVScats_four.csv", index = False)<jupyter_output><empty_output><jupyter_text>**Kaggle Score**  Log Loss = 2.51124<jupyter_code>from keras.utils import plot_model
import matplotlib.pyplot as plt
plot_model(model, to_file="/content/gdrive/My Drive/northwestern/422-msds/week-07/model_four.png")<jupyter_output><empty_output><jupyter_text>Summary<jupyter_code># Create summary dataframe of parameters
summary_df = pd.DataFrame(
    {
        "Model": [1, 2, 3, 4],
        "Optimizer": ["RMSProp", "RMSProp", "RMSProp", "SGD"],
        "ConvolutionalLayers": [3, 5, 3, 3],
        "PoolingLayers": [3, 5, 3, 3],
        "DenseLayers": [2, 3, 2, 2],
        "Epochs": [10,10,50,50],
        "TrainingAccuracy": [0.8062, 0.7392, 0.8531, 0.9356],
        "ValidationAccuracy": [0.7598, 0.7476, 0.7718, 0.7635],
        "LogLossTestKaggle": [1.90651, 1.26014, 3.63742, 2.51124]
    }
)

summary_df
import seaborn as sns
sns.barplot(summary_df["Model"], summary_df["LogLossTestKaggle"]).set_title("Log Loss by Model")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13826891671203048, 'fraction_numerical': 0.04651303453698663, 'mean_word_length': 5.322753346080306, 'pattern_counts': {'":': 19, '<': 63, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '430031', 'license': 'Unlicense', 'n_tokens_mistral': 6267, 'n_tokens_neox': 5693, 'n_words': 1335, 'path': 'homework_assignment_07.ipynb', 'repo_name': 'Jason-Adam/msds-422', 'size': '16533'}