<jupyter_start><jupyter_text>12 - Building & Training Convolutional Neural Network PrefaceNote that this same notebook crashed my laptop when I tried to train my CNN, so I'm migrating this onto AWS. This notebook's code is a similar copy of the previous notebook in this series except for this preface and the section after the model has been successfully trained.<jupyter_code># Install tflearn
import os
os.system("sudo pip install tflearn tqdm boto3 opencv-python")<jupyter_output><empty_output><jupyter_text>Feature Building<jupyter_code>import cv2
import numpy as np
import pandas as pd
import urllib
import math
import boto3
import os
import copy
from tqdm import tqdm
from matplotlib import pyplot as plt
%matplotlib inline
# Connect to s3 bucket
s3 = boto3.resource('s3', region_name = 'ca-central-1')
my_bucket = s3.Bucket('2017edmfasatb')
# Get all files in the project directory under chi_lars_face_detection/photos/
chi_photos = [i.key for i in my_bucket.objects.all() if 'chi_lars_face_detection/photos/chi/' in i.key]
lars_photos = [i.key for i in my_bucket.objects.all() if 'chi_lars_face_detection/photos/lars/' in i.key]
# Define function to convert URL to numpy array
def url_to_image(url):
    # Download the image, convert it to a numpy array, and then read it into OpenCV format
    resp = urllib.urlopen(url)
    image = np.asarray(bytearray(resp.read()), dtype="uint8")
    image = cv2.imdecode(image, cv2.IMREAD_GRAYSCALE)
    
    # Rotate image
    image = np.rot90(image, 3)
    
    # Build resize into function
    image = cv2.resize(image, (0,0), fx=0.03, fy=0.03)

    # Return the image
    return image
# Loop through all files to download into a single array from AWS
url_prefix = 'https://s3.ca-central-1.amazonaws.com/2017edmfasatb'
# Trying out the new tqdm library for progress bar
chi_photos_list = [url_to_image(os.path.join(url_prefix, x)) for x in tqdm(chi_photos)]
# Trying out the new tqdm library for progress bar
lars_photos_list = [url_to_image(os.path.join(url_prefix, x)) for x in tqdm(lars_photos)]
# Convert to numpy arrays
chi_photos_np = np.array(chi_photos_list)
lars_photos_np = np.array(lars_photos_list)
# Temporarily save np arrays
np.save('chi_photos_np_0.03_compress', chi_photos_np)
np.save('lars_photos_np_0.03_compress', lars_photos_np)
# Temporarily load from np arrays
chi_photos_np = np.load('chi_photos_np_0.03_compress.npy')
lars_photos_np = np.load('lars_photos_np_0.03_compress.npy')
# View shape of numpy array
chi_photos_np.shape
# Set width var
width = chi_photos_np.shape[-1]
width<jupyter_output><empty_output><jupyter_text>Scaling Inputs<jupyter_code># Try out scaler on a manually set data (min of 0, max of 255)
from sklearn.preprocessing import MinMaxScaler
# Set test data list to train on (min of 0, max of 255)
test_list = np.array([0, 255]).reshape(-1, 1)
test_list
# Initialize scaler
scaler = MinMaxScaler()
# Fit test list
scaler.fit(test_list)<jupyter_output>/Users/chiwang/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.py:420: DataConversionWarning: Data with input dtype int64 was converted to float64 by MinMaxScaler.
  warnings.warn(msg, DataConversionWarning)<jupyter_text>Reshaping 3D Array To 4D Array<jupyter_code>chi_photos_np.reshape(-1, width, width, 1).shape<jupyter_output><empty_output><jupyter_text>Putting It All Together<jupyter_code># Reshape to prepare for scaler
chi_photos_np_flat = chi_photos_np.reshape(1, -1)
chi_photos_np_flat[:10]
# Scale
chi_photos_np_scaled = scaler.transform(chi_photos_np_flat)
chi_photos_np_scaled[:10]
# Reshape to prepare for scaler
lars_photos_np_flat = lars_photos_np.reshape(1, -1)
lars_photos_np_scaled = scaler.transform(lars_photos_np_flat)<jupyter_output><empty_output><jupyter_text>Now let's reshape.<jupyter_code># Reshape
chi_photos_reshaped = chi_photos_np_scaled.reshape(-1, width, width, 1)
lars_photos_reshaped = lars_photos_np_scaled.reshape(-1, width, width, 1)

print('{} has shape: {}'. format('chi_photos_reshaped', chi_photos_reshaped.shape))
print('{} has shape: {}'. format('lars_photos_reshaped', lars_photos_reshaped.shape))
# Create copy of chi's photos to start populating x_input
x_input = copy.deepcopy(chi_photos_reshaped)

print('{} has shape: {}'. format('x_input', x_input.shape))
# Concatentate lars' photos to existing x_input
x_input = np.append(x_input, lars_photos_reshaped, axis = 0)

print('{} has shape: {}'. format('x_input', x_input.shape))<jupyter_output>x_input has shape: (403, 91, 91, 1)<jupyter_text>Preparing Labels<jupyter_code># Create label arrays
y_chi = np.array([[1, 0] for i in chi_photos_reshaped])
y_lars = np.array([[0, 1] for i in lars_photos_reshaped])

print('{} has shape: {}'. format('y_chi', y_chi.shape))
print('{} has shape: {}'. format('y_lars', y_lars.shape))
# Preview the first few elements
y_chi[:5]
y_lars[:5]
# Create copy of chi's labels to start populating y_input
y_input = copy.deepcopy(y_chi)

print('{} has shape: {}'. format('y_input', y_input.shape))
# Concatentate lars' labels to existing y_input
y_input = np.append(y_input, y_lars, axis = 0)

print('{} has shape: {}'. format('y_input', y_input.shape))<jupyter_output>y_input has shape: (403, 2)<jupyter_text>TrainingI'm going to just copy and paste the CNN structure I used for the MNIST tutorial and see what happens. I'm running this on my own laptop by the way, let's observe the speed.<jupyter_code># TFlearn libraries
import tflearn
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.estimator import regression
# sentdex's code to build the neural net using tflearn
#   Input layer --> conv layer w/ max pooling --> conv layer w/ max pooling --> fully connected layer --> output layer
convnet = input_data(shape = [None, width, width, 1], name = 'input')

convnet = conv_2d(convnet, 32, 10, activation = 'relu')
convnet = max_pool_2d(convnet, 2)

convnet = conv_2d(convnet, 64, 10, activation = 'relu')
convnet = max_pool_2d(convnet, 2)

convnet = fully_connected(convnet, 1024, activation = 'relu')
convnet = dropout(convnet, 0.8)

convnet = fully_connected(convnet, 2, activation = 'softmax')
convnet = regression(convnet, optimizer = 'sgd', learning_rate = 0.01, loss = 'categorical_crossentropy', name = 'targets')<jupyter_output><empty_output><jupyter_text>Train Test SplitI'm just going to do a 90 / 10 train test split here- My training data will consist of roughly 360 training images- My test data will consist of roughly 40 test images<jupyter_code># Import library
from sklearn.cross_validation import train_test_split
print(x_input.shape)
print(y_input.shape)
# Perform train test split
x_train, x_test, y_train, y_test = train_test_split(x_input, y_input, test_size = 0.1, stratify = y_input)<jupyter_output><empty_output><jupyter_text>TrainingLet's try training with 3 epochs.<jupyter_code># Train with data
model = tflearn.DNN(convnet)
model.fit(
    {'input': x_train},
    {'targets': y_train},
    n_epoch = 3,
    validation_set = ({'input': x_test}, {'targets': y_test}),
    snapshot_step = 500,
    show_metric = True
)
# Save model
model.save('model_4_epochs_0.03_compression_99.6.tflearn')<jupyter_output>INFO:tensorflow:/Users/chiwang/Documents/Projects/Dev/chi_lars_face_detection/notebook/model_4_epochs_0.03_compression_99.6.tflearn is not in all_model_checkpoint_paths. Manually adding it.<jupyter_text>TestingOkay, so that was quite the wild ride. I've gotten something to work right now and it's giving me _**99.99% accuracy**_. (0.0147 loss). The loss is cross entropy (measuring node purity, something to the tune of $D=-\sum_{k=1}^{K}\hat{p}_{mk}log{\hat{p}_{mk}}$), so that seems like quite a good loss value to have. Let's try to predict on our test set and generate a simple confusion matrix just to make sure we're sane.<jupyter_code># Predict on test set, generating probabilities of each class (one-hot style)
y_pred_proba = model.predict(x_test)
y_pred_proba[:10]
# Convert probabilities to direct predictions
y_pred_labels = np.array(['chi' if y[0] >= 0.5 else 'lars' for y in y_pred_proba])
y_pred_labels[:10]
# Convert y_test to direct predictions
y_test_labels = np.array(['chi' if y[0] >= 0.5 else 'lars' for y in y_test])
y_test_labels[:10]<jupyter_output><empty_output><jupyter_text>It looks like it's gotten the first 10 right so far... _**creepy**_. Let's make a confusion matrix.<jupyter_code># Confusion matrix
from sklearn.metrics import confusion_matrix
confusion_matrix(y_test_labels, y_pred_labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.105950991831972, 'fraction_numerical': 0.0220536756126021, 'mean_word_length': 5.8954143201930815, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '262221', 'license': 'Apache-2.0', 'n_tokens_mistral': 3164, 'n_tokens_neox': 2953, 'n_words': 855, 'path': 'notebook/12 - Building & Training Convolutional Neural Network (AWS).ipynb', 'repo_name': 'strikingmoose/chi_lars_face_detection', 'size': '8570'}