<jupyter_start><jupyter_text>Final Project DL model Notebook - sinple CNN model with two convolutional layers - v01  imports, settings and constants run log: - AWS p2- full data set- batch_size = 64- epochs = 15- learning_rate = 1e-3- augmented data enabled- dynamic lr reduce enabled- early stopping enabled<jupyter_code>import sys

from keras.models import Sequential
from keras.layers import BatchNormalization, Flatten, Dense, Conv2D, MaxPooling2D
from keras.optimizers import Adam
import numpy as np

from pathlib import Path

sys.path.append(str(Path.cwd().parent))
from utils import *

current_dir = Path.cwd()
HOME_DIR = current_dir.parent
DATA_DIR = HOME_DIR.joinpath('data')

# comment out one of the two path options to toggle between sample directory and all data
path = DATA_DIR
# path = DATA_DIR.joinpath('sample')
train_path = path.joinpath('train')
val_path = path.joinpath('valid')
test_path = path.joinpath('test')
results_path = path.joinpath('tesults')

# training variables
batch_size = 64
epochs = 15
learning_rate = 1e-3<jupyter_output>Using Theano backend.
Using cuDNN version 5103 on context None
Preallocating 10867/11439 Mb (0.950000) on cuda
Mapped name None to device cuda: Tesla K80 (0000:00:1E.0)<jupyter_text>getting training and validation data in batches<jupyter_code>batches = get_in_batches(train_path, batch_size=batch_size, augment=True)
val_batches = get_in_batches(val_path, batch_size=batch_size, augment=True)
test_batches = get_in_batches(test_path, batch_size=batch_size)<jupyter_output>Found 22424 images belonging to 10 classes.
Found 5000 images belonging to 10 classes.
Found 79726 images belonging to 1 classes.<jupyter_text>and getting the classes, labels and filenames for each batch<jupyter_code>trn_classes = batches.classes
val_classes = val_batches.classes
trn_labels = onehot(batches.classes)
val_labels = onehot(val_batches.classes)
trn_filenames = batches.filenames
val_filenames = val_batches.filenames<jupyter_output><empty_output><jupyter_text>defining CNN model<jupyter_code>model = Sequential([
            BatchNormalization(axis=1, input_shape=(3, 224, 224)),
            Conv2D(32, 3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Conv2D(64, 3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3, 3)),
            Flatten(),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dense(10, activation='softmax')
    ])
model.compile(Adam(lr=learning_rate), loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization_1 (Batch (None, 3, 224, 224)       12        
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 32, 222, 222)      896       
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 222, 222)      128       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 32, 74, 74)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 64, 72, 72)        18496     
_________________________________________________________________
batch_normalization_3 (Batch (None, 64, 72, 72)        256       
_________________________________________________________________
max_poolin[...]<jupyter_text>fit_generator() consuming the batches provided by the ImageDataGenerator to fit the model to the databatches.n = samples in the datasetbatches.batch_size = number of samples in a batchusing batches.n // batches.batch_size ensures we go over the entire dataset once per epochvalidation_steps = number of batches of samples to validate against<jupyter_code>history = model.fit_generator(batches,
                    steps_per_epoch=batches.n//batches.batch_size,
                    validation_data=val_batches,
                    validation_steps=1,
                    epochs=epochs,
                    callbacks=[reduce_lr, stop_early])
plot_acc_and_loss(history)<jupyter_output><empty_output><jupyter_text>test performance and plot confusion matrix on one sample batch of 200 images<jupyter_code>sample_val_path = DATA_DIR.joinpath('sample/valid')
sam_val_set = get_in_batches(sample_val_path, shuffle=False, class_mode=None, batch_size=200)
pred_classes = model.predict_generator(sam_val_set, 1)
pred_classes = np.argmax(pred_classes, axis=1)
act_classes = sam_val_set.classes
cm = confusion_matrix(act_classes, pred_classes)
plot_confusion_matrix(cm, val_batches.class_indices)
plt.figure()
plt.show()<jupyter_output>[[20  0  0  0  0  0  0  0  0  0]
 [ 0 20  0  0  0  0  0  0  0  0]
 [ 0  0 20  0  0  0  0  0  0  0]
 [ 0  0  0 20  0  0  0  0  0  0]
 [ 0  0  0  0 20  0  0  0  0  0]
 [ 1  0  0  0  0 19  0  0  0  0]
 [ 0  0  0  0  0  0 20  0  0  0]
 [ 0  0  0  0  0  0  0 20  0  0]
 [ 0  0  0  0  0  0  0  0 19  1]
 [ 1  0  0  0  0  0  0  0  0 19]]<jupyter_text>validating the model performance on the val setRunning the evaluate generator returns the cost and accuracy of the model. Doing it in a loop allows us to confirm that the performance is stable. Results should be very similar for all runs. This takes _very_ long though (as long as training or more.. will have to look into this)<jupyter_code>rnd_batches = get_in_batches(val_path, batch_size=batch_size, shuffle=True)
val_res = [model.evaluate_generator(rnd_batches, rnd_batches.samples) for i in range(5)]
np.round(val_res, 3)<jupyter_output><empty_output><jupyter_text>saving weights and model separately to file for reuse<jupyter_code>model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)

model.save('CNN_two_convs_30122017_1700.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08953988745448527, 'fraction_numerical': 0.04965243296921549, 'mean_word_length': 4.259355961705831, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '339473', 'license': 'MIT', 'n_tokens_mistral': 2054, 'n_tokens_neox': 1857, 'n_words': 536, 'path': 'nbs/finalProjectModelNotebook_CNN_two_convs_v01.ipynb', 'repo_name': 'SchlachterSchmidt/precognitive-octo-engine', 'size': '6042'}