<jupyter_start><jupyter_text>I. Imports<jupyter_code>import keras
import numpy as np

from keras.datasets import mnist
from keras.optimizers import Adam
from keras.models import Sequential
from keras.preprocessing import image
from keras.layers.core import Dense
from keras.layers.core import Lambda
from keras.layers.core import Flatten
from keras.layers.core import Dropout
from keras.layers.pooling import MaxPooling2D
from keras.layers.convolutional import Convolution2D
from keras.layers.normalization import BatchNormalization
from keras.utils.np_utils import to_categorical<jupyter_output>Using Theano backend.
/home/wnixalo/miniconda3/envs/FAI/lib/python2.7/site-packages/theano/gpuarray/dnn.py:135: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to version 5.1.
  warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 6021 on context None
Mapped name None to device cuda: GeForce GTX 870M (0000:01:00.0)<jupyter_text>I want to import Vgg16 as well because I'll want it's low-level features<jupyter_code># import os, sys
# sys.path.insert(1, os.path.join('../utils/'))<jupyter_output><empty_output><jupyter_text>Actually, looks like Vgg's ImageNet weights won't be needed.<jupyter_code># from vgg16 import Vgg16
# vgg = Vgg16()<jupyter_output><empty_output><jupyter_text>II. Load Data<jupyter_code>(x_train, y_train), (x_test, y_test) = mnist.load_data()<jupyter_output><empty_output><jupyter_text>III. PreprocessingKeras Convolutional layers expect color channels, so expand an empty dimension in the input data, to account for no colors.<jupyter_code>x_train = np.expand_dims(x_train, 1) # can also enter <axis=1> for <1>
x_test = np.expand_dims(x_test, 1)
x_train.shape<jupyter_output><empty_output><jupyter_text>One-Hot Encoding the outputs:<jupyter_code>y_train, y_test = to_categorical(y_train), to_categorical(y_test)<jupyter_output><empty_output><jupyter_text>Since this notebook's models are all mimicking Vgg16, the input data should be preprocessed in the same way: in this case normalized by subtracting the mean and dividing by the standard deviation. It turns out this is a good idea generally.<jupyter_code>x_mean = x_train.mean().astype(np.float32)
x_stdv = x_train.std().astype(np.float32)
def norm_input(x): return (x - x_mean) / x_stdv<jupyter_output><empty_output><jupyter_text>Create Data Batch Generator```ImageDataGenerator``` with no arguments will return a generator. Later, when data is augmented, it'll be told how to do so. I don't know what batch-size should be set to: in Lecture it was 64.<jupyter_code>gen = image.ImageDataGenerator()
trn_batches = gen.flow(x_train, y_train, batch_size=64)
tst_batches = gen.flow(x_test, y_test, batch_size=64)<jupyter_output><empty_output><jupyter_text>General workflow, going forward:* Define the model's architecture.* Run 1 Epoch at default learning rate (0.01 ~ 0.001 depending on optimizer) to get it started.* Jack up the learning to 0.1 (as high as you'll ever want to go) and run 1 Epoch, possibly more if you can get away with it.* Lower the learning rate by a factor of 10 and run for a number of Epochs -- repeat until model begins to overfit (acc > valacc)Points on internal architecture:* Each model will have a data-preprocessing ```Lambda``` layer, which normalizes the input and assigns a shape of (1 color-channel x 28 pixels x 28 pixels)* Weights are flattened before entering FC layers* Convolutional Layers will come in 2 pairs (because this is similar to the Vgg model). * Convol layer-pairs will start with 32 3x3 filters and double to 64 3x3 layers* A MaxPooling Layer comes after each Convol-pair.* When Batch-Normalization is applied, it is done after every layer but last (excluding MaxPooling).* Final layer is always an FC softmax layer with 10 outputs for our 10 digits.* Dropout, when applied, should increase toward later layers.* Optimizer used in Lecture was Adam(), all layers but last use a ReLU activation, loss function is categorical cross-entropy. 1. Linear Modelaka 'Dense', 'Fully-Connected'<jupyter_code>def LinModel():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28)),
        Flatten(),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model
Linear_model = LinModel()
Linear_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                          validation_data=tst_batches, nb_val_samples=trn_batches.n)
Linear_model.optimizer.lr=0.1
Linear_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                          validation_data=tst_batches, nb_val_samples=tst_batches.n)
Linear_model.optimizer.lr=0.01
Linear_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4,
                          validation_data=tst_batches, nb_val_samples=tst_batches.n)
Linear_model.optimizer.lr=0.001
Linear_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=8,
                          validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Epoch 1/8
60000/60000 [==============================] - 16s - loss: 0.2648 - acc: 0.9255 - val_loss: 0.2776 - val_acc: 0.9217
Epoch 2/8
60000/60000 [==============================] - 13s - loss: 0.2612 - acc: 0.9265 - val_loss: 0.2699 - val_acc: 0.9249
Epoch 3/8
60000/60000 [==============================] - 11s - loss: 0.2649 - acc: 0.9265 - val_loss: 0.2766 - val_acc: 0.9237
Epoch 4/8
60000/60000 [==============================] - 13s - loss: 0.2563 - acc: 0.9292 - val_loss: 0.2867 - val_acc: 0.9227
Epoch 5/8
60000/60000 [==============================] - 11s - loss: 0.2586 - acc: 0.9283 - val_loss: 0.2894 - val_acc: 0.9208
Epoch 6/8
60000/60000 [==============================] - 13s - loss: 0.2561 - acc: 0.9286 - val_loss: 0.2790 - val_acc: 0.9237
Epoch 7/8
60000/60000 [==============================] - 18s - loss: 0.2564 - acc: 0.9289 - val_loss: 0.2861 - val_acc: 0.9233
Epoch 8/8
60000/60000 [==============================] - 17s - loss: 0.2548 - acc: 0.9292 - val_loss: 0.2733 - [...]<jupyter_text>2. Single Dense LayerThis is what people in the 80s & 90s thought of as a 'Neural Network': a single Fully-Connected hidden layer. I don't yet know why the hidden layer is ouputting 512 units. For natural-image recognition it's 4096. I'll see whether a ReLU or Softmax hidden layer works better.By the way, the training and hyper-parameter tuning process *should* be automated. I want to use a NN to figure out how to do that for me.<jupyter_code>def FCModel():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28)),
        Dense(512, activation='relu'),
        Flatten(),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model
FC_model = FCModel()
FC_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                      validation_data=tst_batches, nb_val_samples=tst_batches.n)
FC_model.optimizer=0.1
FC_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                      validation_data=tst_batches, nb_val_samples=tst_batches.n)
FC_model.optimizer=0.01
FC_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4,
                      validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Epoch 1/4
60000/60000 [==============================] - 44s - loss: 0.0971 - acc: 0.9707 - val_loss: 0.1309 - val_acc: 0.9630
Epoch 2/4
60000/60000 [==============================] - 34s - loss: 0.0819 - acc: 0.9748 - val_loss: 0.1187 - val_acc: 0.9651
Epoch 3/4
60000/60000 [==============================] - 33s - loss: 0.0679 - acc: 0.9783 - val_loss: 0.1309 - val_acc: 0.9661
Epoch 4/4
60000/60000 [==============================] - 29s - loss: 0.0568 - acc: 0.9823 - val_loss: 0.1212 - val_acc: 0.9664<jupyter_text>With an accuracy of 0.9823 and validation accuracy of 0.9664, the model's starting to overfit significantly and hit its limits, so it's time to go on to the next technique.  3. Basic 'VGG' style Convolutional Neural NetworkI'm specifying an output shape equal to the input shape, to suppress the warnings keras was giving me; and it stated it was defaulting to that anyway. Or maybe I should've written ```output_shape=input_shape```Aha: yes it's as I thought. See [this thread](http://forums.fast.ai/t/warning-output-shape-argument-not-specified/416/10) -- output_shape warnings were added to Keras, and neither vgg16.py (nor I until now) were specifying output_shape. It's fine.The first time I ran this, I forgot to have 2 pairs of Conv layers. At the third λr=0.01 epoch I had acc/val of 0.9964, 0.9878Also noticing: in lecture JH was using a GPU which I think was an NVidia Titan X. I'm using an Intel Core i5 CPU on a MacBook Pro. His epochs took on average 6 seconds, mine are taking 180~190. Convolutions are also the most computationally-intensive part of the NN being built here.Interestingly, the model with 2 Conv-layer pairs is taking avg 160s. Best Acc/Val: ```0.9968/0.9944```Final: ```0.9975/0.9918``` - massive overfitting<jupyter_code>def ConvModel():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28), output_shape=(1, 28, 28)),
        Convolution2D(32, 3, 3, activation='relu'),
        Convolution2D(32, 3, 3, activation='relu'),
        MaxPooling2D(),
        Convolution2D(64, 3, 3, activation='relu'),
        Convolution2D(64, 3, 3, activation='relu'),
        MaxPooling2D(),
        Flatten(),
        Dense(512, activation='relu'),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model
CNN_model = ConvModel()
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)
CNN_model.optimizer=0.1
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)
CNN_model.optimizer=0.01
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)
# Running again until validation accuracy stops increasing
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Epoch 1/4
60000/60000 [==============================] - 166s - loss: 0.0104 - acc: 0.9968 - val_loss: 0.0209 - val_acc: 0.9944
Epoch 2/4
60000/60000 [==============================] - 153s - loss: 0.0095 - acc: 0.9970 - val_loss: 0.0299 - val_acc: 0.9931
Epoch 3/4
60000/60000 [==============================] - 156s - loss: 0.0081 - acc: 0.9977 - val_loss: 0.0384 - val_acc: 0.9907
Epoch 4/4
60000/60000 [==============================] - 158s - loss: 0.0075 - acc: 0.9975 - val_loss: 0.0316 - val_acc: 0.9918<jupyter_text>4. Data Augmentation<jupyter_code>gen = image.ImageDataGenerator(rotation_range=8, width_shift_range=0.08, shear_range=0.3,
                           height_shift_range=0.08, zoom_range=0.08)
trn_batches = gen.flow(x_train, y_train, batch_size=64)
tst_batches = gen.flow(x_test, y_test, batch_size=64)
CNN_Aug_model = ConvModel()
CNN_Aug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                  validation_data=tst_batches, nb_val_samples=tst_batches.n)
# upping LR
print("Learning Rate, η = 0.1")
CNN_Aug_model.optimizer.lr=0.1
CNN_Aug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                  validation_data=tst_batches, nb_val_samples=tst_batches.n)
# brining LR back down for more epochs
print("Learning Rate, η = 0.01")
CNN_Aug_model.optimizer.lr=0.01
CNN_Aug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                  validation_data=tst_batches, nb_val_samples=tst_batches.n)
# 4 more epochs at η=0.01
CNN_Aug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                  validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Epoch 1/4
60000/60000 [==============================] - 167s - loss: 0.0378 - acc: 0.9885 - val_loss: 0.0336 - val_acc: 0.9899
Epoch 2/4
60000/60000 [==============================] - 170s - loss: 0.0346 - acc: 0.9891 - val_loss: 0.0393 - val_acc: 0.9884
Epoch 3/4
60000/60000 [==============================] - 169s - loss: 0.0317 - acc: 0.9902 - val_loss: 0.0295 - val_acc: 0.9913
Epoch 4/4
60000/60000 [==============================] - 169s - loss: 0.0308 - acc: 0.9910 - val_loss: 0.0326 - val_acc: 0.9903<jupyter_text>5. Batch Normalization + Data Augmentation[See this thread](http://forums.fast.ai/t/batchnormalization-axis-1-when-used-on-convolutional-layers/214) for info on BatchNorm axis.<jupyter_code>def ConvModelBN():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28), output_shape=(1, 28, 28)),
        Convolution2D(32, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(32, 3, 3, activation='relu'),
        MaxPooling2D(),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        MaxPooling2D(),
        Flatten(),
        BatchNormalization(),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model
CNN_BNAug_model = ConvModelBN()
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.1")
CNN_BNAug_model.optimizer=0.1
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=2, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.01")
CNN_BNAug_model.optimizer=0.01
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
# some more training at 0.1 and 0.01:
print("Learning Rate, η = 0.1")
CNN_BNAug_model.optimizer=0.1
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.01")
CNN_BNAug_model.optimizer=0.01
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Learning Rate, η = 0.1
Epoch 1/1
60000/60000 [==============================] - 316s - loss: 0.0355 - acc: 0.9884 - val_loss: 0.0362 - val_acc: 0.9887
Learning Rate, η = 0.01
Epoch 1/6
60000/60000 [==============================] - 320s - loss: 0.0309 - acc: 0.9896 - val_loss: 0.0314 - val_acc: 0.9898
Epoch 2/6
60000/60000 [==============================] - 335s - loss: 0.0314 - acc: 0.9898 - val_loss: 0.0320 - val_acc: 0.9901
Epoch 3/6
60000/60000 [==============================] - 318s - loss: 0.0298 - acc: 0.9906 - val_loss: 0.0270 - val_acc: 0.9917
Epoch 4/6
60000/60000 [==============================] - 327s - loss: 0.0302 - acc: 0.9902 - val_loss: 0.0321 - val_acc: 0.9906
Epoch 5/6
60000/60000 [==============================] - 337s - loss: 0.0284 - acc: 0.9909 - val_loss: 0.0261 - val_acc: 0.9920
Epoch 6/6
60000/60000 [==============================] - 350s - loss: 0.0255 - acc: 0.9918 - val_loss: 0.0282 - val_acc: 0.9912<jupyter_text>6. Dropout + Batch Normalization + Data Augmentation<jupyter_code>def ConvModelBNDo():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28), output_shape=(1, 28, 28)),
        Convolution2D(32, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(32, 3, 3, activation='relu'),
        MaxPooling2D(),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        MaxPooling2D(),
        Flatten(),
        BatchNormalization(),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model
CNN_BNDoAug_model = ConvModelBNDo()
CNN_BNDoAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.1")
CNN_BNDoAug_model.optimizer.lr=0.1
CNN_BNDoAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                                validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.01")
CNN_BNDoAug_model.optimizer.lr=0.01
CNN_BNDoAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                                validation_data=tst_batches, nb_val_samples=tst_batches.n)
# 6 more epochs at 0.01
CNN_BNDoAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                                validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate η = 0.001")
CNN_BNDoAug_model.optimizer.lr=0.001
CNN_BNDoAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=12, verbose=1,
                                validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Learning Rate η = 0.001
Epoch 1/12
60000/60000 [==============================] - 314s - loss: 0.0346 - acc: 0.9894 - val_loss: 0.0326 - val_acc: 0.9906
Epoch 2/12
60000/60000 [==============================] - 312s - loss: 0.0324 - acc: 0.9902 - val_loss: 0.0234 - val_acc: 0.9936
Epoch 3/12
60000/60000 [==============================] - 310s - loss: 0.0335 - acc: 0.9895 - val_loss: 0.0218 - val_acc: 0.9930
Epoch 4/12
60000/60000 [==============================] - 309s - loss: 0.0329 - acc: 0.9896 - val_loss: 0.0337 - val_acc: 0.9908
Epoch 5/12
60000/60000 [==============================] - 310s - loss: 0.0314 - acc: 0.9902 - val_loss: 0.0211 - val_acc: 0.9928
Epoch 6/12
60000/60000 [==============================] - 309s - loss: 0.0298 - acc: 0.9910 - val_loss: 0.0287 - val_acc: 0.9925
Epoch 7/12
60000/60000 [==============================] - 312s - loss: 0.0275 - acc: 0.9921 - val_loss: 0.0186 - val_acc: 0.9942
Epoch 8/12
60000/60000 [==============================] - 318s - loss: 0.[...]<jupyter_text>7. Ensembling Define a function to automatically train a model:<jupyter_code># I'll set it to display progress at the start of each LR-change
def train_model():
    model = ConvModelBNDo()
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    
    model.optimizer.lr=0.1
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=3, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    
    model.optimizer.lr=0.01
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=11, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    
    model.optimizer.lr=0.001
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=11, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    return model
# Running a little test on the GPU now
testmodel = ConvModelBNDo()
testmodel.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                        validation_data=tst_batches, nb_val_samples=tst_batches.n)<jupyter_output>Epoch 1/1
60000/60000 [==============================] - 17s - loss: 0.2290 - acc: 0.9318 - val_loss: 0.0768 - val_acc: 0.9752<jupyter_text>I finally got my GPU running on my workstation. Decided to leave the ghost of <NAME> alone and put Ubuntu Linux on the second harddrive. This nvidia GTX 870M takes 17 seconds to get through the 60,000 images. The Core i5 on my Mac took an average of 340. A 20x speed up. This also means, at those numbers, a 6-strong ensemble running the regime in ```train_model()``` will take about 49 minutes and 18 seconds, instead of 16 hours and 26 minutes. You can see what the motivation was, for me to spend ~9 hours today and get the GPU working. It's a warm feeling, knowing your computer isn't just good for playing DOOM, but'll be doing its share of work real soon.So, onward:Create an array of models<jupyter_code># this'll take some time
models = [train_model() for m in xrange(6)]<jupyter_output>Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.2289 - acc: 0.9310 - val_loss: 0.0722 - val_acc: 0.9792
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0966 - acc: 0.9700 - val_loss: 0.0529 - val_acc: 0.9835
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0584 - acc: 0.9821 - val_loss: 0.0392 - val_acc: 0.9872
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0314 - acc: 0.9904 - val_loss: 0.0271 - val_acc: 0.9904
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.2352 - acc: 0.9289 - val_loss: 0.0715 - val_acc: 0.9770
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0950 - acc: 0.9706 - val_loss: 0.0708 - val_acc: 0.9777
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0587 - acc: 0.9819 - val_loss: 0.0365 - val_acc: 0.9884
Epoch 1/1
60000/60000 [==============================] - 18s - loss: 0.0354 - acc: 0.9892 - val_loss: 0.0293 - [...]<jupyter_text>Save the models' weights -- bc this wasn't computationally cheap<jupyter_code>from os import getcwd
path = getcwd() + 'data/mnist/'
model_path = path + 'models/'
for i,m in enumerate(models):
    m.save_weights(model_path + 'MNIST_CNN' + str(i) + '.pkl')<jupyter_output><empty_output><jupyter_text>Create an array of predictions from the models on the test-set. I'm using a batch size of ```256``` because that's what was done in lecture, and prediction is such an easier task that I think the large size just helps things go faster.<jupyter_code>ensemble_preds = np.stack([m.predict(x_test, batch_size=256) for m in models])<jupyter_output><empty_output><jupyter_text>Finally, take the average of the predictions:<jupyter_code>avg_preds = ensemble_preds.mean(axis=0)
keras.metrics.categorical_accuracy(y_test, avg_preds).eval()<jupyter_output><empty_output><jupyter_text>Boom. 0.99699.. ~ 99.7% accuracy. Same as achieved in lecture; took roughly 50 minutes to train. Unfortunately I didn't have the h5py module installed when I ran this, so the weight's can't be saved easily -- simple fix of rerunning after install. Trying the above again, this time having h5py installed.<jupyter_code># this'll take some time
models = [train_model() for m in xrange(6)]
from os import getcwd
import os
path = getcwd() + '/data/mnist/'
model_path = path + 'models/'

if not os.path.exists(path):
    os.mkdir('data')
    os.mkdir('data/mnist')
if not os.path.exists(model_path): os.mkdir(model_path)

for i,m in enumerate(models):
    m.save_weights(model_path + 'MNIST_CNN' + str(i) + '.pkl')
ensemble_preds = np.stack([m.predict(x_test, batch_size=256) for m in models])
avg_preds = ensemble_preds.mean(axis=0)
keras.metrics.categorical_accuracy(y_test, avg_preds).eval()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1608141447368421, 'fraction_numerical': 0.08338815789473684, 'mean_word_length': 4.033319536423841, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '539717', 'license': 'MIT', 'n_tokens_mistral': 10049, 'n_tokens_neox': 8524, 'n_words': 2096, 'path': 'FAI_old/lesson3/L3HW_MNIST.ipynb', 'repo_name': 'WNoxchi/Kawkasos', 'size': '24324'}