<jupyter_start><jupyter_text>Dogs v Cats<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Define path to data: (It's a good idea to put it in a subdirectory of your notebooks folder, and then exclude that directory from git control by adding it to .gitignore.)<jupyter_code>root_path = "data/dogscats-redux/"
weights_path = root_path + 'keith_weights_1.h5'

#path = root_path
path = root_path + "sample/"<jupyter_output><empty_output><jupyter_text>A few basic libraries that we'll need for the initial exercises:<jupyter_code>from __future__ import division,print_function

import os, json, shutil, random, math
from glob import glob
import numpy as np
np.set_printoptions(precision=4, linewidth=100)
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>We have created a file most imaginatively called 'utils.py' to store any little convenience functions we'll want to use. We will discuss these as we use them.<jupyter_code>import utils; reload(utils)
from utils import plots
def predict_and_show(batches):
    imgs, labels = next(batches)
    preds, indexes, predicted_labels = vgg.predict(imgs)
    true_labels = [vgg.classes[idx] for idx in np.argmax(labels, axis=1)]
    plots(imgs, titles=['ok' if a == b else 'wrong' for (a,b) in zip(true_labels, predicted_labels)])<jupyter_output><empty_output><jupyter_text>Use our Vgg16 class to finetune a Dogs vs Cats model<jupyter_code>batch_size=16
import vgg16; reload(vgg16)
vgg = vgg16.Vgg16()
batches = vgg.get_batches(path+'train', batch_size=batch_size)
validation_batches = vgg.get_batches(path+'validation', batch_size=batch_size)<jupyter_output>using vgg16 architecture but NO weights
Found 20000 images belonging to 2 classes.
Found 5000 images belonging to 2 classes.<jupyter_text>Calling *finetune()* modifies the model such that it will be trained based on the data in the batches provided - in this case, to predict either 'dog' or 'cat'.<jupyter_code>vgg.finetune(batches)<jupyter_output><empty_output><jupyter_text>Finally, we *fit()* the parameters of the model using the training data, reporting the accuracy on the validation set after every epoch. (An *epoch* is one full pass through the training data.)<jupyter_code>hist = vgg.fit(batches, validation_batches, nb_epoch=3)
def plot_train_validation_curves(title, training_series, validation_series):
    num_epochs = len(training_series)
    epochs = range(1, num_epochs + 1)
    plt.plot(epochs, validation_series, label='validation')
    plt.plot(epochs, training_series, label='train')
    plt.xlabel('Epochs')
    plt.ylabel(title)
    plt.title(title)
    plt.legend()
    plt.grid(True)
    plt.show()
plot_train_validation_curves('Accuracy', hist.history['acc'], hist.history['val_acc'])
plot_train_validation_curves('Loss', hist.history['loss'], hist.history['val_loss'])<jupyter_output><empty_output><jupyter_text>Try it out<jupyter_code>dummy_batches = vgg.get_batches(path+'train', batch_size=6)
imgs, labels = next(dummy_batches)
plots(imgs, titles=labels)
vgg.predict(imgs)
predict_and_show(dummy_batches)<jupyter_output><empty_output><jupyter_text>Kaggle Submission<jupyter_code>from keras.preprocessing.image import ImageDataGenerator

def get_test_batches(path, batch_size):
    gen = ImageDataGenerator()
    return gen.flow_from_directory(path, target_size=(224,224), class_mode=None, shuffle=False, batch_size=batch_size)
test_batches = get_test_batches(path + 'test', batch_size=8)
dog_predictions = vgg.model.predict_generator(test_batches, test_batches.nb_sample)[:,1]
dog_predictions[:5]<jupyter_output><empty_output><jupyter_text>Extract the id of each image in the test batch because Keras ImageDataGenerator will enumerate them in lexical order<jupyter_code>ids = [int(x.split("/")[1].split(".")[0]) for x in test_batches.filenames]<jupyter_output><empty_output><jupyter_text>fill out the submission table<jupyter_code>with open(root_path + 'keith_submission_4.csv', 'w') as f:
    f.write('id,label\n')
    for i in range(len(ids)):
        f.write('{},{}\n'.format(ids[i], dog_predictions[i]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10911736178467507, 'fraction_numerical': 0.011639185257032008, 'mean_word_length': 7.1683168316831685, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '561487', 'license': 'Apache-2.0', 'n_tokens_mistral': 1439, 'n_tokens_neox': 1406, 'n_words': 340, 'path': 'deeplearning1/nbs/lesson1-redux-4.ipynb', 'repo_name': 'klazuka/courses', 'size': '4124'}