<jupyter_start><jupyter_text>Session 11: Transfer LearningLet's see how to use transfer learning to improve our cat/dog modeland to do better image similarity metrics.<jupyter_code>%pylab inline

import numpy as np
import scipy as sp
import pandas as pd
import sklearn
from sklearn import linear_model
import urllib

import os
from os.path import join
import matplotlib.pyplot as plt
import matplotlib.patches as patches

plt.rcParams["figure.figsize"] = (8,8)
from keras.models import Sequential
from keras.layers import Dense, Activation, Conv2D, MaxPooling2D, Flatten
from keras.preprocessing import image
from keras.utils import to_categorical
from keras.optimizers import SGD, RMSprop
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"<jupyter_output><empty_output><jupyter_text>Transfer learning<jupyter_code>from keras.applications.vgg19 import VGG19
from keras.preprocessing import image
from keras.applications.vgg19 import preprocess_input, decode_predictions
from keras.models import Model<jupyter_output><empty_output><jupyter_text>We start by loading a particular neural network model called VGG19. Itcontains 25 layers and over 143 million parameters. The code below readsin the entire model and prints out it structure (unless keras is unavailable,in which case a saved version of the model is printed just for reference).<jupyter_code>vgg19_full = VGG19(weights='imagenet')
vgg19_full.summary()
img_path = join("..", "images", "test", "dog.jpg")
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
x.shape
plt.imshow(img)
plt.axis("off")
y = vgg19_full.predict(x)
for pred in decode_predictions(y)[0]:
    print(pred)<jupyter_output><empty_output><jupyter_text>Transfer learningThe VGG19 model was constructed in order to predict the objects present in an image,but there is a lot more that we can do with the model. The amazing property of deeplearning is that the intermediate results in the neural network operate by detectinglower-level features of the image. For example, the first few detect edges and textures,the next few by understanding shapes, and the latter ones put these together to detectobjects. This is incredibly useful because it means that looking at the intermediateoutputs can tell us something interesting about the images beyond just the 1000predicted categories.Assuming the keras module is installed, we will create a new model that outputs thesecond-to-last output of the model. The prediction of this contains 4096 dimensions.These do not correspond directly to categories, but (in theory) images containingsimilar objects should have similar 4096-dimensional values.<jupyter_code>vgg_fc2 = Model(inputs=vgg19_full.input, outputs=vgg19_full.get_layer('fc2').output)
vgg_fc2.summary()<jupyter_output><empty_output><jupyter_text>And then, we will apply this to the dog image.<jupyter_code>embed = vgg_fc2.predict(x, verbose=True)
embed.shape<jupyter_output><empty_output><jupyter_text>These numbers on their own do not really mean anything specific, butthey do serve as abstract features that we could use in a next processing step.  Applying transfer learning to cats and dogsLet us now apply the transfer learning algorithm to the cats anddogs dataset. Note that this may take a while to run (around 10-15minutes).<jupyter_code>df = pd.read_csv(join("..", "data", "catdog.csv"))
y = np.int32(df.animal.values == "dog")
output = np.zeros((len(df), 224, 224, 3))

for i in range(len(df)):
    img_path = join("..", "images", "catdog", df.filename[i])
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    output[i, :, :, :] = x
    if (i % 100) == 0:
        print("Loaded image {0:03d}".format(i))

output = preprocess_input(output)
img_embed = vgg_fc2.predict(output, verbose=True)<jupyter_output><empty_output><jupyter_text>Check the output of the model, and make sure you understand why thisimage size makes sense.<jupyter_code>img_embed.shape<jupyter_output><empty_output><jupyter_text>Embedding modelWe can use the embedding features to build a predictive model for thedogs and cats. Let's see how well this works. Note that we no longerneed convolutions because the input is no longer raw image data.<jupyter_code>model = Sequential()
model.add(Dense(32, input_shape=(img_embed.shape[1],), activation="relu"))
model.add(Dense(32, activation="relu"))

model.add(Dense(units=2, activation="softmax"))
model.compile(loss='sparse_categorical_crossentropy',
              optimizer=SGD(lr=0.01, momentum=0.8, decay=0.0, nesterov=True),
              metrics=['accuracy'])
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(img_embed, y)
model.fit(X_train, y_train, epochs=5, batch_size=32,
          validation_data=(X_test, y_test))
yhat = model.predict_classes(X_test)
sklearn.metrics.accuracy_score(y_test, yhat)<jupyter_output><empty_output><jupyter_text>We see that the predictive model is significantly better through theuse of transfer learning. And here are the few remaining mistakes:<jupyter_code>plt.figure(figsize=(14, 14))

yhat = model.predict_classes(img_embed)

idx = np.where(yhat != y)[0][:15]

for ind, i in enumerate(idx):
    plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
    plt.subplot(5, 3, ind + 1)

    img = imread(join('..', 'images', 'catdog', df.filename[i]))
    plt.imshow(img)
    plt.axis("off")<jupyter_output><empty_output><jupyter_text>Recommendation systemAnother thing that we can do with deep learning embeddings is to usethe features to find images that are close to one another.<jupyter_code>plt.figure(figsize=(14, 14))

ref_img_num = 750  # change this number!

print(df.iloc[ref_img_num])
idx = np.argsort(np.sum(np.abs(img_embed - img_embed[ref_img_num, :])**2, axis=1))[:9]

for ind, i in enumerate(idx):
    plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
    plt.subplot(3, 3, ind + 1)

    img = imread(join('..', 'images', 'catdog', df.filename[i]))
    plt.imshow(img)
    plt.axis("off")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10144927536231885, 'fraction_numerical': 0.021574440052700924, 'mean_word_length': 5.846674182638106, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '170680', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2040, 'n_tokens_neox': 1932, 'n_words': 607, 'path': 'nb/session11-transfer-learning.ipynb', 'repo_name': 'statsmaths/hilt2019-image-analysis', 'size': '6072'}