<jupyter_start><jupyter_text>Image Captioning with LSTMThis is a partial implementation of "Show and Tell: A Neural Image Caption Generator" (http://arxiv.org/abs/1411.4555), borrowing heavily from <NAME>'s NeuralTalk (https://github.com/karpathy/neuraltalk)This example consists of three parts:1. COCO Preprocessing - prepare the dataset by precomputing image representations using GoogLeNet2. COCO RNN Training - train a network to predict image captions3. COCO Caption Generation - use the trained network to caption new images  OutputThis notebook prepares the dataset by extracting a vector representation of each image using the GoogLeNet CNN pretrained on ImageNet. A link to download the final result is given in the next notebook. PrerequisitesTo run this notebook, you'll need to download the MSCOCO [training](http://msvocds.blob.core.windows.net/coco2014/train2014.zip) and [validation](http://msvocds.blob.core.windows.net/coco2014/val2014.zip) datasets, and unzip them into './coco/'.The [captions](http://cs.stanford.edu/people/karpathy/deepimagesent/caption_datasets.zip) should be downloaded as well and unzipped into './captions/'<jupyter_code>import sklearn
import numpy as np
import lasagne
import skimage.transform

from lasagne.utils import floatX

import theano
import theano.tensor as T

import matplotlib.pyplot as plt
%matplotlib inline

import json
import pickle<jupyter_output><empty_output><jupyter_text>Functions for building the GoogLeNet model with Lasagne are defined in googlenet.py:<jupyter_code>import googlenet<jupyter_output><empty_output><jupyter_text>We need to download parameter values for the pretrained network<jupyter_code>!wget -N https://s3.amazonaws.com/lasagne/recipes/pretrained/imagenet/blvc_googlenet.pkl<jupyter_output><empty_output><jupyter_text>Build the model and select layers we need - the features are taken from the final network layer, before the softmax nonlinearity.<jupyter_code>cnn_layers = googlenet.build_model()
cnn_input_var = cnn_layers['input'].input_var
cnn_feature_layer = cnn_layers['loss3/classifier']
cnn_output_layer = cnn_layers['prob']

get_cnn_features = theano.function([cnn_input_var], lasagne.layers.get_output(cnn_feature_layer))<jupyter_output><empty_output><jupyter_text>Load the pretrained weights into the network<jupyter_code>model_param_values = pickle.load(open('blvc_googlenet.pkl'))['param values']
lasagne.layers.set_all_param_values(cnn_output_layer, model_param_values)<jupyter_output><empty_output><jupyter_text>The images need some preprocessing before they can be fed to the CNN<jupyter_code>MEAN_VALUES = np.array([104, 117, 123]).reshape((3,1,1))

def prep_image(im):
    if len(im.shape) == 2:
        im = im[:, :, np.newaxis]
        im = np.repeat(im, 3, axis=2)
    # Resize so smallest dim = 224, preserving aspect ratio
    h, w, _ = im.shape
    if h < w:
        im = skimage.transform.resize(im, (224, w*224/h), preserve_range=True)
    else:
        im = skimage.transform.resize(im, (h*224/w, 224), preserve_range=True)

    # Central crop to 224x224
    h, w, _ = im.shape
    im = im[h//2-112:h//2+112, w//2-112:w//2+112]
    
    rawim = np.copy(im).astype('uint8')
    
    # Shuffle axes to c01
    im = np.swapaxes(np.swapaxes(im, 1, 2), 0, 1)
    
    # Convert to BGR
    im = im[::-1, :, :]

    im = im - MEAN_VALUES
    return rawim, floatX(im[np.newaxis])<jupyter_output><empty_output><jupyter_text>Let's verify that GoogLeNet and our preprocessing are functioning properly<jupyter_code>im = plt.imread('./coco/val2014/COCO_val2014_000000391895.jpg')
plt.imshow(im)
rawim, cnn_im = prep_image(im)
plt.imshow(rawim)
p = get_cnn_features(cnn_im)
CLASSES = pickle.load(open('blvc_googlenet.pkl'))['synset words']
print(CLASSES[p.argmax()])<jupyter_output><empty_output><jupyter_text>Load the caption data<jupyter_code>dataset = json.load(open('./captions/dataset_coco.json'))['images']<jupyter_output><empty_output><jupyter_text>Iterate over the dataset and add a field 'cnn features' to each item. This will take quite a while.<jupyter_code>def chunks(l, n):
    for i in xrange(0, len(l), n):
        yield l[i:i + n]

for chunk in chunks(dataset, 256):
    cnn_input = floatX(np.zeros((len(chunk), 3, 224, 224)))
    for i, image in enumerate(chunk):
        fn = './coco/{}/{}'.format(image['filepath'], image['filename'])
        try:
            im = plt.imread(fn)
            _, cnn_input[i] = prep_image(im)
        except IOError:
            continue
    features = get_cnn_features(cnn_input)
    for i, image in enumerate(chunk):
        image['cnn features'] = features[i]<jupyter_output><empty_output><jupyter_text>Save the final product<jupyter_code>pickle.dump(dataset, open('coco_with_cnn_features.pkl','w'), protocol=pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11367414796342477, 'fraction_numerical': 0.02493765586034913, 'mean_word_length': 5.054088050314466, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '50092', 'license': 'MIT', 'n_tokens_mistral': 1747, 'n_tokens_neox': 1638, 'n_words': 395, 'path': 'Recurrent Networks/COCO Preprocessing.ipynb', 'repo_name': 'nraythz/Theano-and-Lasagne', 'size': '4821'}