<jupyter_start><jupyter_text>Cross Modal RetrievalIn this notebook, we are going to define and train a model which learns to map sentences and images into a common embedding space in order to be able to retrieve one from the other. We will be using image and captions from FLicker-8K dataset to train our network.We learn a joint embedding iamge-sentence embedding space where sentences are encoded using a gated recurrent neural network. Image features from a deep CNN are projected into the embedding space of the GRU hidden states. A pairwise ranking loss is minimized to learn the rank images and their descriptions. Steps to run this npotebook* Download the data and pre-trained files from [here](https://drive.google.com/open?id=1p2P-DgHjgpa_E5_hnK7UvEQ5D4WDJxF1)* Copy the contents from the downloaded files and paste it to your<jupyter_code>import os
import pickle
import numpy as np
import torch
import torch.nn as nn
import torch.nn.init
import torchvision
import torchvision.models as models
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
import torch.backends.cudnn as cudnn

from tqdm import *
from PIL import Image

from flickr.data import get_loader_single, get_transform, collate_fn
from flickr.evaluation import encode_data
from matplotlib import pyplot as plt
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
torch.manual_seed(0)
np.random.seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

ROOT = 'data/'
IMAGE_FOLDER = '/ssd_scratch/cvit/deep/Flickr-8K/Flicker8k_Dataset'
VOCAB_PATH = 'vocab'
PRETRAINED = 'saves'
with open('{}/flickr_precomp_vocab.pkl'.format(VOCAB_PATH), 'rb') as f:
    vocab = pickle.load(f)
    
batch_size = 32<jupyter_output><empty_output><jupyter_text>Load and visualize dataThe `get_data_loader` function returns training and validation DataLoaders that can load data efficiently and in specified batches. The function has the following parameters:* `root`: path containg images and captions* `vocab`: unique words contained in the descriptions.* `batch_size`: number of images and descriptions in one batch of data<jupyter_code>def get_loaders(root, vocab, batch_size):
        transform = get_transform('train')

        train_loader = get_loader_single(root, 'train',
                                        vocab, transform,
                                        batch_size=batch_size, shuffle=True,
                                        collate_fn=collate_fn)

        transform = get_transform('dev')
        
        val_loader = get_loader_single(root, 'dev',
                                    vocab, transform,
                                    batch_size=batch_size, shuffle=False,
                                    collate_fn=collate_fn)
        transform = get_transform('test')
        
        test_loader = get_loader_single(root, 'test',
                                    vocab, transform,
                                    batch_size=batch_size, shuffle=False,
                                    collate_fn=collate_fn)


        return train_loader, val_loader
train_loader, val_loader = get_loaders(IMAGE_FOLDER, vocab, batch_size)
# helper imshow function
def imshow(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    
#np.transpose(npimg, (1, 2, 0))
# get some images from X
dataiter = iter(train_loader)
# the "_" is a placeholder for no labels
images, _, _, _ = dataiter.next()

# show images
fig = plt.figure(figsize=(12, 8))
imshow(torchvision.utils.make_grid(images))<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).<jupyter_text>ModelThe model is composed of a image encoder network and text encoder network. Image EncoderThe Image encoder is composed of a pretrained VGG19 network and a Linear layer. We take the output from last but one layer. The network sees a 224X244X3 image and passes it through5 convolutional blocks, each of which down samples the image by a factor of 2. The output of the last linear layer from the VGG network acts as an input to another linear layer which embeds the image features to the cross-modal subspace.<jupyter_code>def l2norm(X):
    """L2-normalize columns of X
    """
    norm = torch.pow(X, 2).sum(dim=1, keepdim=True).sqrt()
    X = torch.div(X, norm)
    return X

class EncoderImageFull(nn.Module):

    def __init__(self, embed_size, use_abs=False, no_imgnorm=False):
        """Load pretrained VGG19 and replace top fc layer."""
        super(EncoderImageFull, self).__init__()
        self.embed_size = embed_size
        self.no_imgnorm = no_imgnorm
        self.use_abs = use_abs
        # Load a pre-trained model
        
        self.cnn = models.__dict__['vgg19'](pretrained=True)
        self.cnn.features = nn.DataParallel(self.cnn.features)
        self.cnn.cuda()
        # Replace the last fully connected layer of CNN with a new one
        self.fc = nn.Linear(self.cnn.classifier._modules['6'].in_features,
                            embed_size)
        self.cnn.classifier = nn.Sequential(
            *list(self.cnn.classifier.children())[:-1])

        self.init_weights()

    def init_weights(self):
        """Xavier initialization for the fully connected layer
        """
        r = np.sqrt(6.) / np.sqrt(self.fc.in_features +
                                  self.fc.out_features)
        self.fc.weight.data.uniform_(-r, r)
        self.fc.bias.data.fill_(0)

    def forward(self, images):
        """Extract image feature vectors."""
        features = self.cnn(images)

        # normalization in the image embedding space
        features = l2norm(features)

        # linear projection to the joint embedding space
        features = self.fc(features)

        # normalization in the joint embedding space
        if not self.no_imgnorm:
            features = l2norm(features)
        if self.use_abs:
            features = torch.abs(features)

        # take the absolute value of the embedding (used in order embeddings)
        return features<jupyter_output><empty_output><jupyter_text>Text Encoder networkThe text encoder network comprises of an [embedding](https://pytorch.org/docs/stable/nn.htmltorch.nn.Embedding) layer and a [GRU](https://pytorch.org/docs/stable/nn.html?highlight=grutorch.nn.GRU). The embedding layer takes the size of the dictionary as the input dimension and outputs word embeddings in some specied dimension. We then pass each word embedding to the GRU, at each time step and then proceed to take the output from the hidden state at the last time step. The output from the GRU should have dimesion, same as the dimension of sub-space on which we are projecting the image.<jupyter_code>class EncoderText(nn.Module):

    def __init__(self, vocab_size, word_dim, embed_size, use_abs=False):
        super(EncoderText, self).__init__()

        # word embedding
        self.embed = nn.Embedding(vocab_size, word_dim)
        self.embed_size = embed_size
        self.use_abs = use_abs
        # caption embedding
        self.rnn = nn.GRU(word_dim, embed_size, batch_first=True)

        self.init_weights()

    def init_weights(self):
        self.embed.weight.data.uniform_(-0.1, 0.1)

    def forward(self, x, lengths):
        """Handles variable size captions
        """
        # Embed word ids to vectors
        x = self.embed(x)
        packed = pack_padded_sequence(x, lengths, batch_first=True)

        # Forward propagate RNN
        out, _ = self.rnn(packed)

        # Reshape *final* output to (batch_size, hidden_size)
        padded = pad_packed_sequence(out, batch_first=True)
        I = torch.LongTensor(lengths).view(-1, 1, 1)
        I = Variable(I.expand(x.size(0), 1, self.embed_size)-1).cuda()
        out = torch.gather(padded[0], 1, I).squeeze(1)

        # normalization in the joint embedding space
        out = l2norm(out)

        # take absolute value, used by order embeddings
        if self.use_abs:
            out = torch.abs(out)

        return out<jupyter_output><empty_output><jupyter_text>Contrastive LossThe objective is to learn representations with a small distance d between them for positive pairs, and greater distance than some margin value m for negative pairs.For positive pairs, the loss will be 0 only when the net produces representations for both the two elements in the pair with no distance between them, and the loss (and therefore, the corresponding net parameters update) will increase with that distance.For negative pairs, the loss will be 0 when the distance between the representations of the two pair elements is greater than the margin m. But when that distance is not bigger than m, the loss will be positive, and net parameters will be updated to produce more distant representation for those two elements. The loss value will be at most m, when the distance between ra and rn is 0. The function of the margin is that, when the representations produced for a negative pair are distant enough, no efforts are wasted on enlarging that distance, so further training can focus on more difficult pairs.<jupyter_code>class PairwiseRankingLoss(torch.nn.Module):

    def __init__(self, margin=1.0):
        super(PairwiseRankingLoss, self).__init__()
        self.margin = margin

    def forward(self, im, s):
        margin = self.margin
        # compute image-sentence score matrix
        scores = torch.mm(im, s.transpose(1, 0))
        diagonal = scores.diag()

        # compare every diagonal score to scores in its column (i.e, all contrastive images for each sentence)
        cost_s = torch.max(Variable(torch.zeros(scores.size()[0], scores.size()[1]).cuda()), (margin-diagonal).expand_as(scores)+scores)
        # compare every diagonal score to scores in its row (i.e, all contrastive sentences for each image)
        cost_im = torch.max(Variable(torch.zeros(scores.size()[0], scores.size()[1]).cuda()), (margin-diagonal).expand_as(scores).transpose(1, 0)+scores)

        for i in range(scores.size()[0]):
            cost_s[i, i] = 0
            cost_im[i, i] = 0

        return cost_s.sum() + cost_im.sum()
batch_size = 32
vocab_size = len(vocab)
print('Dictionary size: ' + str(vocab_size))
embed_size = 1024
img_dim = 4096
word_dim = 300
num_epochs = 100

img_enc = EncoderImageFull(embed_size).to(device)
txt_enc = EncoderText(vocab_size, word_dim, embed_size).to(device)
params = list(txt_enc.parameters())
params += list(img_enc.fc.parameters())
criterion = PairwiseRankingLoss(margin=0.2)
optimizer = torch.optim.Adam(params, lr=0.0002)

def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR
       decayed by 10 every 30 epochs"""
    lr = 0.0002 * (0.1 ** (epoch // 15))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
def validate(val_loader, txt_enc, img_enc):
    img_embs, cap_embs = encode_data(txt_enc, img_enc, val_loader)
    r1, r5, r10, medr = i2t(img_embs, cap_embs)
    r1i, r5i, r10i, medri = t2i(img_embs, cap_embs)
    score = r1 + r5 + r10 + r1i + r5i + r10i
    return r1
start_epoch=0
for epoch in range(start_epoch, num_epochs):
    adjust_learning_rate(optimizer, epoch)
    for i, batch in enumerate(tqdm(train_loader)):
        images, captions, lengths, ids = batch
        images = images.to(device)
        captions = captions.to(device)
        img_emb = img_enc(images)
        cap_emb = txt_enc(captions, lengths)
        loss = criterion(img_emb, cap_emb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    rsum = validate(val_loader, txt_enc, img_enc)
    print('Epochs: [%d]/[%d] AvgScore: %.2f Loss: %.2f'%(epoch, num_epochs, rsum, loss.item()))<jupyter_output>Dictionary size: 3446<jupyter_text>Evaluation<jupyter_code>def load_image(file_name):
    image = Image.open(file_name)
    return image

def load_checkpoint(savefile):
    if os.path.exists(savefile):
        checkpoint = torch.load(savefile)
        return checkpoint
    else:
        print('No checkpoints available')
        
def get_captions():
    with open('{}/f8k_train_caps.txt'.format(ROOT), 'r') as f:
        lines = f.readlines()
    captions = [line.strip() for line in lines]
    return captions

def text_retrieval(image_embedding, cap_embs, captions) :
    scores = np.dot(image_embedding, cap_embs.T).flatten()
    sorted_args = np.argsort(scores)[::-1]
    sentences = [captions[a] for a in sorted_args[:10]]
    return sentences

savefile = 'saves/model_best_full.t7'   
img_enc = EncoderImageFull(1024, use_abs=True).to(device)
txt_enc = EncoderText(vocab_size, 300, 1024, use_abs=False).to(device)
img_enc.eval()
txt_enc.eval()
checkpoint = load_checkpoint(savefile)
img_enc.load_state_dict(checkpoint['model'][0])
txt_enc.load_state_dict(checkpoint['model'][1])
if not os.path.exists('saves/%s_joint_embs.pkl' %'flickr'):
    img_embs, cap_embs = encode_data(txt_enc, img_enc, train_loader)
    with open('%saves/s_joint_embs.pkl' %'flickr', 'wb') as f:
        pickle.dump([img_embs, cap_embs], f)
else:
    with open('saves/%s_joint_embs.pkl' %'flickr', 'rb') as f:
        embs = pickle.load(f)
        img_embs, cap_embs = embs[0], embs[1]
query_image = load_image('{}/106490881_5a2dd9b7bd.jpg'.format(IMAGE_FOLDER))
transform = get_transform('dev')
query_image = transform(query_image)
query_image_embedding = img_enc(query_image.unsqueeze(0)).data.cpu().numpy()
captions = get_captions()
ret = text_retrieval(query_image_embedding, cap_embs, captions)
plt.figure()
plt.imshow(np.transpose(query_image, (1, 2, 0)))
plt.show()
for each_ret in ret:
    print(each_ret+'\n')<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
{'fraction_non_alphanumeric': 0.09517959095179591, 'fraction_numerical': 0.013731300137313001, 'mean_word_length': 3.3488372093023258, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '635963', 'license': 'MIT', 'n_tokens_mistral': 4428, 'n_tokens_neox': 4245, 'n_words': 1285, 'path': 'cross-ret.ipynb', 'repo_name': 'Deepayan137/ml_ss19_cross-modal', 'size': '13837'}