<jupyter_start><jupyter_text>Assignment 2 Question 1: Siamese networks & one-shot learning (7pt)The Cifar-100 dataset is similar to the Cifar-10 dataset. It also consists of 60,000 32x32 RGB images, but they are distributed over 100 classes instead of 10. Thus, each class has much fewer examples, only 500 training images and 100 testing images per class. For more info about the dataset, see https://www.cs.toronto.edu/~kriz/cifar.html.*HINT: Import the Cifar-100 dataset directly from Keras, no need to download it from the website. Use* `label_mode="fine"` Task 1.1: Siamese network**a)*** Train a Siamese Network on the first 80 classes of (the training set of) Cifar-100, i.e. let the network predict the probability that two input images are from the same class. Use 1 as a target for pairs of images from the same class (positive pairs), and 0 for pairs of images from different classes (negative pairs). Randomly select image pairs from Cifar-100, but make sure you train on as many positive pairs as negative pairs.* Evaluate the performance of the network on 20-way one-shot learning tasks. Do this by generating 250 random tasks and obtain the average accuracy for each evaluation round. Use the remaining 20 classes that were not used for training. The model should perform better than random guessing.For this question you may ignore the test set of Cifar-100; it suffices to use only the training set and split this, using the first 80 classes for training and the remaining 20 classes for one-shot testing.*HINT: First sort the data by their labels (see e.g.* `numpy.argsort()`*), then reshape the data to a shape of* `(n_classes, n_examples, width, height, depth)`*, similar to the Omniglot data in Practical 4. It is then easier to split the data by class, and to sample positive and negative images pairs for training the Siamese network.**NOTE: do not expect the one-shot accuracy for Cifar-100 to be similar to that accuracy for Omniglot; a lower accuracy can be expected. However, accuracy higher than random guess is certainly achievable.*<jupyter_code>import matplotlib.pyplot as plt
from keras.datasets import cifar100
import numpy as np
import pandas as pd
import os
from keras.layers import Input, Conv2D, Lambda, Dense, Flatten, MaxPooling2D, Dropout, BatchNormalization
from keras.models import Model, Sequential
from keras.regularizers import l2
from keras.layers import Dense, Dropout, Conv2D, MaxPooling2D, Flatten, BatchNormalization, Concatenate
from keras.utils import to_categorical
from keras import backend as K
from sklearn.utils import shuffle
from scipy.spatial import distance
# === add code here ===
(x_train, y_train), (x_test, y_test) = cifar100.load_data(label_mode="fine")
print(x_train.shape)
y_train = y_train.flatten()
y_test = y_test.flatten()
print(y_train.shape)
print(y_test.shape)
class_labels = ['apple', 'aquarium_fish', 'baby', 'bear', 'beaver', 'bed', 'bee', 'beetle', 
    'bicycle', 'bottle', 'bowl', 'boy', 'bridge', 'bus', 'butterfly', 'camel', 
    'can', 'castle', 'caterpillar', 'cattle', 'chair', 'chimpanzee', 'clock', 
    'cloud', 'cockroach', 'couch', 'crab', 'crocodile', 'cup', 'dinosaur', 
    'dolphin', 'elephant', 'flatfish', 'forest', 'fox', 'girl', 'hamster', 
    'house', 'kangaroo', 'keyboard', 'lamp', 'lawn_mower', 'leopard', 'lion',
    'lizard', 'lobster', 'man', 'maple_tree', 'motorcycle', 'mountain', 'mouse',
    'mushroom', 'oak_tree', 'orange', 'orchid', 'otter', 'palm_tree', 'pear',
    'pickup_truck', 'pine_tree', 'plain', 'plate', 'poppy', 'porcupine',
    'possum', 'rabbit', 'raccoon', 'ray', 'road', 'rocket', 'rose',
    'sea', 'seal', 'shark', 'shrew', 'skunk', 'skyscraper', 'snail', 'snake',
    'spider', 'squirrel', 'streetcar', 'sunflower', 'sweet_pepper', 'table',
    'tank', 'telephone', 'television', 'tiger', 'tractor', 'train', 'trout',
    'tulip', 'turtle', 'wardrobe', 'whale', 'willow_tree', 'wolf', 'woman',
    'worm']
example_id = 5000  # pick any integer from 0 to 49999 to visualize a training example
example = x_train[example_id]
label = y_train[example_id]
print("Class label:", class_labels[label])
plt.imshow(example)
plt.show()
img_rows,img_cols,chns = 32, 32, 3
n_classes = 100

x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, chns)
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, chns)
input_shape = (img_rows, img_cols, chns)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255


print(x_train.shape)
print(y_train.shape)
print(x_test.shape)
print(y_test.shape)
indexes_train = y_train.argsort()
indexes_test = y_test.argsort()
print(indexes_train)
print(indexes_test)
x_train_sort = x_train[indexes_train]
y_train_sort = y_train[indexes_train]

x_test_sort = x_test[indexes_test]
y_test_sort = y_test[indexes_test]

print(x_train_sort.shape)
print(y_train_sort.shape)
print(x_test_sort.shape)
print(y_test_sort.shape)

example_id = 39999  # pick any integer from 0 to 49999 to visualize a training example
example = x_train_sort[example_id]
label = y_train_sort[example_id]
print("Class label:", class_labels[label])
plt.imshow(example)
plt.show()
def group(a, b):
    # Get argsort indices, to be used to sort a and b in the next steps
    sidx = b.argsort(kind='mergesort')
    a_sorted = a[sidx]
    b_sorted = b[sidx]

    # Get the group limit indices (start, stop of groups)
    cut_idx = np.flatnonzero(np.r_[True,b_sorted[1:] != b_sorted[:-1],True])

    # Create cut indices for all unique IDs in b
    n = b_sorted[-1]+2
    cut_idxe = np.full(n, cut_idx[-1], dtype=int)

    insert_idx = b_sorted[cut_idx[:-1]]
    cut_idxe[insert_idx] = cut_idx[:-1]
    cut_idxe = np.minimum.accumulate(cut_idxe[::-1])[::-1]

    # Split input array with those start, stop ones
    out = [a_sorted[i:j] for i,j in zip(cut_idxe[:-1],cut_idxe[1:])]
    return np.array(out)
x_train_shape = group(x_train_sort, y_train_sort)
x_test_shape = group(x_test, y_test)
print(x_train_shape.shape)
print(x_test.shape)
x_train_samples = x_train_shape[:80]
y_train_samples = y_train_sort[:40000]

x_train_rem_samples = x_train_shape[80:]
y_train_rem_samples = y_train_sort[40000:]

print(x_train_samples.shape)
print(y_train_samples.shape)
print(x_train_rem_samples.shape)
print(y_train_rem_samples.shape)
input_shape = (32, 32, 3)
left_input = Input(input_shape)
right_input = Input(input_shape)

# build convnet to use in each siamese 'leg'
convnet = Sequential()

convnet.add(Conv2D(64, (3,3), activation='relu', input_shape=input_shape, kernel_regularizer=l2(2e-4)))
convnet.add(MaxPooling2D())
convnet.add(BatchNormalization())
convnet.add(Dropout(0.25))

convnet.add(Conv2D(128, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet.add(MaxPooling2D())
convnet.add(BatchNormalization())
convnet.add(Dropout(0.25))

convnet.add(Conv2D(128, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet.add(BatchNormalization())
convnet.add(Dropout(0.25))

convnet.add(Conv2D(256, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet.add(Flatten())
convnet.add(BatchNormalization())
convnet.add(Dropout(0.25))

convnet.add(Dense(4096, activation="sigmoid", kernel_regularizer=l2(1e-3)))
convnet.summary()

# encode each of the two inputs into a vector with the convnet
encoded_l = convnet(left_input)
encoded_r = convnet(right_input)

# merge two encoded inputs with the L1 distance between them, and connect to prediction output layer
L1_distance = lambda x: K.abs(x[0]-x[1])
both = Lambda(L1_distance)([encoded_l, encoded_r])
prediction = Dense(1, activation='sigmoid')(both)
siamese_net = Model(inputs=[left_input,right_input], outputs=prediction)

siamese_net.compile(loss="binary_crossentropy", optimizer="adam")

siamese_net.summary()
def get_batch(batch_size, X):
    """Create batch of n pairs, half same class, half different class"""
    n_classes, n_examples, w, h, d = X.shape
    # randomly sample several classes to use in the batch
    categories = np.random.choice(n_classes, size=(batch_size,), replace=False)
    # initialize 2 empty arrays for the input image batch
    pairs = [np.zeros((batch_size, h, w, d)) for i in range(2)]
    # initialize vector for the targets, and make one half of it '1's, so 2nd half of batch has same class
    targets = np.zeros((batch_size,))
    targets[batch_size//2:] = 1
    for i in range(batch_size):
        category = categories[i]
        idx_1 = np.random.randint(0, n_examples)
        pairs[0][i, :, :, :] = X[category, idx_1].reshape(w, h, d)
        idx_2 = np.random.randint(0, n_examples)
        # pick images of same class for 1st half, different for 2nd
        if i >= batch_size // 2:
            category_2 = category
        else:
            #add a random number to the category modulo n_classes to ensure 2nd image has different category
            category_2 = (category + np.random.randint(1,n_classes)) % n_classes
        pairs[1][i, :, :, :] = X[category_2,idx_2].reshape(w, h, d)
    return pairs, targets

def batch_generator(batch_size, X):
    """a generator for batches, so model.fit_generator can be used. """
    while True:
        pairs, targets = get_batch(batch_size, X)
        yield (pairs, targets)

def train(model, X_train, batch_size=64, steps_per_epoch=100, epochs=10):
    model.fit_generator(batch_generator(batch_size, X_train), steps_per_epoch=steps_per_epoch, epochs=epochs)
def get_batch_test(batch_size, X):
    """Create batch of n pairs, half same class, half different class"""
    n_classes, n_examples, w, h, d = X.shape
    # randomly sample several classes to use in the batch
    categories = np.random.choice(n_classes, size=(batch_size,), replace=False)
    # initialize 2 empty arrays for the input image batch
    pairs = [np.zeros((batch_size, h, w, d)) for i in range(2)]
    # initialize vector for the targets, and make one half of it '1's, so 2nd half of batch has same class
    targets = np.zeros((batch_size,))
    targets[batch_size//2:] = 1
    for i in range(batch_size):
        category = categories[i]
        idx_1 = np.random.randint(0, n_examples)
        pairs[0][i, :, :, :] = X[category, idx_1].reshape(w, h, d)
        idx_2 = np.random.randint(0, n_examples)
        # pick images of same class for 1st half, different for 2nd
        if i >= batch_size // 2:
            category_2 = category
        else:
            #add a random number to the category modulo n_classes to ensure 2nd image has different category
            category_2 = (category + np.random.randint(1,n_classes)) % n_classes
        pairs[1][i, :, :, :] = X[category_2,idx_2].reshape(w, h, d)
    return pairs, targets



print(x_train_samples.shape)

t1, t2 = get_batch_test(64, x_train_samples)
print(len(t1), len(t2))
print(t1[0].shape)
print(t1[1].shape)
print(t2)
def make_oneshot_task(N, X, c, language=None):
    """Create pairs of (test image, support set image) with ground truth, for testing N-way one-shot learning."""
    n_classes, n_examples, w, h, d = X.shape
    indices = np.random.randint(0, n_examples, size=(N,))
    if language is not None:
        low, high = c[language]
        if N > high - low:
            raise ValueError("This language ({}) has less than {} letters".format(language, N))
        categories = np.random.choice(range(low,high), size=(N,), replace=False)
    else:  # if no language specified just pick a bunch of random letters
        categories = np.random.choice(range(n_classes), size=(N,), replace=False)            
    true_category = categories[0]
    ex1, ex2 = np.random.choice(n_examples, replace=False, size=(2,))
    test_image = np.asarray([X[true_category, ex1, :, :]]*N).reshape(N, w, h, d)
    support_set = X[categories, indices, :, :]
    support_set[0, :, :] = X[true_category, ex2]
    support_set = support_set.reshape(N, w, h, d)
    targets = np.zeros((N,))
    targets[0] = 1
    targets, test_image, support_set = shuffle(targets, test_image, support_set)
    pairs = [test_image, support_set]
    return pairs, targets

def test_oneshot(model, X, c, N=20, k=250, language=None, verbose=True):
    """Test average N-way oneshot learning accuracy of a siamese neural net over k one-shot tasks."""
    n_correct = 0
    if verbose:
        print("Evaluating model on {} random {}-way one-shot learning tasks ...".format(k, N))
    for i in range(k):
        inputs, targets = make_oneshot_task(N, X, c, language=language)
        probs = model.predict(inputs)
        print('---------------------------')
        print(len(inputs))
        print(inputs[0].shape)
        print(len(targets))
#         print(targets.shape)
        print(targets)
        print(len(probs))
        
        print(np.argmax(probs))
        print(np.argmax(targets))
        print('---------------------------')
        if np.argmax(probs) == np.argmax(targets):
            n_correct += 1
    percent_correct = (100.0*n_correct / k)
    if verbose:
        print("Got an average of {}% accuracy for {}-way one-shot learning".format(percent_correct, N))
    return percent_correct
best_acc = 0

# Siamese training
# train(siamese_net, x_train_samples)
# siamese_net.summary()
# 20 way-one shot learning task
test_acc = test_oneshot(siamese_net, x_test_shape, y_test)
if test_acc >= best_acc:
    print("New best one-shot accuracy, saving model ...")
#     siamese_net.save(os.path.join("models", "siamese_omniglot.h5"))
    best_acc = test_acc
print(x_test_shape.shape)
print(y_test.shape)<jupyter_output>(100, 100, 32, 32, 3)
(10000,)<jupyter_text>*****b)** Compare the performance of your Siamese network for Cifar-100 to the Siamese network from Practical 4 for Omniglot. Name three fundamental differences between the Cifar-100 and Omniglot datasets. How do these differences influence the difference in one-shot accuracy?**Answer:** *=== write your answer here ===* *** Task 1.2: One-shot learning with neural codes**a)*** Train a CNN classifier on the first 80 classes of Cifar-100. Make sure it achieves at least 40% classification accuracy on those 80 classes (use the test set to validate this accuracy).* Then use neural codes from one of the later hidden layers of the CNN with L2-distance to evaluate one-shot learning accuracy for the remaining 20 classes of Cifar-100 with 250 random tasks. I.e. for a given one-shot task, obtain neural codes for the test image as well as the support set. Then pick the image from the support set that is closest (in L2-distance) to the test image as your one-shot prediction.<jupyter_code>x_training_CNN = x_train_sort[:40000]
y_training = y_train_sort[:40000]
y_training_CNN = to_categorical(y_training, 80)

x_test_CNN = x_test_sort[:8000]
y_test = y_test_sort[:8000]
y_test_CNN = to_categorical(y_test, 80)

print(x_training_CNN.shape)
print(x_test_CNN.shape)
print(y_training_CNN.shape)
print(y_test_CNN.shape)

example_id = 15900  # pick any integer from 0 to 49999 to visualize a training example
example = x_training_CNN[example_id]
label = y_train_sort[example_id]
print("Class label:", class_labels[label])
plt.imshow(example)
plt.show()
# === add code here ===
input_shape = (32, 32, 3)
model = Sequential()
model.add(Conv2D(64,(3,3),activation = 'relu', input_shape = input_shape))
model.add(Conv2D(64,(3,3),activation = 'relu'))
model.add(MaxPooling2D())
model.add(BatchNormalization())
model.add(Dropout(0.35))
model.add(Conv2D(128,(3,3),activation = 'relu'))
model.add(Conv2D(128,(3,3),activation = 'relu'))
model.add(MaxPooling2D())
model.add(BatchNormalization())
model.add(Dropout(0.35))
model.add(Flatten())
model.add(Dense(512,name = 'hidden_layer', activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(80,activation = 'softmax'))
model.summary()

model.compile(optimizer="adadelta", loss="categorical_crossentropy", metrics=["accuracy"])
batch_size = 100
epochs = 10
print(x_training_CNN.shape)
print(y_training_CNN.shape)
model.fit(x_training_CNN, y_training_CNN,
          batch_size=batch_size,
          epochs=epochs,
          validation_data = (x_test_CNN, y_test_CNN))
loss, accuracy = model.evaluate(x_test_CNN, y_test_CNN, verbose=1)

print('Test loss:', loss)
print('Test accuracy:', accuracy)
neural_codes_model = Model(inputs = model.input, outputs = model.get_layer("hidden_layer").output)
neural_codes_model.summary()
def make_oneshot_task2(N, X):
    """Create pairs of (test image, support set image) with ground truth, for testing N-way one-shot learning."""
    n_classes, n_examples, w, h, d = X.shape
    indices = np.random.randint(0, n_examples, size=(N,))
    categories = np.random.choice(range(n_classes), size=(N,), replace=False)            
    true_category = categories[0]
    ex1, ex2 = np.random.choice(n_examples, replace=False, size=(2,))
    test_image = np.asarray([X[true_category, ex1, :, :]]*N).reshape(N, w, h, d)
    support_set = X[categories, indices, :, :]
    support_set[0, :, :] = X[true_category, ex2]
    support_set = support_set.reshape(N, w, h, d)
    targets = np.zeros((N,))
    targets[0] = 1
    targets, test_image, support_set = shuffle(targets, test_image, support_set)
    #pairs = [test_image, support_set]
    return test_image, support_set, targets
def make_oneshot_task(N, X, c, language=None):
    """Create pairs of (test image, support set image) with ground truth, for testing N-way one-shot learning."""
    n_classes, n_examples, w, h, d = X.shape
    indices = np.random.randint(0, n_examples, size=(N,))
    if language is not None:
        low, high = c[language]
        if N > high - low:
            raise ValueError("This language ({}) has less than {} letters".format(language, N))
        categories = np.random.choice(range(low,high), size=(N,), replace=False)
    else:  # if no language specified just pick a bunch of random letters
        categories = np.random.choice(range(n_classes), size=(N,), replace=False)            
    true_category = categories[0]
    ex1, ex2 = np.random.choice(n_examples, replace=False, size=(2,))
    test_image = np.asarray([X[true_category, ex1, :, :]]*N).reshape(N, w, h, d)
    support_set = X[categories, indices, :, :]
    support_set[0, :, :] = X[true_category, ex2]
    support_set = support_set.reshape(N, w, h, d)
    targets = np.zeros((N,))
    targets[0] = 1
    targets, test_image, support_set = shuffle(targets, test_image, support_set)
    pairs = [test_image, support_set]
    return pairs, targets

def test_oneshot(model, X, c, N=20, k=250, language=None, verbose=True):
    """Test average N-way oneshot learning accuracy of a siamese neural net over k one-shot tasks."""
    n_correct = 0
    if verbose:
        print("Evaluating model on {} random {}-way one-shot learning tasks ...".format(k, N))
    for i in range(k):
        inputs, targets = make_oneshot_task(N, X, c, language=language)
        probs = model.predict(inputs)
        if np.argmax(probs) == np.argmax(targets):
            n_correct += 1
    percent_correct = (100.0*n_correct / k)
    if verbose:
        print("Got an average of {}% accuracy for {}-way one-shot learning".format(percent_correct, N))
    return percent_correct
# One shot learning
N=20
n_correct = 0
k=250

for _ in range(0,250):
    test_image, support_set, targets = make_oneshot_task2(N,x_train_rem_samples)
    test_image_pred = neural_codes_model.predict(test_image)
    support_set_pred = neural_codes_model.predict(support_set)
    distances = []

    for i in range(0,20):
        dist = distance.cityblock(test_image_pred[i],support_set_pred[i])
        distances.append(dist)
    if np.argmin(distances) == np.argmax(targets):
        n_correct += 1
accuracy = (100.0*n_correct / k)
print("Got an average of {}% accuracy for {}-way one-shot learning".format(accuracy, N))<jupyter_output>Got an average of 19.6% accuracy for 20-way one-shot learning<jupyter_text>*****b)** Briefly motivate your CNN architecture, and discuss the difference in one-shot accuracy between the Siamese network approach and the CNN neural codes approach.**Answer:** *=== write your answer here ===*Initially the feature for convolution layer was 32, 64 and 128. but the accuracy was not achieved as there isnt enough features. Dropout of 0.2 to 0.5 was tried with the same architecture. Then we came to conclusion that the features were not enough for it to train on, so decided to choose more features and thus started with 64 and 128 in convolution layer. The maxpoolig was added and dropouts were chosen to be 0.35 to avoid overfitting and drop the similar neurons to reduce the density of the network. BatchNormalization was done in between these steps to normalize the output from maxpooling before dropping out and also to increase the speed of training. The last layer is fully connected dense layer with 100 output classes and the activation is softmax as we need to choose from the classification. The optimizer adadelta continuously learns and is most suitable for classification tasks. Trying with 10 epoch we achieved an accuracy of 40.23% whereas with 30 epochs and having 2 dense layers (one with 512 neurons and output with 100 neurons) we achieved an accuracy 41.84%. This result is due to extensive training of the model with more iterations *** Question 2: Triplet networks & one-shot learning (10pt) Task 2.1: Train a triplet network**a)*** Train a triplet network on the first 80 classes of (the training set of) Cifar-100. * Make sure the network achieves a smaller loss than the margin and the network does not collapse all representations to zero vectors. *HINT: If you experience problems to achieve this goal, it might be helpful to tinker the learning rate.** You are provided with a working example of triplet loss implementation for Keras below. You may directly use it.You may ignore the test set of Cifar-100 for this question as well. It suffices to use only the training set and split this, using the first 80 classes for training and the remaining 20 classes for one-shot testing.```python Notice that ground truth variable is not used for loss calculation. It is used as a function argument to by-pass some Keras functionality. This is because the network structure already implies the ground truth for the anchor image with the "positive" image.import tensorflow as tfdef triplet_loss(ground_truth, network_output):    anchor, positive, negative = tf.split(network_output, num_or_size_splits=3, axis=1)                for embedding in [anchor, positive, negative]:        embedding = tf.math.l2_normalize(embedding)    pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, positive)), axis=1)    neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, negative)), axis=1)        margin =  define your margin    basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), margin)    loss = tf.reduce_mean(tf.maximum(basic_loss, 0.0), axis=0)    return loss```<jupyter_code># Triplet Loss function, created to bypass some Keras loss implementation requirements to work with current use case

import tensorflow as tf
def triplet_loss(ground_truth, network_output):

    anchor, positive, negative = tf.split(network_output, num_or_size_splits=3, axis=1)        

    for embedding in [anchor, positive, negative]:
        embedding = tf.math.l2_normalize(embedding)

    pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, positive)), axis=1)
    neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, negative)), axis=1)

    margin = 0.5# define your margin
    basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), margin)
    loss = tf.reduce_mean(tf.maximum(basic_loss, 0.0), axis=0)

    return loss

# in_dims = [32,32,3]



def get_triplets(batch_size, X):
    """Create batch of n triplets, half same class, half different class"""
    n_classes, n_examples, w, h, d = X.shape
#     print(n_classes, n_examples, w,h,d)
    # randomly sample several classes to use in the batch
    categories = np.random.choice(n_classes, size=(batch_size,), replace=False)
#     print("Categories:")
#     print(categories)
    # initialize 3 empty arrays for the input image batch
    triplets = [np.zeros((batch_size, h, w, d)) for i in range(3)]
    # initialize vector for the targets, all zeroes coz there's both a positive and a negative, and loss doesn't use labels, which is super fascinating, 
    # in a weird way, since the architecture design is unconventional
    targets = np.zeros((batch_size,))
#     targets[batch_size//2:] = 1
    for i in range(batch_size):
        category = categories[i]
        idx_1 = np.random.randint(0, n_examples)
        triplets[0][i, :, :, :] = X[category, idx_1].reshape(w, h, d)
        idx_2 = np.random.randint(0, n_examples)
        triplets[1][i, :, :, :] = X[category, idx_2].reshape(w, h, d)
        idx_3 = np.random.randint(0, n_examples)
        category_2 = (category + np.random.randint(1,n_classes)) % n_classes
        triplets[2][i, :, :, :] = X[category_2, idx_3].reshape(w, h, d)
        # pick images of same class for 1st half, different for 2nd
    return triplets, targets



def triplet_generator(batch_size, X):
    """a generator for batches, so model.fit_generator can be used. """
    while True:
        triplets, targets = get_triplets(batch_size, X)
        yield (triplets, targets)

def train_triplet_network(model, X_train, batch_size=64, steps_per_epoch=100, epochs=40):
    model.fit_generator(triplet_generator(batch_size, X_train), steps_per_epoch=steps_per_epoch, epochs=epochs)
        
# Data for triplet

x_training_triplet = x_train_samples
y_training_triplet = y_train_samples

print(x_training_triplet.shape)
print(y_training_triplet.shape)

x_test_triplet = x_train_rem_samples
y_test_triplet = y_train_rem_samples

print(x_test_triplet.shape)
print(y_test_triplet.shape)

def get_triplet_test(batch_size, X):
    """Create batch of n triplets, half same class, half different class"""
    n_classes, n_examples, w, h, d = X.shape
    print(n_classes, n_examples, w,h,d)
    # randomly sample several classes to use in the batch
    categories = np.random.choice(n_classes, size=(batch_size,), replace=False)
    print("Categories:")
    print(categories)
    # initialize 3 empty arrays for the input image batch
    triplets = [np.zeros((batch_size, h, w, d)) for i in range(3)]
    # initialize vector for the targets, all zeroes coz there's both a positive and a negative, and loss doesn't use labels, which is super fascinating, 
    # in a weird way, since the architecture design is unconventional
    targets = np.zeros((batch_size,))
#     targets[batch_size//2:] = 1
    for i in range(batch_size):
        category = categories[i]
        idx_1 = np.random.randint(0, n_examples)
        triplets[0][i, :, :, :] = X[category, idx_1].reshape(w, h, d)
        idx_2 = np.random.randint(0, n_examples)
        triplets[1][i, :, :, :] = X[category, idx_2].reshape(w, h, d)
        idx_3 = np.random.randint(0, n_examples)
        category_2 = (category + np.random.randint(1,n_classes)) % n_classes
        triplets[2][i, :, :, :] = X[category_2, idx_3].reshape(w, h, d)
        # pick images of same class for 1st half, different for 2nd
    return triplets, targets



print(x_training_triplet.shape)

t1, t2 = get_triplets(64, x_training_triplet)
print(len(t1), len(t2))
print(t1[0].shape)
print(t1[1].shape)
print(t1[2].shape)
print(t2)
train_triplet_network(triplet_net, x_training_triplet)
print(x_training_CNN.shape)

print(y_training_CNN.shape)
print(y_training_CNN)


input_shape = (32, 32, 3)
anchor_input = Input(input_shape)
positive_input = Input(input_shape)
negative_input = Input(input_shape)

# build convnet to use in each siamese 'leg'
convnet_triplet = Sequential()

convnet_triplet.add(Conv2D(64, (3,3), activation='relu', input_shape=input_shape, kernel_regularizer=l2(2e-4)))
convnet_triplet.add(MaxPooling2D())
convnet_triplet.add(BatchNormalization())
convnet_triplet.add(Dropout(0.25))

convnet_triplet.add(Conv2D(128, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet_triplet.add(MaxPooling2D())
convnet_triplet.add(BatchNormalization())
convnet_triplet.add(Dropout(0.25))

convnet_triplet.add(Conv2D(128, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet_triplet.add(BatchNormalization())
convnet_triplet.add(Dropout(0.25))

convnet_triplet.add(Conv2D(256, (3,3), activation='relu', kernel_regularizer=l2(2e-4)))
convnet_triplet.add(Flatten())
convnet_triplet.add(BatchNormalization())
convnet_triplet.add(Dropout(0.25))

# convnet_triplet.add(Dense(6561, activation="sigmoid", kernel_regularizer=l2(1e-3)))
convnet_triplet.summary()

# encode each of the two inputs into a vector with the convnet
encoded_anchor = convnet_triplet(anchor_input)
encoded_positive = convnet_triplet(positive_input)
encoded_negative = convnet_triplet(negative_input)

# merge two encoded inputs with the L1 distance between them, and connect to prediction output layer
# L2_distance = lambda x: K.abs(x[0]-x[1])

# all_three = Lambda(L2_distance)([encoded_anchor, encoded_positive, encoded_negative])
# prediction_t = Dense(1, activation='sigmoid')(all_three)

merged_vector = Concatenate(axis=-1)([encoded_anchor, encoded_positive, encoded_negative])
# prediction_t = Dense(6561, activation="sigmoid", kernel_regularizer=l2(1e-3))(merged_vector)
triplet_net = Model(inputs=[anchor_input, positive_input ,negative_input], outputs=merged_vector)
# triplet_net = Model(inputs=[anchor_input, positive_input ,negative_input])
# triplet_net.add(prediction_t)


triplet_net.compile(loss=triplet_loss, optimizer="adam")

triplet_net.summary()
# Model layers of Triplet network

baseModel = Sequential()
baseModel.add(Conv2D(128,(7,7),padding='same',input_shape=(in_dims[0],in_dims[1],in_dims[2],),activation='relu',name='conv1'))
baseModel.add(MaxPooling2D((2,2),(2,2),padding='same',name='pool1'))
baseModel.add(Conv2D(256,(5,5),padding='same',activation='relu',name='conv2'))
baseModel.add(MaxPooling2D((2,2),(2,2),padding='same',name='pool2'))
baseModel.add(Flatten(name='flatten'))
baseModel.add(Dense(4,name='embeddings'))
# model.add(Dense(600))

anchor_input = Input((32,32,3, ), name='anchor_input')
positive_input = Input((32,32,3, ), name='positive_input')
negative_input = Input((32,32,3, ), name='negative_input')

encoded_anchor = baseModel(anchor_input)
encoded_positive = baseModel(positive_input)
encoded_negative = baseModel(negative_input)

merged_vector = concatenate([encoded_anchor, encoded_positive, encoded_negative], axis=-1, name='merged_layer')

model = Model(inputs=[anchor_input,positive_input, negative_input], outputs=merged_vector)
model.compile(loss=triplet_loss, optimizer=adam_optim)

model.summary()<jupyter_output><empty_output><jupyter_text>*** Task 2.2: One-shot learning with triplet neural codes**a)*** Use neural codes from the triplet network with L2-distance to evaluate one-shot learning accuracy for the remaining 20 classes of Cifar-100 with 250 random tasks. I.e. for a given one-shot task, obtain neural codes for the test image as well as the support set. Then pick the image from the support set that is closest (in L2-distance) to the test image as your one-shot prediction.* Explicitly state the accuracy.<jupyter_code>def make_oneshot_task(N, X, c, language=None):
    """Create pairs of (test image, support set image) with ground truth, for testing N-way one-shot learning."""
    n_classes, n_examples, w, h, d = X.shape
    indices = np.random.randint(0, n_examples, size=(N,))
    if language is not None:
        low, high = c[language]
        if N > high - low:
            raise ValueError("This language ({}) has less than {} letters".format(language, N))
        categories = np.random.choice(range(low,high), size=(N,), replace=False)
    else:  # if no language specified just pick a bunch of random letters
        categories = np.random.choice(range(n_classes), size=(N,), replace=False)            
    true_category = categories[0]
    ex1, ex2 = np.random.choice(n_examples, replace=False, size=(2,))
    test_image = np.asarray([X[true_category, ex1, :, :]]*N).reshape(N, w, h, d)
    support_set = X[categories, indices, :, :]
    support_set[0, :, :] = X[true_category, ex2]
    support_set = support_set.reshape(N, w, h, d)
    targets = np.zeros((N,))
    targets[0] = 1
    targets, test_image, support_set = shuffle(targets, test_image, support_set)
    pairs = [test_image, support_set]
    return pairs, targets

def test_oneshot(model, X, c, N=20, k=250, language=None, verbose=True):
    """Test average N-way oneshot learning accuracy of a siamese neural net over k one-shot tasks."""
    n_correct = 0
    if verbose:
        print("Evaluating model on {} random {}-way one-shot learning tasks ...".format(k, N))
    for i in range(k):
        inputs, targets = make_oneshot_task(N, X, c, language=language)
        probs = model.predict(inputs)
        if np.argmax(probs) == np.argmax(targets):
            n_correct += 1
    percent_correct = (100.0*n_correct / k)
    if verbose:
        print("Got an average of {}% accuracy for {}-way one-shot learning".format(percent_correct, N))
    return percent_correct<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12220174617081055, 'fraction_numerical': 0.02338297936618229, 'mean_word_length': 4.545652538113587, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '211005', 'license': 'Unlicense', 'n_tokens_mistral': 11469, 'n_tokens_neox': 10858, 'n_words': 3168, 'path': 'Siamese_Triplet_OneShotLearning/.ipynb_checkpoints/Assignment_2_2_TriplenetModeltrained-checkpoint.ipynb', 'repo_name': 'Natty07/Deep_Learning', 'size': '33101'}