title image similarity estimation using a siamese network with a contrastive loss mehdi date created 20210506 last modified 20220910 description similarity learning using a siamese network trained with a contrastive loss accelerator gpu introduction siamese networkshttps en wikipedia orgwikisiameseneuralnetwork are neural networks which share weights between two or more sister networks each producing embedding vectors of its respective inputs in supervised similarity learning the networks are then trained to maximize the contrast distance between embeddings of inputs of different classes while minimizing the distance between embeddings of similar classes resulting in embedding spaces that reflect the class segmentation of the training inputs setup hyperparameters load the mnist dataset change the data type to a floating point format define training and validation sets keep 50 of trainval in validation set create pairs of images we will train the model to differentiate between digits of different classes for example digit 0 needs to be differentiated from the rest of the digits 1 through 9 digit 1 from 0 and 2 through 9 and so on to carry this out we will select n random images from class a for example for digit 0 and pair them with n random images from another class b for example for digit 1 then we can repeat this process for all classes of digits until digit 9 once we have paired digit 0 with other digits we can repeat this process for the remaining classes for the rest of the digits from 1 until 9 creates a tuple containing image pairs with corresponding label arguments x list containing images each index in this list corresponds to one image y list containing labels each label with datatype of int returns tuple containing two numpy arrays as pairsofsamples labels where pairsofsamples shape is 2lenx 2 nfeaturesdims and labels are a binary array of shape 2lenx add a matching example add a nonmatching example make train pairs make validation pairs make test pairs we get pairstrain shape 60000 2 28 28 we have 60 000 pairs each pair contains 2 images each image has shape 28 28 split the training pairs split the validation pairs split the test pairs visualize pairs and their labels creates a plot of pairs and labels and prediction if it s test dataset arguments pairs numpy array of pairs to visualize having shape number of pairs 2 28 28 toshow int number of examples to visualize default is 6 toshow must be an integral multiple of numcol otherwise it will be trimmed if it is greater than numcol and incremented if if it is less then numcol numcol int number of images in one row default is 3 for test and train respectively it should not exceed 3 and 7 predictions numpy array of predictions with shape toshow 1 default is none must be passed when testtrue test boolean telling whether the dataset being visualized is train dataset or test dataset default false returns none define numrow if toshow numcol 0 trim toshow to trim toshow limit numrow to the point where toshow numcol 0 if toshownumcol 0 then it means numcol is greater then toshow increment toshow to increment toshow set numrow to 1 toshow must be an integral multiple of numcol we found numrow and we have numcol to increment or decrement toshow to make it integral multiple of numcol simply set it equal to numrow numcol plot the images if the number of rows is 1 the axes array is onedimensional inspect training pairs inspect validation pairs inspect test pairs define the model there are two input layers each leading to its own network which produces embeddings a lambda layer then merges them using an euclidean distancehttps en wikipedia orgwikieuclideandistance and the merged output is fed to the final network provided two tensors t1 and t2 euclidean distance sqrtsumsquaret1t2 find the euclidean distance between two vectors arguments vects list containing two tensors of same length returns tensor containing euclidean distance as floating point value between vectors as mentioned above siamese network share weights between tower networks sister networks to allow this we will use same embedding network for both tower networks define the contrastive loss provides contrastiveloss an enclosing scope with variable margin arguments margin integer defines the baseline for distance for which pairs should be classified as dissimilar default is 1 returns contrastiveloss function with data margin attached contrastive loss mean 1truevalue squareprediction truevalue square maxmarginprediction 0 calculates the contrastive loss arguments ytrue list of labels each label is of type float32 ypred list of predictions of same length as of ytrue each label is of type float32 returns a tensor containing contrastive loss as floating point value compile the model with the contrastive loss train the model visualize results plots the given metric from history arguments history history attribute of history object returned from model fit metric metric to plot a string value present as key in history title a string to be used as title of plot hasvalid boolean true if valid data was passed to model fit else false returns none plot the accuracy plot the contrastive loss evaluate the model visualize the predictions example available on huggingface trained model demo generic badgehttps img shields iobadgef09fa49720modelsiamese20networkblack svghttps huggingface cokerasiosiamesecontrastive generic badgehttps img shields iobadgef09fa49720spacessiamese20networkblack svghttps huggingface cospaceskerasiosiamesecontrastive introduction siamese networks https en wikipedia org wiki siamese_neural_network are neural networks which share weights between two or more sister networks each producing embedding vectors of its respective inputs in supervised similarity learning the networks are then trained to maximize the contrast distance between embeddings of inputs of different classes while minimizing the distance between embeddings of similar classes resulting in embedding spaces that reflect the class segmentation of the training inputs setup hyperparameters margin for contrastive loss load the mnist dataset change the data type to a floating point format define training and validation sets keep 50 of train_val in validation set create pairs of images we will train the model to differentiate between digits of different classes for example digit 0 needs to be differentiated from the rest of the digits 1 through 9 digit 1 from 0 and 2 through 9 and so on to carry this out we will select n random images from class a for example for digit 0 and pair them with n random images from another class b for example for digit 1 then we can repeat this process for all classes of digits until digit 9 once we have paired digit 0 with other digits we can repeat this process for the remaining classes for the rest of the digits from 1 until 9 creates a tuple containing image pairs with corresponding label arguments x list containing images each index in this list corresponds to one image y list containing labels each label with datatype of int returns tuple containing two numpy arrays as pairs_of_samples labels where pairs_of_samples shape is 2len x 2 n_features_dims and labels are a binary array of shape 2len x add a matching example add a non matching example make train pairs make validation pairs make test pairs we get pairs_train shape 60000 2 28 28 we have 60 000 pairs each pair contains 2 images each image has shape 28 28 split the training pairs x_train_1 shape is 60000 28 28 split the validation pairs x_val_1 shape 60000 28 28 split the test pairs x_test_1 shape 20000 28 28 visualize pairs and their labels creates a plot of pairs and labels and prediction if it s test dataset arguments pairs numpy array of pairs to visualize having shape number of pairs 2 28 28 to_show int number of examples to visualize default is 6 to_show must be an integral multiple of num_col otherwise it will be trimmed if it is greater than num_col and incremented if if it is less then num_col num_col int number of images in one row default is 3 for test and train respectively it should not exceed 3 and 7 predictions numpy array of predictions with shape to_show 1 default is none must be passed when test true test boolean telling whether the dataset being visualized is train dataset or test dataset default false returns none define num_row if to_show num_col 0 trim to_show to trim to_show limit num_row to the point where to_show num_col 0 if to_show num_col 0 then it means num_col is greater then to_show increment to_show to increment to_show set num_row to 1 to_show must be an integral multiple of num_col we found num_row and we have num_col to increment or decrement to_show to make it integral multiple of num_col simply set it equal to num_row num_col plot the images if the number of rows is 1 the axes array is one dimensional inspect training pairs inspect validation pairs inspect test pairs define the model there are two input layers each leading to its own network which produces embeddings a lambda layer then merges them using an euclidean distance https en wikipedia org wiki euclidean_distance and the merged output is fed to the final network provided two tensors t1 and t2 euclidean distance sqrt sum square t1 t2 find the euclidean distance between two vectors arguments vects list containing two tensors of same length returns tensor containing euclidean distance as floating point value between vectors as mentioned above siamese network share weights between tower networks sister networks to allow this we will use same embedding network for both tower networks define the contrastive loss provides contrastive_loss an enclosing scope with variable margin arguments margin integer defines the baseline for distance for which pairs should be classified as dissimilar default is 1 returns contrastive_loss function with data margin attached contrastive loss mean 1 true_value square prediction true_value square max margin prediction 0 calculates the contrastive loss arguments y_true list of labels each label is of type float32 y_pred list of predictions of same length as of y_true each label is of type float32 returns a tensor containing contrastive loss as floating point value compile the model with the contrastive loss train the model visualize results plots the given metric from history arguments history history attribute of history object returned from model fit metric metric to plot a string value present as key in history title a string to be used as title of plot has_valid boolean true if valid data was passed to model fit else false returns none plot the accuracy plot the contrastive loss evaluate the model visualize the predictions example available on huggingface trained model demo generic badge https img shields io badge f0 9f a4 97 20model siamese 20network black svg https huggingface co keras io siamese contrastive generic badge https img shields io badge f0 9f a4 97 20spaces siamese 20network black svg https huggingface co spaces keras io siamese contrastive
import random
import numpy as np
import keras
from keras import ops
import matplotlib.pyplot as plt



epochs = 10
batch_size = 16
margin = 1  


(x_train_val, y_train_val), (x_test, y_test) = keras.datasets.mnist.load_data()


x_train_val = x_train_val.astype("float32")
x_test = x_test.astype("float32")





x_train, x_val = x_train_val[:30000], x_train_val[30000:]
y_train, y_val = y_train_val[:30000], y_train_val[30000:]
del x_train_val, y_train_val





def make_pairs(x, y):
    

    num_classes = max(y) + 1
    digit_indices = [np.where(y == i)[0] for i in range(num_classes)]

    pairs = []
    labels = []

    for idx1 in range(len(x)):
        
        x1 = x[idx1]
        label1 = y[idx1]
        idx2 = random.choice(digit_indices[label1])
        x2 = x[idx2]

        pairs += [[x1, x2]]
        labels += [0]

        
        label2 = random.randint(0, num_classes - 1)
        while label2 == label1:
            label2 = random.randint(0, num_classes - 1)

        idx2 = random.choice(digit_indices[label2])
        x2 = x[idx2]

        pairs += [[x1, x2]]
        labels += [1]

    return np.array(pairs), np.array(labels).astype("float32")



pairs_train, labels_train = make_pairs(x_train, y_train)


pairs_val, labels_val = make_pairs(x_val, y_val)


pairs_test, labels_test = make_pairs(x_test, y_test)





x_train_1 = pairs_train[:, 0]  
x_train_2 = pairs_train[:, 1]



x_val_1 = pairs_val[:, 0]  
x_val_2 = pairs_val[:, 1]



x_test_1 = pairs_test[:, 0]  
x_test_2 = pairs_test[:, 1]





def visualize(
    pairs, labels, to_show=6, num_col=3, predictions=None, test=False
):
    

    
    
    
    
    
    
    
    
    
    
    num_row = to_show // num_col if to_show // num_col != 0 else 1

    
    
    
    
    
    to_show = num_row * num_col

    
    fig, axes = plt.subplots(num_row, num_col, figsize=(5, 5))
    for i in range(to_show):
        
        if num_row == 1:
            ax = axes[i % num_col]
        else:
            ax = axes[i // num_col, i % num_col]

        ax.imshow(
            ops.concatenate([pairs[i][0], pairs[i][1]], axis=1), cmap="gray"
        )
        ax.set_axis_off()
        if test:
            ax.set_title(
                "True: {} | Pred: {:.5f}".format(labels[i], predictions[i][0])
            )
        else:
            ax.set_title("Label: {}".format(labels[i]))
    if test:
        plt.tight_layout(rect=(0, 0, 1.9, 1.9), w_pad=0.0)
    else:
        plt.tight_layout(rect=(0, 0, 1.5, 1.5))
    plt.show()




visualize(pairs_train[:-1], labels_train[:-1], to_show=4, num_col=4)



visualize(pairs_val[:-1], labels_val[:-1], to_show=4, num_col=4)



visualize(pairs_test[:-1], labels_test[:-1], to_show=4, num_col=4)






def euclidean_distance(vects):
    

    x, y = vects
    sum_square = ops.sum(ops.square(x - y), axis=1, keepdims=True)
    return ops.sqrt(ops.maximum(sum_square, keras.backend.epsilon()))


input = keras.layers.Input((28, 28, 1))
x = keras.layers.BatchNormalization()(input)
x = keras.layers.Conv2D(4, (5, 5), activation="tanh")(x)
x = keras.layers.AveragePooling2D(pool_size=(2, 2))(x)
x = keras.layers.Conv2D(16, (5, 5), activation="tanh")(x)
x = keras.layers.AveragePooling2D(pool_size=(2, 2))(x)
x = keras.layers.Flatten()(x)

x = keras.layers.BatchNormalization()(x)
x = keras.layers.Dense(10, activation="tanh")(x)
embedding_network = keras.Model(input, x)


input_1 = keras.layers.Input((28, 28, 1))
input_2 = keras.layers.Input((28, 28, 1))




tower_1 = embedding_network(input_1)
tower_2 = embedding_network(input_2)

merge_layer = keras.layers.Lambda(euclidean_distance, output_shape=(1,))(
    [tower_1, tower_2]
)
normal_layer = keras.layers.BatchNormalization()(merge_layer)
output_layer = keras.layers.Dense(1, activation="sigmoid")(normal_layer)
siamese = keras.Model(inputs=[input_1, input_2], outputs=output_layer)





def loss(margin=1):
    

    
    
    def contrastive_loss(y_true, y_pred):
        

        square_pred = ops.square(y_pred)
        margin_square = ops.square(ops.maximum(margin - (y_pred), 0))
        return ops.mean((1 - y_true) * square_pred + (y_true) * margin_square)

    return contrastive_loss




siamese.compile(
    loss=loss(margin=margin), optimizer="RMSprop", metrics=["accuracy"]
)
siamese.summary()




history = siamese.fit(
    [x_train_1, x_train_2],
    labels_train,
    validation_data=([x_val_1, x_val_2], labels_val),
    batch_size=batch_size,
    epochs=epochs,
)




def plt_metric(history, metric, title, has_valid=True):
    
    plt.plot(history[metric])
    if has_valid:
        plt.plot(history["val_" + metric])
        plt.legend(["train", "validation"], loc="upper left")
    plt.title(title)
    plt.ylabel(metric)
    plt.xlabel("epoch")
    plt.show()



plt_metric(history=history.history, metric="accuracy", title="Model accuracy")


plt_metric(history=history.history, metric="loss", title="Contrastive Loss")



results = siamese.evaluate([x_test_1, x_test_2], labels_test)
print("test loss, test acc:", results)



predictions = siamese.predict([x_test_1, x_test_2])
visualize(
    pairs_test, labels_test, to_show=3, predictions=predictions, test=True
)